CSC/ECE 517 Fall 2019 - E1969. Fixes for reviews not being available

E1969 - Investigating inabilities to review

Issue #1321

Reported Problem

It was reported by a student that an assignment was not able to be reviewed even though it was during the review period. This bug was thought to be fixed in the past when Issue #915 was merged. However, in December of 2018, another student reported this issue occurring again.

Investigation Approach

It is suspected that if this bug exists, it is caused by an abnormal user flow path. That is, caused by a user navigating to the point where they should be able to conduct a review, but doing so in an indirect way. To begin our investigation of this bug, a flow chart showing most plausible user flow paths was created. We then used this flow chart to guide our investigation efforts. This flow chart is found below:

From this point, we began to manually test the user flow paths represented in our flow chart on our local development environment. Doing this revealed no evidence that the bug exists. We then set out to create an automated test suite that would test the user flow paths in a more quick and reliable way. The code for these tests, as well as the test cases they cover are produced in the testing section below.

Testing

The purpose of the following test is to iterate over a set of plausible user flow paths that terminate in being able to begin a review. The following flow paths are tested, where each step in the path is either a user clicking in the UI or navigating to a page directly using their browser:

Clicking around in UI like normal

1. Assignments -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin
2. Contact Us -> Assignments -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin
3. Home -> Assignments -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin
4. Profile -> Assignments -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin

Going from different pages directly to the specific assignment page using address bar

1. Assignments -> /student_task/view?id=3 -> Others' work -> Request a new submission to review -> Begin
2. Contact Us -> /student_task/view?id=3 -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin
3. Home -> /student_task/view?id=3 -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin
4. Profile -> /student_task/view?id=3 -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin

Going from different pages directly to the specific assignment review page using address bar

1. Assignments -> /student_review/list?id=3 -> Request a new submission to review -> Begin
2. Contact Us-> /student_review/list?id=3 -> Request a new submission to review -> Begin
3. Home -> /student_review/list?id=3 -> Request a new submission to review -> Begin
4. Profile -> /student_review/list?id=3 -> Request a new submission to review -> Begin

Going from assignment list to specific assignment page, back to assignment list and then finishing review

1. student_task/list -> ReviewTestAssignment -> /student_task/list -> ReviewTestAssignment -> Others' work -> Request a new submission to review -> Begin

These test cases are implemented using the following code. In an attempt to keep the code DRY, we loop over the test cases contained in a large array. This array consists of smaller arrays, each representing a user flow path. Each test case array also contains smaller arrays which are the steps in this user flow path. The first element in these "step" arrays are the name of the button/link the browser should click/visit. The second element is what should be present on the page once the navigation is complete. This approach not only keeps the code DRY, but makes it trivial to add further test cases in the future, should the need arise.

There is further evidence, in addition to the empirical data produced by the above test, that this bug does not exist - the lack of reproducibility. Over the course of the past two semesters, there have been roughly 130 students participating in the course. Each student is required to do at least 2 reviews for each of the 3 peer reviewed assignments. This totals up to at least 780 reviews. Given that students are able to do up to 4 reviews per assignment for extra credit, this can really be as high as 1,170 reviews. Considering there has only been one report of this bug, with no further evidence that it exists (either given then, or present at this time), it is likely that this bug report was a fabrication.

Results

When run in a local development environment, all tests within the suite pass. Further, when the pull request was put in, all tests run by Travis CI pass.

Conclusion

Given that this bug was not able to be reproduced by any of our testing, including a suite of automated tests, it is our opinion that this bug does not exist.