CSC/ECE 517 Fall 2018 E1864: Issues related to Reviewing

Project Introduction & Motivation

Expertiza is an open source web-based peer review system developed and maintained by students and faculty members at North Carolina State University. It has been developed as Open Source Software using Ruby on Rails framework. Expertiza allows students to form teams, review each other's work such as assignments and projects. It provides the role based access so that instructor can impose restriction what students can view/edit.

The primary motivation behind this project is to give the students to review the projects of their peers effortlessly and fix all the issues of the previous implementation.

Description of Project

Expertiza Project : E1864 ‘Issues Related to Reviewing' is intended to fix five known issues in the current Expertiza Version related to reviewing student work. Issues are centered around peer reviewing and assignment feature calibration. The motivation behind this project is to identify and resolve each bug in an attempt to improve the current system version. The project has been divided into required fixes. Four of the five problems are presented in Figure 1 - Application Flow Diagram below. This image is a general representation describing when each of the issues is encountered during a user session. Problem 5 (Issue # 1142) is not present in this figure as the problem is simply associated with the coding of a view file and not directly connected to the flow of students and reviews. Additional discussion on proposed approaches to meet the requirements are explained in subsequent sections.

Figure 1 - Application Flow Diagram

-----------------------------------------------

UML diagram for reviw

Actors:

• Instructor - This actor is responsible for creating assignments and adding students to the assignment.
• Student - This actor is responsible for submitting, reviewing other students work.

Database:

• The database where all the data of Expertiza is getting stored.

Problem Statements & Descriptions

For this project, we have been asked to fix the follow bugs that have been found in the current Expertiza system :

Problem 1 : Issue #1093

• When a student requests a project to review, duplicate reviews appear in the list of available projects.
• Example : The project name for which the review was performed earlier should not appear for the next list of reviews when a student requests a review to perform. This could lead to students reviewing the same project more than once and the requirement is to review different projects each time.

Steps to reproduce the Issue #1093: Figure 1 gives the holistic view on stages in which we see the issues. In this part, we will discuss the exact steps that need to be followed to reproduce the bugs.

• Login as Instructor Create new assignment and associate to the course as shown in Figure 4
• Add a participant to the assignment. there are two ways to do this in bulk first way is importing all the course participants to assignment . second approach is creating a file containing student details and import them to expertiza
• Create topics and teams in expertiza either entering them one by one or from the file
• In actual implementation topic association to the teams involves bidding process but for ease of use, we let instructor assign the topic (Figure 5). set min and max number of reviews allowed as shown in
• Set the rubrics and number of review rounds (min one round is needed) (Figure 6)
• Review assignment should be automatic not instructor controlled(scope of the stated issue is here) as shown in Figure 7
• Set the deadlines for submission and review (shown in Figure 8)
• Login as a student and submit work(this needs to be done for each team), request for review
• If the issue persist system assign the same topic to review multiple times(as shown in Figure2).difficulty with this problem is that its reproducibility factor which is very low.

Problem 2 : Issue # 1097

• When a project author resubmits their assignment, the reviewers should be able to update, not edit, their reviews. This should hold even if those reviews have already been submitted. The current system does not allow reviewers to update their reviews.
• Example : When the reviewer submits a review for a project, the reviewer cannot edit/ update his review again. Before submitting it says "Once submitted, the review cannot be changed again". After clicking yes the reviewer is only able to view his reviews and not update even before the deadline.

This situation can occur in the following scenario. When a reviewer submits the review-1 a few minutes after the deadline, the rubric for review 2 occurs. So the review done for the first round actually becomes review for round 2. Hence the reviewer would not be able to update this review. In situations like these, updating the submitted review becomes useful.

Steps to reproduce the issue:

• Initial steps to reproduce this problem is same as the issue #1093
• Change the deadlines for the review to a few days after the present date and check if you are able to update the review. The picture shows deadline to submit is after 14 days. But the reviews are only able to be viewed.

Problem 3 : Issue # 1029

• When a reviewer tries to review a project, sometimes he/she is shown that the project has not yet been submitted when in fact it has.
• Example : An author of a project has submitted his/her assignment to Expertiza. When a reviewer tries to review the work, sometimes it happens that the system shows the reviewer that the work has not been submitted yet. The reviewer should have been shown the work of the author to review.

Steps to reproduce the Issue #1029:

• steps to reproduce this issue is similar to issue #1093, only the difference is once reviewer requests the assignment to review system throws a message saying there is no work has been submitted even though there is actually submitted work. This issue is not always reproducible

Problem 4 : Issue # 972

• When updating the number of review rounds for an existing project, and error is generated and this action fails.
• Example : An instructor attempts to edit an existing assignment and change the number of review rounds. The reassignment causes an error in the system and this prevents the project update to occur.

Steps to reproduce the Issue #972 :

• The Github Bug Report states that "You can change the # of rounds from 1 to 2. You can then change the # of rounds back to 1, and the assignment will appear to be saved with one round of review. But if you edit it again, the 2 rounds will still be present." Several attempts to reproduce this issue were made. Below are two of such test cases.

- Test Case 1 : Increment beyond current number of rounds then decrements back to the original assigned number of rounds for an existing assignment.

• Step 1 - Log in as instructor6
• Step 2 - Go to the Manage Courses page and select - CSC 517 Spring 2015
• Step 3 - Select the Edit option for - Final projects
• Step 4 - Select the due dates tab as seen in figure 12
• Step 5a - Change the number of rounds from 2 to 3
• Step 6 - Press the Set option
• Step 7a - Select Save

- One can see from Figure 13 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

• Step 5b - Change the number of rounds from 3 back to 2
• Step 6 - Press the Set option
• Step 7b - Select Save

- One can see from Figure 14 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

- Test Case 2 : Decrement below current number of rounds an existing assignment with submissions.

• Step 1 - Log in as instructor6
• Step 2 - Go to the Manage Courses page and select - CSC 517 Spring 2016
• Step 3 - Select the Edit option for - Program 1: Class portal
• Step 4 - Select the due dates tab as seen in figure 12
• Step 5a - Change the number of rounds from 2 to 1
• Step 6 - Press the Set option
• Step 7 - Confirm the reduction below the current number of rounds
• Step 8 - Press OK to exit the notification popup

Addition steps 6-8 can be seen in Figure 15 below.

• Step 7a - Select Save

- One can see from Figure 16 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

Several similar tests were conducted in an attempt to reproduce this issue; however, none were successful.

Problem 5 : Issue # 1142

• “Summary Result Assignment” page shows raw HTML when it should displayed as processed HTML.
• Example : As seen in Figure 17 below, the result of hovering over a review result displays the message in raw HTML. The intended result is for review comment to be rendered properly while displaying.

Figure 17 was taken from the actual deployed version of Expertiza through a person student account.

-Test Case 1 : Hover for Reviewer Comment

• Step 1 - Log in as instructor6
• Step 2 - Select Manage and Impersonate User
• Step 3 - Impersonate student6370
• Step 4 - Selec OSS project/Writing assignment 2
• Step 5 - Hover over any underlined review score

Figure 18 was taken from the development version of Expertiza through the test case steps mentioned above.

• After testing and research into this issue, it has been determined that the issue has been adjusted in the development image and has not yet been merged with the deployed version.

Files to be Modified

• app/models/review_assignment.rb
• app/controllers/assignments_controller.rb
• app/views/grades/view_team.html.erb

Solution Design

Problem 1 : Issue # 1093

• If the issue reproducible consistantly we need to fix reject_previously_reviewed_submissions(contributor_set, reviewer) method in review_assignment.rb model file. This method is responsible for the seperating the current review topic and previously reviewed topics. solution flow diagram is shown below

Problem 2 : Issue # 1097

• To solve this problem we can let the reviewers to update the review until the deadline and remove the confirmation message "Once submitted, it cannot be changed again" before the deadline.
• We can let the reviewers update their reviews multiple times before the deadline. We can change the system design which says only "view" after submitting the review to "view and Update" until the deadline and after which it can be changed back to only "view".

Problem 3 : Issue # 1029

• Solution to this issue involves identifying students work as correctly submitted or not submitted so that system will always know that submission status this identification should be available to the method which assigns topics to the reviewer.

Problem 4 : Issue # 972

• As no attempts to reproduce the stated issue were successful, no solution to this problem can be proposed. In the Github Bug Report there is a reference to a similar Github Bug Report that encompasses this same issue that has been marked closed. For the purposed of this project, testing of this believed fix in the update_feedback_assignment_form_attributes function of assignments_controller.rb will be tested to ensure correctness.

Problem 5 : Issue # 1142

• In this case, we need to identify the mistakes in HTML file and then correct them. final expected outcome is to have processed HTML file.

- We believe that this issue stems from one of two places:

(1) The reviewer content is being captured and stored into the database as a string still containing the html formatting and then being populated into the view as a string containing those tags.

(2) There is a tag missing from view_team.html.erb where this hovered content is being requested. There are eight instances where hovered content is referenced. More testing review of the view file will lend to narrowing down the actual line where the problem resides.

Testing Plan

Testing Plan will be implemented upon the discovery of fixed bugs described in the project document as a means of verifying the fix.

To ensure correctness of bug fixes following test plan was implemented:

• Obtain the appropriate testing environment via a provided virtual box image and RSPEC testing framework.
• Acquire a deeper understanding of the necessary models and views, their functionalities and dependencies.
• Create factories and doubles to assist in the testing of these models and views methods.
• Mock message passing and expected outcomes.
• Apply generated helper objects and mocks to achieve high test coverage of methods within the models and views that were updated.

References

• Project Document
• Tutorial videos on creating assignments