CSC/ECE 517 Fall 2018 E1864: Issues related to Reviewing

Project Introduction & Motivation

Expertiza is an open source web-based peer review system developed and maintained by students and faculty members at North Carolina State University. It has been developed as Open Source Software using Ruby on Rails framework. Expertiza allows students to form teams, review each other's work such as assignments and projects. It provides the role based access so that instructor can impose restriction what students can view/edit.

The primary motivation behind this project is to give the students to review the projects of their peers effortlessly and fix all the issues of the previous implementation.

Description of Project

Expertiza Project : E1864 ‘Issues Related to Reviewing' is intended to fix five known issues in the current Expertiza Version related to reviewing student work. Issues are centered around peer reviewing and assignment feature calibration. The motivation behind this project is to identify and resolve each bug in an attempt to improve the current system version. The project has been divided into required fixes. Four of the five problems are presented in Figure 1 - Application Flow Diagram below. This image is a general representation describing when each of the issues is encountered during a user session. Problem 5 (Issue # 1142) is not present in this figure as the problem is simply associated with the coding of a view file and not directly connected to the flow of students and reviews. Additional discussion on proposed approaches to meet the requirements are explained in subsequent sections.

Figure 1 - Application Flow Diagram

-----------------------------------------------

UML diagram for reviw

Actors:

• Instructor - This actor is responsible for creating assignments and adding students to the assignment.
• Student - This actor is responsible for submitting, reviewing other students work.

Database:

• The database where all the data of Expertiza is getting stored.

Problem Statements & Descriptions

For this project, we have been asked to fix the follow bugs that have been found in the current Expertiza system :

Problem 1 : Issue #1093

• When a student requests a project to review, duplicate reviews appear in the list of available projects.
• Example : The project name for which the review was performed earlier should not appear for the next list of reviews when a student requests a review to perform. This could lead to students reviewing the same project more than once and the requirement is to review different projects each time.

Steps to reproduce the Issue #1093: Figure 1 gives the holistic view on stages in which we see the issues. In this part, we will discuss the exact steps that need to be followed to reproduce the bugs.

• Login as Instructor Create new assignment and associate to the course as shown in Figure 4
• Add a participant to the assignment. there are two ways to do this in bulk first way is importing all the course participants to assignment . second approach is creating a file containing student details and import them to expertiza
• Create topics and teams in expertiza either entering them one by one or from the file
• In actual implementation topic association to the teams involves bidding process but for ease of use, we let instructor assign the topic (Figure 5). set min and max number of reviews allowed as shown in
• Set the rubrics and number of review rounds (min one round is needed) (Figure 6)
• Review assignment should be automatic not instructor controlled(scope of the stated issue is here) as shown in Figure 7
• Set the deadlines for submission and review (shown in Figure 8)
• Login as a student and submit work(this needs to be done for each team), request for review
• If the issue persist system assign the same topic to review multiple times(as shown in Figure2).difficulty with this problem is that its reproducibility factor which is very low.

Problem 2 : Issue # 1097

• When a project author resubmits their assignment, the reviewers should be able to update, not edit, their reviews. This should hold even if those reviews have already been submitted. The current system does not allow reviewers to update their reviews.
• Example : When the reviewer submits a review for a project, the reviewer cannot edit/ update his review again. Before submitting it says "Once submitted, the review cannot be changed again". After clicking yes the reviewer is only able to view his reviews and not update even before the deadline.

This situation can occur in the following scenario. When a reviewer submits the review-1 a few minutes after the deadline, the rubric for review 2 occurs. So the review done for the first round actually becomes review for round 2. Hence the reviewer would not be able to update this review. In situations like these, updating the submitted review becomes useful.

Steps to reproduce the issue:

• Initial steps to reproduce this problem is same as the issue #1093
• Change the deadlines for the review to a few days after the present date and check if you are able to update the review. The picture shows deadline to submit is after 14 days. But the reviews are only able to be viewed.

Problem 3 : Issue # 1029

• When a reviewer tries to review a project, sometimes he/she is shown that the project has not yet been submitted when in fact it has.
• Example : An author of a project has submitted his/her assignment to Expertiza. When a reviewer tries to review the work, sometimes it happens that the system shows the reviewer that the work has not been submitted yet. The reviewer should have been shown the work of the author to review.

Steps to reproduce the Issue #1029:

• steps to reproduce this issue is similar to issue #1093, only the difference is once reviewer requests the assignment to review system throws a message saying there is no work has been submitted even though there is actually submitted work. This issue is not always reproducible

Problem 4 : Issue # 972

• When updating the number of review rounds for an existing project, and error is generated and this action fails.
• Example : An instructor attempts to edit an existing assignment and change the number of review rounds. The reassignment causes an error in the system and this prevents the project update to occur.

Steps to reproduce the Issue #972 :

• The Github Bug Report states that "You can change the # of rounds from 1 to 2. You can then change the # of rounds back to 1, and the assignment will appear to be saved with one round of review. But if you edit it again, the 2 rounds will still be present." Several attempts to reproduce this issue were made. Below are two of such test cases.

- Test Case 1 : Increment beyond current number of rounds then decrements back to the original assigned number of rounds for an existing assignment.

• Step 1 - Log in as instructor6
• Step 2 - Go to the Manage Courses page and select - CSC 517 Spring 2015
• Step 3 - Select the Edit option for - Final projects
• Step 4 - Select the due dates tab as seen in figure 12
• Step 5a - Change the number of rounds from 2 to 3
• Step 6 - Press the Set option
• Step 7a - Select Save

- One can see from Figure 13 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

• Step 5b - Change the number of rounds from 3 back to 2
• Step 6 - Press the Set option
• Step 7b - Select Save

- One can see from Figure 14 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

- Test Case 2 : Decrement below current number of rounds an existing assignment with submissions.

• Step 1 - Log in as instructor6
• Step 2 - Go to the Manage Courses page and select - CSC 517 Spring 2016
• Step 3 - Select the Edit option for - Program 1: Class portal
• Step 4 - Select the due dates tab as seen in figure 12
• Step 5a - Change the number of rounds from 2 to 1
• Step 6 - Press the Set option
• Step 7 - Confirm the reduction below the current number of rounds
• Step 8 - Press OK to exit the notification popup

Addition steps 6-8 can be seen in Figure 15 below.

• Step 7a - Select Save

- One can see from Figure 16 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

• Several similar tests were conducted in an attempt to reproduce this issue; however, none were successful.
• Unstated Issue: there is an issue that arises from the reduction of rounds for assignments that currently contain submissions. After the udpate, When an current student attempts to go and review his/her scores, this causes a NoMethodError in Grades#view_team. A bug should be created for this issue for another round of project issues.

Problem 5 : Issue # 1142

• “Summary Result Assignment” page shows raw HTML when it should displayed as processed HTML.
• Example : As seen in Figure 17 below, the result of hovering over a review result displays the message in raw HTML. The intended result is for review comment to be rendered properly while displaying.

Figure 17 was taken from the actual deployed version of Expertiza through a person student account.

-Test Case 1 : Hover for Reviewer Comment

• Step 1 - Log in as instructor6
• Step 2 - Select Manage and Impersonate User
• Step 3 - Impersonate student6370
• Step 4 - Selec OSS project/Writing assignment 2
• Step 5 - Hover over any underlined review score

Figure 18 was taken from the development version of Expertiza through the test case steps mentioned above.

• After testing and research into this issue, it has been determined that the issue has been adjusted in the development image and has not yet been merged with the deployed version.

Files to be Modified

• app/views/student_review/_responses.html.erb
• app/views/response/response.html.erb
• app/controllers/response_controller.rb
• spec/controllers/response_controller_spec.rb
• spec/features/assignment_round_adjustment.rb

Solution Design

Problem 1 : Issue # 1093

• Several attempts have been made to reproduce this problem. We tested with both old and latest database but the issue did not reproduce.
• We also tried with creating a new assignment, adding participants to the assignment, assign reviews to the student then logging in as student and request for a topic to review even with this issue did not reproduce
• After discussing with mentor and Dr.Gehringer we are marking this issue as not reproducible

Problem 2 : Issue # 1097

• To solve this problem we will edit code in app/views/student_review/_responses.html.erb, app/views/response/response.html.erb, and app/controllers/response_controller.rb to enable a reviewer to update the review until the round deadline. We will also remove the confirmation message "Once submitted, it cannot be changed again" before the deadline to appropriate reflect this edit.
• Once these changes have been made, the new view will be reflected as seen in the figure below.

-The reviewer now has the option to both view and update their submitted reviews multiple times before the round deadline This is in contrast to the previous build where the reviewer could only 'view' their submission.

1) app/views/student_review/_responses.html.erb - added a condition such that a user is allowed to update a review even after submission.

<% if (last_response_round == current_round)%>
  <td>
  <% if (!@latest_response.is_submitted) %>
    <%= link_to "Edit", {:controller => 'response', :action => 'edit', :id => @latest_response.id} %>
  <%end%>
  </td>
<% else %>
  <td>
  <%= link_to "Update", {:controller => 'response', :action => 'new', :id => map.map_id} %>
  </td>
<% end %>

<% if (last_response_round == current_round)%>
  <td>
  <% if (!@latest_response.is_submitted) %>
    <%= link_to "Edit", {:controller => 'response', :action => 'edit', :id => @latest_response.id} %>
  <% else %>
    <%= link_to "Update", {:controller => 'response', :action => 'edit', :id => @latest_response.id} %>
  <%end%>
  </td>
<% else %>
  <td>
  <%= link_to "Update", {:controller => 'response', :action => 'new', :id => map.map_id} %>
  </td>
<% end %>

2) app/views/response/response.html.erb - update confirmation message

 
if(!confirm('Once a review has been submitted, you cannot edit it again')){
  e.preventDefault();
  e.stopPropagation();
  return;
}else{
  jQuery('#isSubmit').val('Yes');
  window.location.href= "../../../student_review/list?id=<%= @map.reviewer.id %>";
}

if(!confirm('Are you sure you would like to Submit?')){
  e.preventDefault();
  e.stopPropagation();
  return;
}else{
  jQuery('#isSubmit').val('Yes');
  window.location.href= "../../../student_review/list?id=<%= @map.reviewer.id %>";
}

3) app/controllers/response_controller.rb - commented out line 14 that returned false if the response was submitted because we no longer need the value of this attribute

when 'edit' # If response has been submitted, no further editing allowed
  return false if response.is_submitted
  return current_user_id?(user_id)
  # Deny access to anyone except reviewer & author's team

 
when 'edit' # If response has been submitted, no further editing allowed
  #return false if response.is_submitted
  return current_user_id?(user_id)
  # Deny access to anyone except reviewer & author's team

• Required Updates to spec/controllers/response_controller_spec.rb

- The test to verify the restricted nature of editing a submitted review was removed as this restriction no longer applies. This testcase can be found on line 33 of the rspec test file.

Problem 3 : Issue # 1029

• As no attempt to reproduce this issue were not successful, After discussing with Dr.Gehringer we realized that issue is not reproducible. Hence marking this issue as not reproducible

Problem 4 : Issue # 972

• As no attempts to reproduce the stated issue were successful, no solution to this problem can be proposed. In the Github Bug Report there is a reference to a similar Github Bug Report that encompasses this same issue that has been marked closed. For the purposed of this project, testing of this believed fix in the update_feedback_assignment_form_attributes function of assignments_controller.rb will be tested to ensure correctness. This test has been added to the features spec folder of Expertiza as expertiza/spec/features/assignment_round_adjustment_spec.rb. A video capture of this feature test has been included in the Related Links section of this wiki.

Problem 5 : Issue # 1142

• As no attempts to reproduce the stated issue were successful, no solution to this problem can be proposed. Upon further exploration of the view file view_team.html.erb, it was found that a patch was introduced to fix this bug by assuring that the html was rendered safely.Because this bug was already fixed prior to the distribution of our project, no further action is needed to address the issue. This action has been confirmed by Dr. Gehringer on Novemebr 27,2018.

Testing Plan

Testing Plan will be implemented upon the discovery of fixed bugs described in the project document as a means of verifying the fix.

To ensure correctness of bug fixes following test plan was implemented:

• Issue 1:

- No Testing Required - Bug not Reproducible

• Issue 2:

- Verify that upon submission of the review, the reviewer maintains an option to "update" the review until review deadline ends

• Issue 3:

- No Testing Required - Bug not Reproducible

• For Issue 4:

- Verify assignment holds the number of review rounds value

- Verify that the increasing number of review rounds after creating the assignment saves successfully and the new value is retained

- Verify that the decreasing number of review rounds after creating the assignment saves successfully and the new value is retained

• Issue 5:

- Verify that when user views score review comments rendered as processed HTML only(there should not be any rendered HTML tags)

• The setup for testing both Issues 2 & 5 is complex as there are many necessary associations and procedures that are required to achieve the appropriate testing state. Assuming that a sufficient setup is achieved, a feature test for Issue 2 would simulate logging in as a student, visiting the Others Work page and submitting a review submission which would allow for the verification that an option to update his or her review is available, upon returning to the student assignment page. Further, a feature test for Issue 5 would simulate logging in as a student, visiting the Your Scores page, where he or she would be able to hover over a response link to validate the comment rendering.
• The follow is a list of several suitable models and possible relationships:

- Models:

• Course
• Assignment
• Assignment_Mapping
• Response Questionnaire
• Review_Mapping
• Response
• Team
• Topic
• Participants - reviewer & reviewee
• Submission

- Relationships:

• A course must contain participants and assignments
• An assignment must be associated with participants and response questionnaires
• A response questionnaire must be associated with participants and responses
• It is also required that a reviewee submit an assignment to then be reviewed by a reviewer.

References & Related Links

• Accompanying Videos for Issues and Testing:

-[Issue 1]: Replication Attempts <-- insert video link here

-[Issue 2]: Run of Manual Test <-- insert video link here

-[Issue 3]: Replication Attempts <-- insert video link here

-Issue 4: Run of Feature Test

-Issue 5: Run of Manual Test

• Project Links

-Project Screencast <-- insert video link here

-Pull request

-Project Document

-Tutorial videos on creating assignments

-GitHub: Forked Repository