CSC/ECE 517 Fall 2018 E1864: Issues related to Reviewing

Project Introduction & Motivation

Expertiza is an open source web-based peer review system developed and maintained by students and faculty members at North Carolina State University. It has been developed as Open Source Software using Ruby on Rails framework. Expertiza allows students to form teams and review each other's work, such as assignments and projects. It provides role based access so that an instructor can impose restrictions allowing variations in what students can view/edit.

The primary motivation behind this project is to fix assigned issues and enable students' ability to review projects of their peers effortlessly.

Description of Project

Expertiza Project : E1864 ‘Issues Related to Reviewing' is intended to fix five known issues in the current Expertiza System related to reviewing student work. Issues are centered around peer reviewing and assignment feature calibration. The motivation behind this project is to identify and resolve each bug in an attempt to improve the current system version. The project has been divided into required fixes. Four of the five problems are presented in Figure 1 - Application Flow Diagram below. This image is a general representation describing when each of the issues is encountered during a user session. Problem 5 (Issue # 1142) is not present in this figure as the problem is simply associated with the coding of a view file and not directly connected to the flow of students and reviews. Additional discussion on proposed approaches to meet the requirements are explained in subsequent sections.

Figure 1 - Application Flow Diagram

-----------------------------------------------

UML diagram for "Review"

Actors:

Instructor - This actor is responsible for creating assignments and adding students to the assignment.
Student - This actor is responsible for submitting, reviewing other students work.

Database:

The database where all the data of Expertiza is getting stored.

Problem Statements & Descriptions

For this project, we have been asked to fix the follow bugs that have been found in the current Expertiza system :

Problem 1 : Issue #1093

When a student requests a project to review, duplicate reviews appear in the list of available projects.
Example : The project name for which the review was performed earlier should not appear for the next list of reviews when a student requests a review to perform. This could lead to students reviewing the same project more than once and the requirement is to review different projects each time.

Steps to reproduce the Issue #1093: Figure 1 gives the holistic view on stages in which we see the issues. In this part, we will discuss the exact steps that need to be followed to reproduce the bugs.

Login as Instructor Create new assignment and associate to the course as shown in Figure 4
Add a participant to the assignment. there are two ways to do this in bulk first way is importing all the course participants to assignment . second approach is creating a file containing student details and import them to expertiza
Create topics and teams in expertiza either entering them one by one or from the file
In actual implementation topic association to the teams involves bidding process but for ease of use, we let instructor assign the topic (Figure 5). set min and max number of reviews allowed as shown in
Set the rubrics and number of review rounds (min one round is needed) (Figure 6)
Review assignment should be automatic not instructor controlled(scope of the stated issue is here) as shown in Figure 7
Set the deadlines for submission and review (shown in Figure 8)
Login as a student and submit work(this needs to be done for each team), request for review
If the issue persist system assign the same topic to review multiple times(as shown in Figure2).difficulty with this problem is that its reproducibility factor which is very low.

Problem 2 : Issue # 1097

When a project author resubmits their assignment, the reviewers should be able to update, not edit, their reviews. This should hold even if those reviews have already been submitted. The current system does not allow reviewers to update their reviews.
Example : When the reviewer submits a review for a project, the reviewer cannot edit/ update his review again. Before submitting it says "Once submitted, the review cannot be changed again". After clicking yes the reviewer is only able to view his reviews and not update even before the deadline.

This situation can occur in the following scenario. When a reviewer submits the review-1 a few minutes after the deadline, the rubric for review 2 occurs. So the review done for the first round actually becomes review for round 2. Hence the reviewer would not be able to update this review. In situations like these, updating the submitted review becomes useful.

Steps to reproduce the issue:

Initial steps to reproduce this problem is same as the issue #1093
Change the deadlines for the review to a few days after the present date and check if you are able to update the review. The picture shows deadline to submit is after 14 days. But the reviews are only able to be viewed.

Problem 3 : Issue # 1029

When a reviewer tries to review a project, sometimes he/she is shown that the project has not yet been submitted when in fact it has.
Example : An author of a project has submitted his/her assignment to Expertiza. When a reviewer tries to review the work, sometimes it happens that the system shows the reviewer that the work has not been submitted yet. The reviewer should have been shown the work of the author to review.

Steps to reproduce the Issue #1029:

steps to reproduce this issue is similar to issue #1093, only the difference is once reviewer requests the assignment to review system throws a message saying there is no work has been submitted even though there is actually submitted work. This issue is not always reproducible

Problem 4 : Issue # 972

When updating the number of review rounds for an existing project, and error is generated and this action fails.
Example : An instructor attempts to edit an existing assignment and change the number of review rounds. The reassignment causes an error in the system and this prevents the project update to occur.

Steps to reproduce the Issue #972 :

The Github Bug Report states that "You can change the # of rounds from 1 to 2. You can then change the # of rounds back to 1, and the assignment will appear to be saved with one round of review. But if you edit it again, the 2 rounds will still be present." Several attempts to reproduce this issue were made. Below are two of such test cases.

- Test Case 1 : Increment beyond current number of rounds then decrements back to the original assigned number of rounds for an existing assignment.

Step 1 - Log in as instructor6
Step 2 - Go to the Manage Courses page and select - CSC 517 Spring 2015
Step 3 - Select the Edit option for - Final projects
Step 4 - Select the due dates tab as seen in figure 12
Step 5a - Change the number of rounds from 2 to 3
Step 6 - Press the Set option
Step 7a - Select Save

- One can see from Figure 13 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

Step 5b - Change the number of rounds from 3 back to 2
Step 6 - Press the Set option
Step 7b - Select Save

- One can see from Figure 14 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

- Test Case 2 : Decrement below current number of rounds an existing assignment with submissions.

Step 1 - Log in as instructor6
Step 2 - Go to the Manage Courses page and select - CSC 517 Spring 2016
Step 3 - Select the Edit option for - Program 1: Class portal
Step 4 - Select the due dates tab as seen in figure 12
Step 5a - Change the number of rounds from 2 to 1
Step 6 - Press the Set option
Step 7 - Confirm the reduction below the current number of rounds
Step 8 - Press OK to exit the notification popup

Addition steps 6-8 can be seen in Figure 15 below.

Step 7a - Select Save

- One can see from Figure 16 that the change in round number is successfully updated and reflected in the tab. To ensure that the assignment was saved correctly, logging out of the user session, steps 1 - 4 were repeated and the previous changes were found to persist.

Several similar tests were conducted in an attempt to reproduce this issue; however, none were successful.
Unstated Issue: there is an issue that arises from the reduction of rounds for assignments that currently contain submissions. After the udpate, When an current student attempts to go and review his/her scores, this causes a NoMethodError in Grades#view_team. A bug should be created for this issue for another round of project issues.

Problem 5 : Issue # 1142

“Summary Result Assignment” page shows raw HTML when it should displayed as processed HTML.
Example : As seen in Figure 17 below, the result of hovering over a review result displays the message in raw HTML. The intended result is for review comment to be rendered properly while displaying.

Figure 17 was taken from the actual deployed version of Expertiza through a person student account.

-Test Case 1 : Hover for Reviewer Comment

Step 1 - Log in as instructor6
Step 2 - Select Manage and Impersonate User
Step 3 - Impersonate student6370
Step 4 - Selec OSS project/Writing assignment 2
Step 5 - Hover over any underlined review score

Figure 18 was taken from the development version of Expertiza through the test case steps mentioned above.

After testing and research into this issue, it has been determined that the issue has been adjusted in the development image and has not yet been merged with the deployed version.

Files to be Modified

app/views/student_review/_responses.html.erb
app/views/response/response.html.erb
app/controllers/response_controller.rb
spec/controllers/response_controller_spec.rb
spec/features/assignment_round_adjustment.rb

Solution Design

Problem 1 : Issue # 1093

Several attempts have been made to reproduce this problem. We tested with both old and latest database but the issue did not reproduce.
We also tried with creating a new assignment, adding participants to the assignment, assign reviews to the student then logging in as student and request for a topic to review even with this issue did not reproduce
After discussing with mentor and Dr.Gehringer we are marking this issue as not reproducible.

Problem 2 : Issue # 1097

To solve this problem we edited code in app/views/student_review/_responses.html.erb, app/views/response/response.html.erb, and app/controllers/response_controller.rb to enable a reviewer to update the review until the round deadline. We also adapted the confirmation message "Once submitted, it cannot be changed again" before the deadline to appropriate reflect this edit.
The following figure shows you those new changes to the view.

-The reviewer now has the option to both view and update their submitted reviews multiple times before the round deadline This is in contrast to the previous build where the reviewer could only 'view' their submission.

1) app/views/student_review/_responses.html.erb - added a condition such that a user is allowed to update a review even after submission.

<% if (last_response_round == current_round)%>
  <td>
  <% if (!@latest_response.is_submitted) %>
    <%= link_to "Edit", {:controller => 'response', :action => 'edit', :id => @latest_response.id} %>
  <%end%>
  </td>
<% else %>
  <td>
  <%= link_to "Update", {:controller => 'response', :action => 'new', :id => map.map_id} %>
  </td>
<% end %>

<% if (last_response_round == current_round)%>
  <td>
  <% if (!@latest_response.is_submitted) %>
    <%= link_to "Edit", {:controller => 'response', :action => 'edit', :id => @latest_response.id} %>
  <% else %>
    <%= link_to "Update", {:controller => 'response', :action => 'edit', :id => @latest_response.id} %>
  <%end%>
  </td>
<% else %>
  <td>
  <%= link_to "Update", {:controller => 'response', :action => 'new', :id => map.map_id} %>
  </td>
<% end %>

2) app/views/response/response.html.erb - update confirmation message

 
if(!confirm('Once a review has been submitted, you cannot edit it again')){
  e.preventDefault();
  e.stopPropagation();
  return;
}else{
  jQuery('#isSubmit').val('Yes');
  window.location.href= "../../../student_review/list?id=<%= @map.reviewer.id %>";
}

if(!confirm('Are you sure you would like to Submit?')){
  e.preventDefault();
  e.stopPropagation();
  return;
}else{
  jQuery('#isSubmit').val('Yes');
  window.location.href= "../../../student_review/list?id=<%= @map.reviewer.id %>";
}

3) app/controllers/response_controller.rb - commented out line 14 that returned false if the response was submitted because we no longer need the value of this attribute

when 'edit' # If response has been submitted, no further editing allowed
  return false if response.is_submitted
  return current_user_id?(user_id)
  # Deny access to anyone except reviewer & author's team

 
when 'edit' # If response has been submitted, no further editing allowed
  #return false if response.is_submitted
  return current_user_id?(user_id)
  # Deny access to anyone except reviewer & author's team

Required Updates to spec/controllers/response_controller_spec.rb

- The test to verify the restricted nature of editing a submitted review was removed as this restriction no longer applies. This testcase can be found on line 33 of the rspec test file.

Problem 3 : Issue # 1029

As all attempts to reproduce this issue were not successful, and after discussing with Dr.Gehringer, we realized that this issue is not reproducible. Hence marking this issue as not reproducible.

Problem 4 : Issue # 972

As no attempts to reproduce the stated issue were successful, no solution to this problem can be proposed. In the Github Bug Report there is a reference to a similar Github Bug Report that encompasses this same issue that has been marked closed. For the purposed of this project, testing of this believed fix in the update_feedback_assignment_form_attributes function of assignments_controller.rb has been tested to ensure correctness. This test has been added to the features spec folder of Expertiza as expertiza/spec/features/assignment_round_adjustment_spec.rb. A video capture of this feature test has been included in the References & Related Links section of this wiki.
Feature Test Explanation : three states have been tested to ensure correctness.

1) Saving of original number of rounds within a project to confirm the data was captured.

2) Updated the number of rounds by increasing the current value and confirming that this data was captured upon saving.

3) Updated the number of rounds by decreasing the current value and confirming that this data was captured upon saving.

Problem 5 : Issue # 1142

As no attempts to reproduce the stated issue were successful, no solution to this problem can be proposed. Upon further exploration of the view file view_team.html.erb, it was found that a patch was introduced to fix this bug by assuring that the html was rendered safely.Because this bug was already fixed prior to the distribution of our project, no further action is needed to address the issue. A video capture of this manual test has been included in the References & Related Links section of this wiki.

Testing Plan

Testing Plan implemented upon the discovery of fixed bugs described in the project document as a means of verifying the fix.

To ensure correctness of bug fixes following test plan was implemented:

Problem 1 : Issue #1093

- No Testing Required - Bug not Reproducible

Problem 2 : Issue #1097

- Verify that upon submission of the review, the reviewer maintains an option to "update" the review until review deadline ends

Problem 3 : Issue #1029

- No Testing Required - Bug not Reproducible

Problem 4 : Issue #927

- Verify assignment holds the number of review rounds value

- Verify that the increasing number of review rounds after creating the assignment saves successfully and the new value is retained

- Verify that the decreasing number of review rounds after creating the assignment saves successfully and the new value is retained

Problem 5 : Issue #1142

- Verify that when user views score review comments rendered as processed HTML only(there should not be any rendered HTML tags)

The setup for testing both problems 2 & 5 is complex as there are many necessary associations and procedures that are required to achieve the appropriate testing state. Assuming that a sufficient setup is achieved, a feature test for Issue 2 would simulate logging in as a student, visiting the Others Work page and submitting a review submission which would allow for the verification that an option to update his or her review is available, upon returning to the student assignment page. Further, a feature test for Issue 5 would simulate logging in as a student, visiting the Your Scores page, where he or she would be able to hover over a response link to validate the comment rendering.
The follow is a list of several suitable models and possible relationships:

- Models:

Course
Assignment
Assignment_Mapping
Response Questionnaire
Review_Mapping
Response
Team
Topic
Participants - reviewer & reviewee
Submission

- Relationships:

A course must contain participants and assignments
An assignment must be associated with participants and response questionnaires
A response questionnaire must be associated with participants and responses
It is also required that a reviewee submit an assignment to then be reviewed by a reviewer.

References & Related Links

Accompanying Videos for Issues and Testing:

-Problem 2 : Issue #1097: Run of Manual Test

-Problem 4 : Issue #927: Run of Feature Test

-Problem 5 : Issue #1142: Run of Manual Test

Project Links

-Project Screencast

-Pull request

-Project Document

-Tutorial videos on creating assignments

-GitHub: Forked Repository

-Screencast Presentation Slides

CSC/ECE 517 Fall 2018 E1864: Issues related to Reviewing

Contents

Project Introduction & Motivation

Description of Project

UML diagram for "Review"

Problem Statements & Descriptions

Problem 1 : Issue #1093

Problem 2 : Issue # 1097

Problem 3 : Issue # 1029

Problem 4 : Issue # 972

Problem 5 : Issue # 1142

Files to be Modified

Solution Design

Problem 1 : Issue # 1093

Problem 2 : Issue # 1097

Problem 3 : Issue # 1029

Problem 4 : Issue # 972

Problem 5 : Issue # 1142

Testing Plan

References & Related Links

Navigation menu

CSC/ECE 517 Fall 2018 E1864: Issues related to Reviewing

Project Introduction & Motivation

Description of Project

UML diagram for "Review"

Problem Statements & Descriptions

Problem 1 : Issue #1093

Problem 2 : Issue # 1097

Problem 3 : Issue # 1029

Problem 4 : Issue # 972

Problem 5 : Issue # 1142

Files to be Modified

Solution Design

Problem 1 : Issue # 1093

Problem 2 : Issue # 1097

Problem 3 : Issue # 1029

Problem 4 : Issue # 972

Problem 5 : Issue # 1142

Testing Plan

References & Related Links

Navigation menu

Search