CSC/ECE 517 Fall 2018 E1864: Issues related to Reviewing

Project Introduction & Motivation

Expertiza is an open source web-based peer review system developed and maintained by students and faculty members at North Carolina State University. It has been developed as Open Source Software using Ruby on Rails framework. Expertiza allows students to form teams, review each other's work such as assignments and projects. It provides the role based access so that instructor can impose restriction what students can view/edit.

The primary motivation behind this project is to give the students to review the projects of their peers effortlessly and fix all the issues of the previous implementation.

Description of Project

Expertiza Project : E1864 ‘Issues Related to Reviewing' is intended to fix five known issues in the current Expertiza Version related to reviewing student work. Issues are centered around peer reviewing and assignment feature calibration. The motivation behind this project is to identify and resolve each bug in an attempt to improve the current system version. The project has been divided into required fixes. Four of the five problems are presented in the flow diagram below. Problem 5 (Issue # 1142) is not present as the problem is simply associated with the coding of a view file and not directly connected to the flow of students and reviews. Additional discussion on proposed approaches to meet the requirements are explained in subsequent sections.

Figure 1.Application flow diagram

-----------------------------------------------

UML diagram for reviw

Figure 2.UML diagram

-----------------------------------------------

Actors:

Instructor: This actor is responsible for creating assignments and adding students to the assignment.

Student: This actor is responsible for submitting, reviewing other students work.

Database:

The database where all the data of Expertiza is getting stored.

Problem Statements & Descriptions

For this project, we have been asked to fix the follow bugs that have been found in the current Expertiza system :

Problem 1 : Issue #1093

• When a student requests a project to review, duplicate reviews appear in the list of available projects.
• Example : The project name for which the review was performed earlier should not appear for the next list of reviews when a student requests a review to perform. This could lead to students reviewing the same project more than once and the requirement is to review different projects each time.

Figure 3.Duplicate topic issue

-----------------------------------------------

Steps to reproduce the Issue #1093: Figure 1 gives the holistic view on stages in which we see the issues. In this part, we will discuss the exact steps that need to be followed to reproduce the bugs.

• Login as Instructor Create new assignment and associate to the course as shown in Figure 4
• Add a participant to the assignment. there are two ways to do this in bulk first way is importing all the course participants to assignment . second approach is creating a file containing student details and import them to expertiza
• Create topics and teams in expertiza either entering them one by one or from the file
• In actual implementation topic association to the teams involves bidding process but for ease of use, we let instructor assign the topic (Figure 5). set min and max number of reviews allowed as shown in
• Set the rubrics and number of review rounds (min one round is needed) (Figure 6)
• Review assignment should be automatic not instructor controlled(scope of the stated issue is here) as shown in Figure 7
• Set the deadlines for submission and review (shown in Figure 8)
• Login as a student and submit work(this needs to be done for each team), request for review
• If the issue persist system assign the same topic to review multiple times(as shown in Figure2).difficulty with this problem is that its reproducibility factor which is very low.

Figure 4.Assignment creation

-----------------------------------------------

Figure 5.Topic assosciation to teams

---------------------------------------------------

Figure 6.Rubric settings

-----------------------------------------------

Figure 7.Review settings

-----------------------------------------------

Figure 8.Deadline settings

-----------------------------------------------

Problem 2 : Issue # 1097

• When a project author resubmits their assignment, the reviewers should be able to update, not edit, their reviews. This should hold even if those reviews have already been submitted. The current system does not allow reviewers to update their reviews.
• Example : When the reviewer submits a review for a project, the reviewer cannot edit/ update again. Before submitting it says "Once submitted, the review cannot be changed again".

Real life example : When a reviewer submits the review 1 a few minutes after the deadline, the rubric for review 2 occurs. So the review 1 actually becomes review 2 and the reviewer was not able to update this review. In situations like these, update the submitted review becomes useful.

Steps to reproduce the issue:

• Login as Instructor Create new assignment and associate to the course as shown in Figure 4
• Add a participant to the assignment. there are two ways to do this in bulk first way is importing all the course participants to assignment . second approach is creating a file containing student details and import them to expertiza
• Create topics and teams in expertiza either entering them one by one or from the file
• In actual implementation topic association to the teams involves bidding process but for ease of use, we let instructor assign the topic (Figure 5). Set min and max number of reviews allowed as shown in (Figure 6)
• Set the rubrics and number of review rounds (min one round is needed)
• Review assignment should be automatic not instructor controlled(scope of the stated issue is here) as shown in Figure 7
• Set the deadlines for submission and review (shown in Figure 8)
• Login as a student and submit work(this needs to be done for each team), request for review.
• Change the deadlines for the review to a few days after the present date and check if you are able to update the review. The picture shows deadline to submit is after 14 days. But the reviews are only able to be viewed.

Possible solution is to let the reviewers update the review until the deadline and possibly remove the message " Once submitted , it cannot be changed again.

Problem 3 : Issue # 1029

• When a reviewer tries to review a project, sometimes he/she is shown that the project has not yet been submitted when in fact it has.
• Example : An author of a project has submitted his/her assignment to Expertiza. When a reviewer tries to review the work, sometimes it happens that the system shows the reviewer that the work has not been submitted yet. The reviewer should have been shown the work of the author to review.

Steps to reproduce the Issue #1029:

• steps to reproduce this issue is similar to issue #1093, only the difference is once reviewer requests the assignment to review system throws a message saying there is no work has been submitted even though there is actually submitted work. This issue is not always reproducible

Problem 4 : Issue # 972

• When updating the number of review rounds for an existing project, and error is generated and this action fails.
• Example : An instructor attempts to edit an existing assignment and change the number of review rounds. The reassignment causes an error in the system and this prevents the project update to occur.

Steps to reproduce the Issue #972 :

• We will choose one of the existing assignments in expertiza and try to increase the number of review rounds from "x" to "y"(x<y). when we try to save the assignment we get the error message as shwon in the figure below also number of reviews revrts back to "x".

Figure 9.Error message in Issue #972

-----------------------------------------------

Problem 5 : Issue # 1142

• “Show calibration results” page shows raw HTML when it should displayed as processed HTML.
• Example : The Calibration Results page has several instances of raw HTML. The intended result was for the raw HTML to be rendered properly while displaying on the page.

Files Modified

• views/review_mapping/_calibration_report.html.erb - this file must be altered to handle the bug fix for problem #5.
• app/models/review_assignment.rb

Solution Design

Problem 1 : Issue # 1093

• If the issue reproducible consistantly we need to fix reject_previously_reviewed_submissions(contributor_set, reviewer) method in review_assignment.rb model file. This method is responsible for the seperating the current review topic and previously reviewed topics. solution flow diagram is shown below

Figure 10.Proposed solution to Issue #1093

------------------------------------------------------------

Problem 2 : Issue # 1097

• [An appropriate solution design and implementation will be suggested in the final design document submission]

Problem 3 : Issue # 1029

• Solution to this issue involves identifying students work as correctly submitted or not submitted so that system will always know that submission status this identification should be available to the method which assigns topics to the reviewer.

Problem 4 : Issue # 972

• The proposed solution to this issue is, we need to enable edit operation for the number of review rounds option provided new number is greater than the previous number. The reason behind this is if we allow the instructor to reduce the number of rounds we lose all review data and grades associated with them. also, we need to make sure that increasing the number of rounds should populate same number "review round" attributes (such as deadline submission rights)

Problem 5 : Issue # 1142

• In this case, we need to identify the mistakes in HTML file and then correct them. final expected outcome is to have processed HTML file.

Testing Plan

Testing Plan will be implemented upon the discovery of fixed bugs described in the project document as a means of verifying the fix.

To ensure correctness of bug fixes following test plan was implemented:

• Obtain the appropriate testing environment via a provided virtual box image and RSPEC testing framework.
• Acquire a deeper understanding of the necessary models and views, their functionalities and dependencies.
• Create factories and doubles to assist in the testing of these models and views methods.
• Mock message passing and expected outcomes.
• Apply generated helper objects and mocks to achieve high test coverage of methods within the models and views that were updated.

References

• Project Document
• Tutorial videos on creating assignments