E1839 Review Requirements and Thresholds

Description

This wiki page is the description of the E1839 OSS Writeup project - Review Requirements and Thresholds - for Fall 2018, CSC/ECE 517.

Introduction

Expertiza is an open source project based on Ruby on Rails framework. The Expertiza project is a software that creates reusable learning objects through peer review. It is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in some courses at NC State University and by professors at several other universities. It supports team projects, and the submission of almost any document type, including URLs and wiki pages. Expertiza enables the instructor to create new and customize existing assignments. It also enables the instructor to create a list of topics the students can sign up for as part of a project. Students can form teams in Expertiza to work on various projects and assignments. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Description

In Expertiza, there are two ways of assigning reviews to reviewers:

1. The instructor decides who reviews whom (“instructor-selected”)

2. “Auto-selected,” in which case reviews are not assigned until a student seeks to choose something to review.

This section is mostly about auto-selected reviewing. A reviewer is ordinarily allowed to choose only among work that has received the minimum number of reviews so far. For example, if all submissions have received at least one review, the next person who tries to review is allowed to choose only a submission that has only one review so far, unless there are no more such submissions, in which case the reviewer chooses from among submissions that have 2 reviews (if there are any), or the minimum number of reviews, whatever that is.

For assignments that have topics, this severely constrains a reviewer’s choice of what topic to review on. (For assignments that don’t have topics, a reviewer doesn’t have anything to choose; (s)he just gets one of the submissions that have the fewest reviews so far.) To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far.

There's another issue. Suppose that the minimum number of reviews for any submission is 2, but that the person has reviewed all the submissions that have only 2 reviews. Then they are not allowed to review at all (unless k > 0). That’s wrong; they should always be allowed to review the work with the least reviews that I have not already reviewed. And that should generalize to situations in which k > 0.

Another issue is that there is no way for Expertiza to tell a reviewer how many reviews are required. In the case of instructor-selected reviewing, that’s not a problem, but for auto-selected reviewing, there is no way to specify how many reviews are required (or even how many are allowed, in case students are allowed to do extra reviews).

Implementation

Drawbacks and Solutions

Issue #402 : To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0).

• Solution: This issue has been fixed previously. The reviewer will get assigned a submission even if it has fulfilled the required number of reviews, to ensure that the reviewer always receives a new submission to review as long as it is not his own.

Issue #228 : To Allow a reviewer who has already reviewed all submissions that have the minimum number of reviews m to review any submission that has ≤ m+k reviews.

• Solution: This issue is similar to issue 402 and has been already fixed. If the student has already reviewed the least number of reviewed assignment, then he/she will get the next least reviewed assignment. Refer to expertiza/app/models/review_assignment.rb file

 def candidate_topics_to_review(reviewer)
   return nil if sign_up_topics.empty? # This is not a topic assignment
   # Initialize contributor set with all teams participating in this assignment
   contributor_set = Array.new(contributors)
   # Reject contributors that have not selected a topic, or have no submissions
   contributor_set = reject_by_no_topic_selection_or_no_submission(contributor_set)
   # Reject contributions of topics whose deadline has passed, or which are not reviewable in the current stage
   contributor_set = reject_by_deadline(contributor_set)
   # Filter submissions already reviewed by reviewer
   contributor_set = reject_previously_reviewed_submissions(contributor_set, reviewer)
   # Filter submission by reviewer him/her self
   contributor_set = reject_own_submission(contributor_set, reviewer)
   # Filter the contributors with the least number of reviews
   # (using the fact that each contributor is associated with a topic)
   contributor_set = reject_by_least_reviewed(contributor_set)
   contributor_set = reject_by_max_reviews_per_submission(contributor_set)
   # if this assignment does not allow reviewer to review other artifacts on the same topic,
   # remove those teams from candidate list.
   contributor_set = reject_by_same_topic(contributor_set, reviewer) unless self.can_review_same_topic?
   # Add topics for all remaining submissions to a list of available topics for review
   candidate_topics = Set.new
   contributor_set.each do |contributor|
     candidate_topics.add(signed_up_topic(contributor))
   end
   candidate_topics
 end

This snippet of code will do the following features:

1. Return nil if the array of choices of review is empty

2. Reject contributors that have not selected a topic, or have no submissions

3. Reject contributions of topics whose deadline has passed, or which are not reviewable in the current stage

4. Filter submissions already reviewed by a reviewer

5. Filter the contributors with the least number of reviews

6. If this assignment does not allow the reviewer to review other artifacts on the same topic, remove those teams

7. Add topics for all remaining submissions to a list of available topics for review

Issue #417: Implement a num_reviews_required (and num_reviews_allowed) field in the assignments table to say how many reviews per reviewer are required, and how many are allowed (default should be # allowed = # req’d.). Make it settable from the Review Strategy tab and viewable when a student clicks on “Others’ work”. Do the same for metareviews (“reviews of reviews”), which students can also be assigned to do for an Expertiza assignment.

• Solution: This issue has been resolved by adding the following piece of code in the file /expertiza/app/views/student_review/list.html.erb

   Number of reviews left: <%= @assignment.num_reviews_allowed - @num_reviews_completed %>

    Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Design Pattern

1. MVC

The project is implemented in Ruby on Rails that uses MVC architecture. It separates an application’s data model, user interface, and control logic into three distinct components (model, view and controller, respectively).

2. Dry Principle

We are trying to reuse the existing functionalities in Expertiza, thus avoiding code duplication. Whenever possible, code modification based on the existing classes, controllers, or tables will be done instead of creating the new one.

Team Information

Rayan Dasoriya (rdasori@ncsu.edu)

Prakshatkumar Shah (pmshah2@ncsu.edu)

Ravindersingh Rajpal (rkrajpal@ncsu.edu)

Mentor: Ed Gehringer (efg@ncsu.edu)