E1839 Review Requirements and Thresholds

Description

This wiki page is the description of the E1839 OSS Writeup Project - Review Requirements and Thresholds - for Fall 2018, CSC/ECE 517.

Introduction

Expertiza is an open source project based on Ruby on Rails framework. The Expertiza project is a software that creates reusable learning objects through peer review. It is a web application where students can submit and peer-review learning objects (articles, code, websites, etc). It is used in some courses at NC State University and by professors at several other universities. It supports team projects, and the submission of almost any document type, including URLs and wiki pages. Expertiza enables the instructor to create new and customize existing assignments. It also enables the instructor to create a list of topics the students can sign up for as part of a project. Students can form teams in Expertiza to work on various projects and assignments. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Description

In Expertiza, there are two ways of assigning reviews to reviewers:

1. Instructor-selected: The instructor decides who reviews whom.

2. Auto-selected: Case reviews are not assigned until a student seeks to choose something to review.

This section is mostly about auto-selected reviewing. A reviewer is ordinarily allowed to choose only among work that has received the minimum number of reviews so far. For example, if all submissions have received at least one review, the next person who tries to review is allowed to choose only a submission that has only one review so far, unless there are no more such submissions, in which case the reviewer chooses from among submissions that have 2 reviews (if there are any), or the minimum number of reviews, whatever that is.

For assignments that have topics, this severely constrains a reviewer’s choice of what topic to review on. (For assignments that don’t have topics, a reviewer doesn’t have anything to choose; (s)he just gets one of the submissions that have the fewest reviews so far.) To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far.

There's another issue. Suppose that the minimum number of reviews for any submission is 2, but that the person has reviewed all the submissions that have only 2 reviews. Then they are not allowed to review at all (unless k > 0). That’s wrong; they should always be allowed to review the work with the least reviews that I have not already reviewed. And that should generalize to situations in which k > 0.

Another issue is that there is no way for Expertiza to tell a reviewer how many reviews are required. In the case of instructor-selected reviewing, that’s not a problem, but for auto-selected reviewing, there is no way to specify how many reviews are required (or even how many are allowed, in case students are allowed to do extra reviews).

Implementation

Issue #402 : To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0).

• Solution: This issue has been fixed previously. The reviewer will get assigned a submission even if it has fulfilled the required number of reviews, to ensure that the reviewer always receives a new submission to review as long as it is not his own.

Issue #228 : To Allow a reviewer who has already reviewed all submissions that have the minimum number of reviews m to review any submission that has ≤ m+k reviews.

• Solution: This issue is similar to issue 402 and has been already fixed. If the student has already reviewed the least number of reviewed assignment, then he/she will get the next least reviewed assignment. Refer to expertiza/app/models/review_assignment.rb file

A combined explanation of issue 402 and 228 has been discussed below with the help of existing code.

  def candidate_topics_to_review(reviewer)
    return nil if sign_up_topics.empty? # This is not a topic assignment

    # Initialize contributor set with all teams participating in this assignment
    contributor_set = Array.new(contributors)

    # Reject contributors that have not selected a topic, or have no submissions
    contributor_set = reject_by_no_topic_selection_or_no_submission(contributor_set)

    # Reject contributions of topics whose deadline has passed, or which are not reviewable in the current stage
    contributor_set = reject_by_deadline(contributor_set)

    # Filter submissions already reviewed by reviewer
    contributor_set = reject_previously_reviewed_submissions(contributor_set, reviewer)

    # Filter submission by reviewer him/her self
    contributor_set = reject_own_submission(contributor_set, reviewer)

    # Filter the contributors with the least number of reviews
    # (using the fact that each contributor is associated with a topic)
    contributor_set = reject_by_least_reviewed(contributor_set)
    contributor_set = reject_by_max_reviews_per_submission(contributor_set)

    # if this assignment does not allow reviewer to review other artifacts on the same topic,
    # remove those teams from candidate list.
    contributor_set = reject_by_same_topic(contributor_set, reviewer) unless self.can_review_same_topic?

    # Add topics for all remaining submissions to a list of available topics for review
    candidate_topics = Set.new
    contributor_set.each do |contributor|
      candidate_topics.add(signed_up_topic(contributor))
    end
    candidate_topics
  end

This snippet of code will do the following features:

1. Return nil if the array of choices of review is empty

2. Reject contributors that have not selected a topic, or have no submissions

3. Reject contributions of topics whose deadline has passed, or which are not reviewable in the current stage

4. Filter submissions already reviewed by a reviewer

5. Filter the contributors with the least number of reviews

6. If this assignment does not allow the reviewer to review other artifacts on the same topic, remove those teams

7. Add topics for all remaining submissions to a list of available topics for review

Issue #417: Implement a num_reviews_required (and num_reviews_allowed) field in the assignments table to say how many reviews per reviewer are required, and how many are allowed (default should be # allowed = # required.). Make it settable from the Review Strategy tab and viewable when a student clicks on “Others’ work”. Do the same for meta-reviews (“reviews of reviews”), which students can also be assigned to do for an Expertiza assignment.

• Solution: This issue has been resolved by adding the following piece of code in the file /expertiza/app/views/student_review/list.html.erb

   <h4> Number of reviews left: <%= @assignment.num_reviews_allowed - @num_reviews_completed %></h4>

   <h4> Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %></h4>

Test Plans

Issue #402 and Issue # 228: This is shown graphically with dummy topics and assignment. We create a dummy assignment with two different topics oss1 and oss2. oss1 has 2 slots and oss2 has 1 slot. So when a student who has submitted oss2 tries to review others work he will not be able to select oos2 at all because he cannot review his own submission. When a person who has oss1 as the topic goes to this page and he asks to get a new submission for review it will give another slot’s oss1. If he does it again it says there are no more submissions available to review for this topic. So if he has already reviewed all the other submissions of his topic oss1 and he requests for one more submission of oss1, he will not get any more submission to review because the only one left is his own submission.

Issue #417 : We have added the reviews and meta-reviews left option. It can be demonstrated from the image below. Here is the preview of the old information.

In the below diagram, we can see the updated information about the number of reviews left.

Improved UI and Ruby Code practices

• a. The two statements "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." are not well differentiated. It can be very confusing for an instructor to give values to them.

Solution: Information buttons have been provided beside these statements in the review strategy tab to make sure that the instructor knows the difference between the two.

• b) When the number of allowed or required reviews are not set on the Review Strategy tab, the system does not have a message to display to a reviewer about how many submissions of work they are required to and allowed to review depending on the values set by the instructor in the "allowed number of reviewers per reviewer" field and the "Set Required Number of Reviews per reviewer." field.

Solution: Added a toggle hidden option to show the options only when we are setting the max reviews and meta-reviews limit.

• c)The capitalization and punctuation of statements such as "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." in review strategy tab are incorrect.

Solution: This view has been fixed by making changes in the respective files.

All these modifications are made in the app/views/assignments/edit/_review_strategy.html.erb

Here is the snippet of the old layout with lack of info icon.

The updated image has been demonstrated below.

Working Video

There are two versions of this project. They are:

1. Old version
2. New version

Design Pattern

MVC

The project is implemented in Ruby on Rails that uses MVC architecture. It separates an application’s data model, user interface, and control logic into three distinct components (model, view and controller, respectively).

Dry Principle

We are trying to reuse the existing functionalities in Expertiza, thus avoiding code duplication. Whenever possible, code modification based on the existing classes, controllers, or tables will be done instead of creating the new one.

Team Information

1. Rayan Dasoriya (rdasori@ncsu.edu)
2. Prakshatkumar Shah (pmshah2@ncsu.edu)
3. Ravindersingh Rajpal (rkrajpal@ncsu.edu)
4. Mentor: Ed Gehringer (efg@ncsu.edu)

References

1. Expertiza on GitHub
2. GitHub Repository
3. GitHub Pull Request
4. The Live Expertiza Website
5. Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin