E1839 Review Requirements and Thresholds

Description

This wiki page is the description of the E1839 OSS Writeup project - Review Requirements and Thresholds - for Fall 2018, CSC/ECE 517.

Introduction

Expertiza is an open source project based on Ruby on Rails framework. The Expertiza project is a software that creates reusable learning objects through peer review. It is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in some courses at NC State University and by professors at several other universities. It supports team projects, and the submission of almost any document type, including URLs and wiki pages. Expertiza enables the instructor to create new and customize existing assignments. It also enables the instructor to create a list of topics the students can sign up for as part of a project. Students can form teams in Expertiza to work on various projects and assignments. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Description

In Expertiza, there are two ways of assigning reviews to reviewers: either the instructor decides who reviews whom (“instructor-selected”), or “auto-selected,” in which case reviews are not assigned until a student seeks to choose something to review. This project is mostly about auto-selected reviewing. A reviewer is ordinarily allowed to choose only among work that has received the minimum number of reviews so far. For example, if all submissions have received at least one review, the next person who tries to review is allowed to choose only a submission that has only one review so far, unless there are no more such submissions, in which case the reviewer chooses from among submissions that have 2 reviews (if there are any), or the minimum number of reviews, whatever that is.

For assignments that have topics, this severely constrains a reviewer’s choice of what topic to review on. (For assignments that don’t have topics, a reviewer doesn’t have anything to choose; (s)he just gets one of the submissions that have the fewest reviews so far.) To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far.

There are some rough edges to this strategy. At one time, a reviewer was allowed to choose a topic that only (s)he had submitted on, only to be told by the system that no more submissions were reviewable for this topic (because no one is allowed to review their own submission). I think that bug has been fixed (please check!), but there is another similar situation that I think still remains. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0). That’s wrong; I should always be allowed to review the work with the least reviews that I have not already reviewed. And that should generalize to situations in which k > 0.

Another issue is that there is no way for Expertiza to tell a reviewer how many reviews are required. In the case of instructor-selected reviewing, that’s not a problem, but for auto-selected reviewing, there is no way to specify how many reviews are required (or even how many are allowed, in case students are allowed to do extra reviews).

Design Pattern

1. MVC

The project is implemented in Ruby on Rails that uses MVC architecture. It separates an application’s data model, user interface, and control logic into three distinct components (model, view and controller, respectively).

2. Dry Principle

We are trying to reuse the existing functionalities in Expertiza, thus avoiding code duplication. Whenever possible, code modification based on the existing classes, controllers, or tables will be done instead of creating the new one.

Team Information

Rayan Dasoriya (rdasori@ncsu.edu)

Prakshatkumar Shah (pmshah2@ncsu.edu)

Ravindersingh Rajpal (rkrajpal@ncsu.edu)

Mentor: Ed Gehringer (efg@ncsu.edu)