CSC/ECE 517 Spring 2016 E1625 Show confidence ratings for grade based on reputations of reviewers

This page documents the project to implement confidence ratings for a student peer review grade based on the reputations of the reviewers

Introduction to Expertiza

Expertiza is a  web application developed using the Ruby on Rails framework as a combined effort of students and faculty. The main advantage of using Expertiza, in an educational environment, is for the instructor to introduce peer reviewing among the students. The instructors can create and customize assignments, create a list of topics the students can sign up for, have students work on teams and then review each other's assignments at the end. Expertiza supports submission of almost any document type, including the URLs and wiki pages.

Expertiza is supported by National Science Foundation under Grant No. 0536558. Additional funding from the NCSU Learning in a Technology-Rich Environment (LITRE) program, the NCSU Faculty Center for Teaching and Learning, the NCSU STEM Initiative, and the Center for Advanced Computing and Communication.

The E1625 Project

Motivation

Confidence ratings are meant to provide objective value to student assigned peer review scores. Students select from a list of tasks to be performed and then prepare their work and submit it to a peer-review system. The work is then reviewed by other students who offer comments/graded feedback to help the submitters improve their work.

During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.

Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.

Description of confidence rating and our system

Hamer’s and Lauw’s algorithm

There are two algorithms intended for use in calculation of the reputation values for participants.

There is a web-service (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the participants table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the paper on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”

Reputation range of Hamer’s algorithm is

red                   value < 0.5

yellow              value is >= 0.5 and <= 1

orange             value is > 1 and <= 1.5

light green       value is > 1.5 and <= 2

green               value is > 2

The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. This project determines reputation by subtracting the absolute value of the leniency from 1. Additionally, the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].

Reputation range of Lauw’s algorithm is

red                   value is < 0.2

yellow              value is >= 0.2 and <= 0.4

orange             value is > 0.4 and <= 0.6

light green       value is > 0.6 and <= 0.8

green               value is > 0.8

The instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm. A Sample of the webpage displaying reputations with Lauw's algorithm is given below.

Design Pattern - Model-view-controller(MVC)

In an MVC pattern , a software application is divided into three major components : Model, View and Controller. This allows separation of internal representations of information from the information that needs to be presented to the user.

Model - Maintains the data, logic and the relationships between objects and the database. It also handles validations, associations and transactions.

View -  Displays data received from the controller in the User Interface.

Controller - Accepts input from client and converts it into actionable items for the model or the view. Its responsibilities include querying models and organizing data into a structure that is displayed by the view

Documentation

All the documentation for the Expertiza system can be found in the following links: Expertiza on Github ,   Wiki documentation for Expertiza

Implementation

Models

participant

Views

grades/view.html.erb

grades/view_my_scores.html.erb

grades/_review_table.html.erb

assignment_form.rb, edit/_general.html.erb

Controllers

Reputation_web_service_controller.rb

RSpec Files

reputation_web_service_controller_spec.rb

assignment_creation_spec.rb

grades_spec.rb

Standards

All our code follows the global rules on Ruby style.

Tasks Performed

1. Write a migration to add reputation field in participant table. (The default value of this field is 1, set any nil to 1)
2. Change the send_post_request method in reputation_web_service_controller to obtain initial reputation values from DB instead of hard-coded.
3. Write a new method in reputation_web_service_controller to extract reputation values from JSON response and insert into DB.
4. On view_my_score (student-end) and view_scores (instructor-end) pages: use different font color on students’ names to distinguish different level of reputation values

Testing

We will be using the existing test suite used by gem to test any new code modification. We will be writing new test cases for any new public method exposed by existing classes.Our priority is to add functional tests associated with reputation for instructor, assignment and student controller. For example, One test workflow would be

Testcase 1

1) Instructor selects type of reputation algorithm

2) Chooses the assignment and the round for which the reputation needs to calculated

3) Receives the json response

4) Check if the response is in the right format and the that the reputation for each student is stored in the database.

Testcase 2

1) Student logs in to Expertiza.

2) Selects a particular assignment.

3) Clicks on 'View My Scores'.

4) Student should be able to see his/her reputation value along with the scores.

References

1.    Expertiza on GitHub (Original version)

2.    Expertiza