CSC/ECE 517 Spring 2016 E1625 Show confidence ratings for grade based on reputations of reviewers

Show confidence ratings for grade, based on reputations of reviewers

Introduction to Expertiza

Expertiza is a  web application developed using the Ruby on Rails framework as a combined effort of students and faculty. The main advantage of using Expertiza, in an educational environment, is for the instructor to introduce peer reviewing among the students. The instructors can create and customize assignments, create a list of topics the students can sign up for, have students work on teams and then review each other's assignments at the end. Expertiza supports submission of almost any document type, including the URLs and wiki pages.

Expertiza is supported by National Science Foundation under Grant No. 0536558. Additional funding from the NCSU Learning in a Technology-Rich Environment (LITRE) program, the NCSU Faculty Center for Teaching and Learning, the NCSU STEM Initiative, and the Center for Advanced Computing and Communication

Motivation

Students select from a list of tasks to be performed, then they prepare their work and submit it to a peer-review system. The work is then reviewed by other students, who offer comments/feedback to help the submitters improve their work.

During the peer review period it is important to determine which reviews are more accurate and show higher quality. Reputation is one way to achieve this goal; it is a quantization measurement to judge which peer reviewers are more reliable.

Peer reviewers can use expertiza to score an author. If Expertiza shows a confidence ratings for grades based upon the reviewers reputation then authors can more easily determine the legitimacy of the peer assigned score. In addition, the teaching staff can examine the quality of each peer review based on reputation values and, potentially, crowd-source a significant portion of the grading function.

Description of confidence rating and our system

Hamer’s and Lauw’s algorithm

There are two algorithms intended for use in calculation of the reputation values for participants.

There is a web-service (the link accessible only to the instructors) available which serves a JSON response containing the reputation value based on the seed provided in the form of the last known reputation value which we store in the participants table. An instructor can specify which algorithm to use for a particular assignment to calculate the confidence rating.

As the paper on reputation system by observes, “the Hamer-peer algorithm has the lowest maximum absolute bias and the Lauw-peer algorithm has the lowest overall bias.This indicates, from the instructor’s perspective, if there are further assignments of this kind, expert grading may not be necessary.”

The main difference between the Hamer-peer and the Lauw-peer algorithm is that the Lauw-peer algorithm keeps track of the reviewer's leniency (“bias”), which can be either positive or negative. A positive leniency indicates the reviewer tends to give higher scores than average. Additionally the range for Hamer’s algorithm is (0,∞) while for Lauw’s algorithm it is [0,1].

Section: Design Pattern - Model-view-controller(MVC)

In an MVC pattern, a software application is divided into three major components : Model, View and Controller. This allows separation of internal representations of informations from the information that needs to be presented to the user.

Model - Maintains the data, logic and the relationships between objects and the database. It also handles validations, associations and transactions.

View -  Displays data received from the controller in the User Interface.

Controller - Accepts input from client and converts it into actionable items for the model or the view. Its responsibilities include querying models and organizing data into a structure that is displayed by the view

Section: Documentation

All the documentation for the Expertiza system can be found in the following links: Expertiza on Github ,   Wiki documentation for Expertiza

Models:

participant

Views:

grades/view.html.erb

grades/view_my_scores.html.erb

Controllers:

Reputation_web_service_controller.rb

Standards:

All our code follows the global rules on Ruby style.

Section: Tasks Performed:

●    Write a migration to add reputation field in participant table. (The default value of this field is 1, set any nil to 1)

●    Change the send_post_request method in reputation_web_service_controller to obtain initial reputation values from DB instead of hard-coded.

●    Write a new method in reputation_web_service_controller to extract reputation values from JSON response and insert into DB.

●    On view_my_score (student-end) and view_scores (instructor-end) pages: use different font color on students’ names to distinguish different level of reputation values

○    Reputation range of Hamer’s algorithm is :

red                   value < 0.5

yellow              value is > 0.5 to 1

orange             value is > 1 to 1.5

light green       value is > 1.5 to 2

green               value is > 2

○    Reputation range of Lauw’s algorithm is

red                   value is < 0.2

yellow              value is < 0.2 to 0.4

orange             value is > 0.4 to 0.6

light green       value is > 0.6 to 0.8

green               value is > 0.8.

●    Instructor can choose to show results from Hamer’s algorithm or Lauw’s algorithm. The default algorithm should be Lauw’s algorithm.

Section: Testing

We will be using the existing test suite used by gem to test any new code modification. We will be writing new test cases for any new public method exposed by existing classes.

Section: References

1.    Expertiza on GitHub (Original version)

2.    Expertiza

Previous Design documents http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015