CSC/ECE 517 Spring 2013/Final Project E730

E370. Reputation Integration

• Contact Info:

1. Mark Hall (mlhall3@ncsu.edu)
2. Dan Howard(drhowar5@ncsu.edu)
3. Eric Lumpkin(eblumpki@ncsu.edu)
4. Ashray Nagaraju (ashray@ncsu.edu)

Introduction

Purpose

In Expertiza, students are able to grade each other’s work. This is accomplished through a system of peer reviews, and subsequently reviewing those reviews. One major problem with this system is that while all reviews are not equal in quality, all reviews are equally weighted.

Some students are better at reviewing than others. Additionally, some students have better intentions than others when it comes to writing reviews. In order to avoid unfairly rewarding or penalizing students, an algorithm has been implemented that gives an appropriate weight to a review based upon its merit. This algorithm checks to see how closely the score a reviewer gives is to the average score given by all reviewers. This is used to determine how “good” a reviewer is.

The algorithm that has been implemented is based upon Hamer’s algorithm [[Hamer's Algorithm] http://crpit.com/abstracts/CRPITV42Hamer.html]. However, this code was not integrated into the Expertiza codebase.

This project takes the existing, unimplemented code, and integrates it into the Expertiza codebase. In addition, this project aim to allow the unobtrusive display of the reputation value, as well as the ability to toggle the reputation effect on the score. Finally, we will also provide for the instructors to delete rogue reviews.

 

Problem Definition

Integrate Hamer's algorithm into the Expertiza codebase, making it available for score calculation.

Display reputation values for reviewers unobtrusively by use of a tool-tip.

Allow instructors to toggle the reputation value on or off with respect to affecting the student's grade.

Allow instructors to delete "rogue" reviews.

Proposed Design

1. Add a button by each review that will allow an instructor to delete reviews which have been determined to "rogue" (ie, an outlier in terms of the average review score.)

1. Implement a toggle that allows an instructor to use either the equally weighted or reputation-based grading systems. This will allow the instructor to choose which weighting system to use at the assignment level. All reviews for a given assignment will use the same weighting system.

1. Use a tool tip to display a given user’s reputation when the instructor hovers over the reviewer’s name with the mouse pointer.

1. Add Unit/Functional test cases to test the functionality of the algorithm.

Use Cases

Use-case: 1

Name: Turn reviewer weighting on and off in calculations for a review by use of an ‘enabled’ field/toggle.

Actors: Instructor/Admin

Other participants: None

Precondition: There is no weighted grading system enabled

Primary Sequence:

1. Login to Expertiza (Admin or an equivalent person).

1. Click on Create an assignment.

1. Enable weighted grading for the particular assignment by actuating an enabled toggle.

1. Save the changes.

Postcondition: Weighted grading system is enabled for the created assignment

Use-case: 2

Name: Display reviewers reputation by hovering over a field.

Actors: Instructor/Admin

Other participants: None

Precondition: Mouse is not over a score field.

Primary Sequence:

1. Login to Expertiza (Admin or an equivalent person).

1. Click on Create an assignment

1. Hover the mouse over a score field.

Postcondition: The reviewers reputation will be displayed in a tool tip.

Use-case: 3

Name: Disable a rogue review.

Actors: Instructor/Admin

Other participants: None

Precondition: A review has been identified as rogue by the instructor.

Primary Sequence:

1. Login to Expertiza (Instructor or an equivalent person).
2. Navigate to the rogue review.
3. Toggle the review to be disabled.
4. Save the changes.

Postcondition: The rogue review is not included in the average weighted score.

Test Cases

1. Ensure that the weighting algorithm returns the correct score for a range of values
2. Ensure that switching between the equal weighting and reputation-based weighting calculates and displays the score correctly.
3. Ensure that once rogue reviews are detected and counted out they change the scores appropriately.
4. Ensure that the correct reputation score is displayed in a tooltip when highlighting a score.
5. Ensure that only the instructor can view the reputation scores for students.
6. Ensure that only the instructor can mark a review as rogue.
7. Ensure that weighted scores are displayed only after deadline, else unweighted scores need to be displayed.