CSC/ECE 517 Spring 2013/Final Project E730

E370. Reputation Integration

• Contact Info:

1. Mark Hall (mlhall3@ncsu.edu)
2. Dan Howard (drhowar5@ncsu.edu)
3. Eric Lumpkin (eblumpki@ncsu.edu)
4. Ashray Nagaraju (ashray@ncsu.edu)

Introduction

Purpose

In Expertiza, students are able to grade each other’s work. This is accomplished through a system of peer reviews, and subsequently reviewing those reviews. One major problem with this system is that while all reviews are not equal in quality, all reviews are equally weighted.

Some students are better at reviewing than others. Additionally, some students have better intentions than others when it comes to writing reviews. In order to avoid unfairly rewarding or penalizing students, an algorithm has been implemented that gives an appropriate weight to a review based upon its merit. This algorithm checks to see how closely the score a reviewer gives is to the average score given by all reviewers. This is used to determine how “good” a reviewer is.

The algorithm that has been implemented is based upon Hamer's algorithm. However, this code was not integrated into the Expertiza codebase.

This project takes the existing, unimplemented code, and integrates it into the Expertiza codebase. In addition, this project aim to allow the unobtrusive display of the reputation value, as well as the ability to toggle the reputation effect on the score. Finally, we will also provide for the instructors to delete rogue reviews.

Problem Definition

Integrate Hamer's algorithm into the Expertiza codebase, making it available for score calculation.

Display reputation values for reviewers unobtrusively by use of a tool-tip.

Allow instructors to toggle the reputation value on or off with respect to affecting the student's grade.

Allow instructors to delete "rogue" reviews.

Proposed Design

1. Add a button by each review that will allow an instructor to delete reviews which have been determined to "rogue" (ie, an outlier in terms of the average review score.)
2. Implement a toggle that allows an instructor to use either the equally weighted or reputation-based grading systems. This will allow the instructor to choose which weighting system to use at the assignment level. All reviews for a given assignment will use the same weighting system.
3. Use a tool tip to display a given user’s reputation when the instructor hovers over the reviewer’s name with the mouse pointer.
4. Add Unit/Functional test cases to test the functionality of the algorithm.

Running the Code

First, download the scrubbed database to a location referred to here as "DB_PATH".

"PATH" is simply where you want to clone your repository to contain your working code.

Next, reserve the "Ruby on Rails" image in VCL. Open a terminal window once the reservation is ready.

1. rvm use 1.8.7
2. mkdir /PATH/project3
3. cd /PATH/project3
4. unset SSH_ASKPASS
5. git clone http://github.com/ashrayn/expertiza.git
6. cd expertiza
7. git stash
8. git checkout e730-reputation
9. sudo rvm gemset empty
10. gem install bundle
11. sudo yum install aspell-devel
12. gem install raspell -v 1.3
13. bundle install
14. rake db:migrate
15. mysqladmin -uroot password admin
16. mysql -uroot -p pg_development < /DB_PATH/expertiza-scrubbed.sql
17. mysql -u root admin
18. SET PASSWORD FOR root@localhost=PASSWORD();
19. create user expertiza@localhost identified by "";
20. grant all on pg_development.* to expertiza@localhost;
21. exit
22. ./script/console
23. u = User.find_by_name("admin")
24. u.clear_password="admin"
25. u.clear_password_confirmation="admin"
26. u.save
27. quit
28. ./script/server

Use Cases

Use-case: 1

Name: Turn reviewer weighting on and off in calculations for a review by use of an ‘enabled’ field/toggle.

Actors: Instructor/Admin

Other participants: None

Precondition: There is no weighted grading system enabled

Primary Sequence:

1. Login to Expertiza (Admin or an equivalent person).
2. Click on Create an assignment.
3. Enable weighted grading for the particular assignment by actuating an enabled toggle.
4. Save the changes.

Postcondition: Weighted grading system is enabled for the created assignment

Use-case: 2

Name: Display reviewers reputation by hovering over a field.

Actors: Instructor/Admin

Other participants: None

Precondition: Mouse is not over a score field.

Primary Sequence:

1. Login to Expertiza (Admin or an equivalent person).
2. Click on Create an assignment
3. Hover the mouse over a score field.

Postcondition: The reviewers reputation will be displayed in a tool tip.

Use-case: 3

Name: Disable a rogue review.

Actors: Instructor/Admin

Other participants: None

Precondition: A review has been identified as rogue by the instructor.

Primary Sequence:

1. Login to Expertiza (Instructor or an equivalent person).
2. Navigate to the rogue review.
3. Toggle the review to be disabled.
4. Save the changes.

Postcondition: The rogue review is not included in the average weighted score.

Test Cases

1. Ensure that the weighting algorithm returns the correct score for a range of values
2. Ensure that switching between the equal weighting and reputation-based weighting calculates and displays the score correctly.
3. Ensure that once rogue reviews are detected and counted out they change the scores appropriately.
4. Ensure that the correct reputation score is displayed in a tooltip when highlighting a score.
5. Ensure that only the instructor can view the reputation scores for students.
6. Ensure that only the instructor can mark a review as rogue.
7. Ensure that weighted scores are displayed only after deadline, else unweighted scores need to be displayed.

View Nesting

One of the primary difficulties encountered in altering the view to correctly display the reputation was in determining which view should contain the view code. This was difficult due to the way the views were nested, and rendered as partials. In addition, it was not always clear what variables were available to a view (due simply to not knowing the entry point into the view because of this nesting) or what parameters were accessible.

After examining the code in detail, a diagram of the view nesting was created, which would prove useful to anyone further modifying the reputation code and display, or any other aspect related to displaying scores.