CSC/ECE 517 Spring 2018- Project E1819: Improve self review, link self and peer review to derive grades

Introduction and Problem Statement

Expertiza has the ability to require self-reviews on projects, however, these self-review scores do not have much functionality. In the current system, self-review scores are created and stored, but are not used afterwards. This project seeks to improve upon the self-review system, and extend its functionality. The main points of this project are as follows:

• Display the self-review scores in the "View Scores" page juxtaposed with peer-review scores.
• Display the self-review scores in the heat map of peer-reviews, with some way to denote they are a different type of review.
• Create a method to calculate a linked score between self and peer-reviews. The closer the self-review is to the peer-review, the greater the score. Display this score on the "View Scores" page.
• Ensure that users cannot view peer-reviews until they complete their self-review.

Planned Approach

Allow instructors to decide if students should be allowed to view peer-reviewers before completing their self-review.

Currently when instructors create a new assignment, or edit an existing one, they have the option to require self-reviews.

This will be updated to ask instructors if peer-reviews should be visible before self-reviews are completed.

When checked the system will check if a student has completed a self-review when they access the scores view. If they have not completed their self-review they will see a message indicating such. If they have completed the review, they will see their peer-reviews as normal.

Find or create a function that will derive a score from self-reviews and peer-reviews.

This score should be reflective of the difference between the students self-review score, and their peer's reviews. The purpose behind this score is to teach students to become better at reviewing their own work. By providing a score that reflects how similar their reviews are to their peer's reviews, students receive feedback on their self-assessment abilities. More information on the function creation/decision proccess can be found in the section below.

Create the different displays for self-reviews

Instructors and students both have a heat-map view of scores from peer-reviews

(Instructor view pictured) student's self-review scores will be added to this view, but need to be denoted in a way that makes it easy to see this is a self-review not a peer-review. One potential option is to use an icon that shows this is a self-review score.

Which makes it simple for users to understand that particular column represents the scores from a self-review. Other potential options are to title the column "Self-review", or to highlight/border the column with a particular color (ex. blue), both denoting that the score comes from a self-review. Similar approaches can be taken to differentiate between self-review and peer-review scores in the regular "view scores" page.

Score Formulas

Several potential formulas to calculate the derived score between self and peer-reviews were discussed. A naive approach is to find the difference between the max score and the difference between self and peer-reviews.

A slight change to this approach is to take into account the peer score, so the greater your peers believe you performed, the greater your score is (proportional to the difference between reviews still). However, this approach can prove to be harmful. Below it can be seen that even when you perfectly reflect the score of your peers your grade can be very low. This approach punishes students twice for low peer-review grades, and it should be avoided.

A third approach is discussed here. Which results in a score that should accurately reflect the relation between self and peer scores. However, it can again be seen that when peer-review scores are low students are punished twice unless they can accurately reflect their peer's reviews.

A fourth approach is to derive the grade based on the percent difference between the self-review score, and the peer-review scores. This results in a smooth gradient which does not punish students as harshly as the previous two approaches. However, at low scores students will see greater swings in this derived grade.

Testing

Test cases are as follows:

• Peer-reviews cannot be viewed before a self-review is completed.
• Peer-reviews can be viewed immediately when self-reviews are not required.
• The derived scores can be saved to the database.
• Self-review scores are displayed with peer-review scores
• Student can only see their own self-review

Reference

Our repo link

our pull request link

Edward F. Gehringer, "Self-assessment to improve learning and evaluation," ASEE Annual Conference and Exposition, Columbus, OH, June 25–28, 2017