CSC/ECE 517 Fall 2020 - E2078. Improve self-review Link peer review & self-review to derive grades
Project Overview
In Expertiza, it is currently possible to check the “Allow self-review” box on the Review Strategy tab of assignment creation, and then an author will be asked to review his/her own submission in addition to the submissions of others. But as currently implemented, nothing is done with the scores on these self-reviews.
There has been a previous attempt at solving this problem, but there were several issues with that implementation:
- The formula for weighting self-reviews is not modular. It needs to be, since different instructors may want to use different formulas, so several should be supported.
- There are not enough comments in the code.
- It seems to work for only one round of review.
View documentation for previous implementation here.
Objectives
Our objectives for this project are the following:
- Display the self-review score with peer-review scores for the logged in user
- Implement a way to achieve a composite score with the combination of the self-review score and peer-review scores
- Implement a requirement for the logged in user to self-review before viewing peer-reviews
- Assure that we overcome the issues outlined for the previous implementation of this project
Team
Courtney Ripoll (ctripoll)
Jonathan Nguyen (jhnguye4)
Justin Kirschner (jkirsch)
Files Involved
Tasks
Display Self-Review Scores w/ Peer-Reviews
It should be possible to see self-review scores juxtaposed with peer-review scores. Design a way to show them in the regular "View Scores" page and the alternate (heat-map) view. They should be shown amidst the other reviews, but in a way that highlights them as being a different kind of review.
Design Plan
In the current implementation of Expertiza, students can view a compilation of all peer review scores for each review question and an average of those peer-reviews. For our project, we plan to add the self-review score alongside the peer-review scores for each review question. In the wire-frame below, note that for each criterion there is a column for each peer-review score and a single column for the self-review score. The avg column then takes an average of all review scores (peer-review and self-review). Additionally, the composite score is displayed on the page under the average peer review score. How we plan to derive a composite score is explained in detail in the follow section Derive Composite Score.
In the alternate view, our plan does not alter the current interface too much. The only addition we plan to implement is an additional column describing the composite score a student has received for an assignment. Likewise, there is now an additional doughnut chart providing a visual of the composite score (green) alongside the final review score (yellow).
Derive Composite Score
Implement a way to combine self-review and peer-review scores to derive a composite score. The basic idea is that the authors get more points as their self-reviews get closer to the scores given by the peer reviewers. So the function should take the scores given by peers to a particular rubric criterion and the score given by the user. The result of the formula should be displayed in a conspicuous page on the score view.
Implementation Plan [reference, DOI: 10.21125]
The formula we use to determine a final grade from, 1) the peer reviews and 2) how closely self reviews match the peer reviews, uses a type of additive scoring rule, which computes a weighted average between team score (peer reviews) and student rating (self review). More specifically, it uses a type of mixed additive-multiplicative scoring rule, which multiplies student score (self review) by a function of the team score (peer reviews), and adds its weighted version to the weighted peer review score. This is also known as 'assessment by adjustment'. The formula is a practical scoring rule for additive scoring with unsigned percentages (grades from 0%-100%).
The pseudo-code for a function that implements the formula is as follow:
function(avg_peer_rev_score, self_rev_score, w)
grade = w*(avg_peer_rev_score) + (1-w)*(SELF)
where:
SELF = avg_peer_review_score * (1 - (|avg_peer_review_score - self_review_score|/avg_peer_review_score))
and where: avg_peer_review_score is simply the mechanism already existing in Expertiza for assigning a grade from peer review scores and where: w - weight - (0 <= w <= 1) is the inverse proportion of how much of the final grade is determined by the closeness of the self review to the average of the peer reviews (w is the proportion of the grade to be determined by the original grade determination: the peer review scores).
An example:
- average peer review score is 4/5, self review score is 5/5
- the instructor chooses w to equal 0.95, so that 5% of the grade is determined from the deviation of the self review from the peer reviews.
The final grade, instead of being the peer review score of 4.5 (80%) is now: 0.95*(4/5) + 0.05*(4/5*(1-|4/5-5/5|/(4/5))) = 79% If the instructor chose w to equal 0.85 (instead of 0.95), the grade is 77% (instead of 79%) because deviation from peer reviews is a larger weighted value of the final grade.
The above is a basic version of the grading formula. It is basic in that it only allows deviations of the self review scores from the peer review scores to result in a decrease in the final grade and a deviation will always result in a decrease of the final grade. We propose another parameter to the formula, l - leniency - as another way (in addition to w - weight) for the instructor to modularly determine the final grade for an assignment. The parameter l - leniency - can determine a threshold by which the grade final will account/adjust for self reviews' deviations from peer reviews only when the deviation reaches a certain threshold (measured in percentage deviation from the average peer review). If the difference does not meet the threshold, no penalty will be subtracted from the peer review. In addition, if the difference does not meet the threshold (the self review score is sufficiently close to the peer review scores), the TA can choose to add points to final grade based on the magnitude of the difference. Since the formula is a mixed additive-multiplicative scoring rule (mentioned above), the instructor needs to simply pick l - leniency - as a percentage (similar to the functionality of w). To recap: w should be chosen based on the instructor's desired percentage of the final grade to be determined from peer reviews and the instructor's desired percentage of the final grade to be determined by the extent to which self reviews deviate from peer reviews. In addition, l - leniency - should be chosen based on the instructor's desired percentage of the deviation of self review from peer review that could result in no grade deduction resulting from the deviation if the deviation is sufficiently small (or even a grade increase if the instructor wants to increase the score of individuals with a small deviation).
The following is psuedo-code for if a instructor wishes to not subtract from the final grade if the deviation is sufficiently small: if |avg_peer_review_score - self_review_score|/avg_peer_review_score <= l(leniency)
grade = avg_peer_review_score
else
run formula
We see that
As mentioned, instead of assigning grade to the avg_peer_review score if the leniency conditions is met, the grade can be adjusted (increased) if the instructor wishes to do so, since the self review is sufficiently close (determined by l) to the peer reviews). The formula for determining the final grade would thus add the small extent of deviation to the final grade rather than subtracting it (in SELF, 1 - ..., is changed to 1 + ...)
function(avg_peer_rev_score, self_rev_score, w) grade = w*(avg_peer_rev_score) + (1-w)*(SELF)
where SELF = avg_peer_review_score * (1 + (|avg_peer_review_score - self_review_score|/avg_peer_review_score))
In this case, the pseudo code is:
grade = w*(avg_peer_rev_score) + (1-w)*(SELF)
where:
SELF = avg_peer_review_score * (1 - (|avg_peer_review_score - self_review_score|/avg_peer_review_score))
if |avg_peer_review_score - self_review_score|/avg_peer_review_score <= l(leniency)
grade = w*(avg_peer_rev_score) + (1-w)*(avg_peer_review_score * (1 + (|avg_peer_review_score - self_review_score|/avg_peer_review_score))
else
grade = w*(avg_peer_rev_score) + (1-w)*(avg_peer_review_score * (1 - (|avg_peer_review_score - self_review_score|/avg_peer_review_score))
Using the previous example (average peer review score is 4/5, self review score is 5/5), the self review score different by 25% (1/5) of the peer review score (4/5). In other words, |avg_peer_review_score - self_review_score|/avg_peer_review_score = 1/4 = 25%. Based on l - leniency - the instructor can decided
1) if a 25% deviation is sufficiently large to warrant penalizing the final grade by (1-w)*(SELF)
2) if a 25% deviation is sufficiently small to warrant keeping the final grade as grade = avg_peer_review_score, with no penalty for the deviation
3) if a 25% deviation is sufficiently small to warrant increasing the final grade by (1-w)*(SELF), where the SELF formula contains a 1 + ..., instead of a 1 - ...
In order to incorporate the combined score into grading, we will change the logic in grades_controller.rb, which implements the grading formula. We can obtain the peer review ratings and the score/grade derived from them by calling scores() and compute_assignment_score(), respectively, from assignment_participant.rb.
- Ask for Dr G's input about Expertiza's file structure/design and if the previous group got it correct. (Their diagram is hard to follow without a thorough background.) **
Implement Requirement to Review Self before Viewing Peer Reviews
There would be no challenge in giving the same self-review scores as the peer reviewers gave if the authors could see peer-review scores before they submitted their self-reviews. The user should be required to submit their self-evaluation(s) before seeing the results of their peer evaluations.
Implementation Plan
Test Plan
Manual Testing
RSpec Testing
Relevant Links
Our repository: https://github.com/jhnguye4/expertiza/tree/beta
Pull request: Does not exist yet
Video demo: Does not exist yet
References
Scoring models for peer assessment in team-based learning projects