How Self Reviews are combined with Peer Reviews for Final Grade

Information on Vossen Formula:

  grade = w * (avg_peer_review_score) + (1 - w) * (avg_peer_review_score * (1 +/- ((avg_peer_review_score - self_review_score).abs() / avg_peer_review_score)))

More fully, as in the code:

  if |avg_peer_review_score - self_review_score|/avg_peer_review_score <= l(leniency)
     grade = w*(avg_peer_rev_score) + (1-w)*(avg_peer_review_score * (1 + (|avg_peer_review_score - self_review_score|/avg_peer_review_score)))
  else
     grade = w*(avg_peer_rev_score) + (1-w)*(avg_peer_review_score * (1 - (|avg_peer_review_score - self_review_score|/avg_peer_review_score)))

The formula that determines a final grade from, 1) the peer reviews and 2) how closely self reviews match the peer reviews, uses a type of additive scoring rule, which computes a weighted average between team score (peer reviews) and student rating (self review). More specifically, it uses a type of mixed additive-multiplicative scoring rule, which multiplies student score (self review) by a function of the team score (peer reviews), and adds its weighted version to the weighted peer review score. The function is the deviation (in percentage) of the self-review score from the average peer review score. This is also known as 'assessment by adjustment'. The formula is a practical scoring rule for additive scoring with unsigned percentages (grades from 0%-100%).

w should be chosen based on the instructor's desired percentage (w) of the final grade to be determined from peer reviews and, conversely, the instructor's desired percentage (1-w) of the final grade to be determined by the extent to which self reviews deviate from peer reviews. In addition, l - leniency, should be chosen based on the instructor's desired percentage of the deviation of self review from peer review that could result in no grade deduction from the deviation if the deviation is sufficiently small (or even a grade increase if the instructor wants to increase the score of individuals with a small deviation). l is not able to be chosen from the UI - it is hard coded into Expertiza because it depends on the assignment grading scale. The Expertiza review scale is out of 5, so l = 0.25 because the minimum deviation is: 1 / (max score -1) (and max score is 5).

The parameter l - leniency - can determine a threshold by which the final grade will account/adjust for self reviews' deviations from peer reviews only when the deviation reaches this threshold (measured in percentage deviation from the average peer review). If the difference does not meet the threshold, no penalty will be subtracted from the peer review. In addition, if the difference does not meet the threshold (the self review score is sufficiently close to the peer review scores), the instructor can choose to add points to final grade based on the magnitude of the difference.

Using an example where the average peer review score is 4/5, the self review score is 5/5, and w = 0.95, the self review score (5/5) differs by 25% of the peer review score (4/5). In other words, |avg_peer_review_score - self_review_score|/avg_peer_review_score = 1/4 = 25%. Based on l - leniency 0.25 (the required minimum deviation for a grading scale out of 5):

A 25% deviation is sufficiently small (it is the minimum deviation) to warrant increasing the final grade by (1-w)*(avg_peer_review_score * (1 + (|avg_peer_review_score - self_review_score|/avg_peer_review_score))) If the deviation were larger (self-review score was 2/5, 1/5, or 0/5 while self-review score was still 4/5), the formula would simply apply the 'else' statement instead of the 'if' statement since the deviation would be greater than 25%.