Scoring & Grading Methods (Fall '21)

Scoring and Grading was given a refactor in the Fall of 2021. This doc page describes the key files for scoring and grading, their methods, and how to use them.

At the lowest level, a score needs to be calculated anytime anyone fills out a rubric. This score is calculated in Response.rb. It is computed as a weighted sum of the (scorable) items in the rubric. If, for example, the assignment has two rounds of review, and reviewer x does a review in each round, there will be a score for the review done by reviewer x in Round 1 and a score for the review done by reviewer x in Round 2.

In ResponseMap.rb, the scores given by an individual reviewer in different rounds are accumulated. If, for example, the Round 1 review is weighted 70% and the Round 2 review 30%, the ... method in ResponseMap.rb computes Round 1 score from reviewer x * 70% + Round 2 score from reviewer x * 30% .

The scores from different reviewers are averaged in AssignmentTeam.rb. This may be a simple average of reviewer x_i's score, for all i = 0, ..., m-1, or if the reviewers have been assigned different reputations, it may be an average where each reviewer's score is weighted by that reviewer's reputation.

Then, to get a final score, we need to add together the scores for the different kind of reviews, e.g., teammate reviews, author feedback, and reviews of submitted work. [Where is this done?]

Key Files & Methods

assignment_team.rb

assignment_helper.rb

Methods used in the now-defunct on_the_fly_calc were merged into this file as part of the refactor, since any previous calls to the methods took place through an instance of assignment. There is currently an attempt to further abstract these methods in assignment_helper.rb so that assignment is passed as a parameter (instead of using self).

The following methods in assignment_helper are part of the Scoring and Grading refactor:

• compute_total_score(scores): When called with assignment.compute_total_score(scores) (such as in assignment_team.rb), it will sum the weighted scores of each questionnaire in an assignment using the scores passed to it and return the total. Ex) 60% Second Round Review Questionnaire, 40% Author Feedback Questionnaire on an assignment make up the total score.

• compute_reviews_hash: When called with @assignment.compute_reviews_hash (such as in report_formatter_helper.rb), it will itself call the private methods scores_varying_rubrics or scores_non_varying_rubrics based on the @assignment.vary_by_round value. When calling these private methods, it passes an empty review_scores hash and the response_maps of the assignment. The chosen private method then iterates through each of the response_maps for the assignment and returns the filled review_scores hash in the format {response_map.reviewer_id: respective_scores}. This respective_scores value is calculated through calc_review_score - another private method in assignment_helper.rb.

• compute_avg_and_ranges_hash When called with @assignment.compute_avg_and_ranges_hash (such as in report_formatter_helper.rb), it will find the contributors (assignment_teams) for the assignment, and then find two variables for each team - questions and assessments. The first, questions, are the peer review questions for that team. The second, assessments are the team's owned assessments in ReviewResponseMap. Then Response.compute_scores(assessments, questions) is called to calculate scores. Finally, a hash of {contributor.id: <computed scores>} is returned.

These are the public methods which can be called from assignment_helper. The private methods are only called by these above methods.

response_map.rb

A significant part of this refactor involved moving methods from assignment.rb and assignment_participant.rb into response_map.rb. Below are some of the most commonly used methods in response_map.rb.

• self.scores(assignment, questions): Computes and returns the scores of assignment for participants and teams
• self.participant_scores(participant, questions): Return scores that this participant has been given
• self.compute_assignment_score(participant, questions, scores): Called by participant_scores() to retrieve the assignment score. This method itself calls Response.compute_scores() to find the total score for a list of responses to a questionnaire. It then fills out a scores hash filled with questionnaire symbols with this total score data.
• self.merge_scores(participant, scores): For each assignment review all scores and determine a max, min and average value
• self.update_max_or_min(scores, round_sym, review_sym, symbol): Called by merge_scores() to update a rounds min/max scores.

These are the public methods which can be called from response_map. The private methods are only called by these above methods.

response.rb

A significant part of this refactor involved moving methods from answer.rb into response.rb. Below are some of the most commonly used methods in response.rb.

• self.compute_scores(assessments, questions): Computes the total score for a list of assessments of some type (author feedback, teammate review). The questions parameter is a list of what was filled out in the process of doing those assessments. Called by several files, including bookmarks_controller, assignment_helper, assignment_team, and response_map.
• self.assessment_score(params): Computes the total score for a specific assessment (not a list). Called by compute_scores on each individual assessment. If there is no score, it returns -1.0.

Display Files

grades_controller.rb

Largely generic methods, most of which assist in rendering views of grades. Includes redirection commands, a bar chart method, and a view model for grades.

grades_helper.rb

Helper methods for rendering views of grades. Includes notable methods regarding charts, vectors, and heatmaps of grades.

Significant Pull Requests

These are the pull requests responsible for the refactor:

• Refactor on_the_fly_calc
• Refactor scores in assignment and participant
• Grade refactoring
• Answer refactoring