Reviewer Assignment implementation: Difference between revisions
| (10 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
= Overview = | |||
The review‑assignment subsystem supports both '''static''' and '''dynamic''' allocation of reviews within an assignment. The goal is to give instructors flexible control over how reviewers are matched to submissions, while also supporting on‑demand, fairness‑aware assignment for students who request reviews dynamically. | The review‑assignment subsystem ''in the reimplemented Expertiza'' supports both '''static''' and '''dynamic''' allocation of reviews within an assignment. The goal is to give instructors flexible control over how reviewers are matched to submissions, while also supporting on‑demand, fairness‑aware assignment for students who request reviews dynamically. | ||
This page documents the behavior implemented in ReviewMappingsController and the strategy classes under ReviewMappingStrategies. | This page documents the behavior implemented in ReviewMappingsController and the strategy classes under ReviewMappingStrategies. | ||
| Line 37: | Line 37: | ||
For each row: | For each row: | ||
The reviewer is located by email and matched to an AssignmentParticipant. | *The reviewer is located by email and matched to an AssignmentParticipant. | ||
*The team is located by name within the assignment. | |||
The team is located by name within the assignment. | *A mapping is created if both are found. | ||
A mapping is created if both are found. | |||
This allows instructors to pre‑compute assignments externally and import them directly. | This allows instructors to pre‑compute assignments externally and import them directly. | ||
| Line 59: | Line 57: | ||
'''Eligibility rules:''' | '''Eligibility rules:''' | ||
A reviewer may not be assigned to their own team. | *A reviewer may not be assigned to their own team. | ||
*A reviewer may not be assigned to a team they have already reviewed. | |||
A reviewer may not be assigned to a team they have already reviewed. | *Among eligible teams, the one with the lowest review count is selected. | ||
Among eligible teams, the one with the lowest review count is selected. | |||
This ensures a balanced distribution of reviews across submissions. | This ensures a balanced distribution of reviews across submissions. | ||
| Line 80: | Line 76: | ||
The first eligible topic (deterministically) is selected. | The first eligible topic (deterministically) is selected. | ||
Within that topic, the reviewer is assigned to a team that: | Within that topic, the reviewer is assigned to a team that: | ||
*is not their own team, | |||
*has not already been reviewed by them, | |||
*has the fewest reviews among eligible teams. | |||
This ensures: | This ensures: | ||
*Reviewers can restrict themselves to topics they prefer. | |||
Reviewers can restrict themselves to topics they prefer. | *Topics remain approximately balanced, with fairness controlled by the threshold k. | ||
Topics remain approximately balanced, with fairness controlled by the threshold k. | |||
= Calibration Review Assignment = | = Calibration Review Assignment = | ||
| Line 115: | Line 109: | ||
Instructors may grade individual reviews. This action: | Instructors may grade individual reviews. This action: | ||
Locates the review mapping | *Locates the review mapping | ||
*Records the instructor’s grade and comment | |||
Records the instructor’s grade and comment | *Returns a success message | ||
Returns a success message | |||
= Strategy Classes = | = Strategy Classes = | ||
| Line 125: | Line 117: | ||
All strategies inherit from ReviewMappingStrategies::BaseStrategy, which defines two abstract methods: | All strategies inherit from ReviewMappingStrategies::BaseStrategy, which defines two abstract methods: | ||
each_review_pair — for static strategies | *each_review_pair — for static strategies | ||
assign_one — for dynamic strategies | *assign_one — for dynamic strategies | ||
Each concrete strategy implements one of these depending on its purpose. | Each concrete strategy implements one of these depending on its purpose. | ||
| Line 147: | Line 139: | ||
Across all dynamic strategies, a reviewer is never assigned to: | Across all dynamic strategies, a reviewer is never assigned to: | ||
Their own team | *Their own team | ||
*A team they have already reviewed | |||
A team they have already reviewed | *A team outside the selected topic (for topic‑based assignment) | ||
A team outside the selected topic (for topic‑based assignment) | |||
Latest revision as of 20:06, 15 March 2026
Overview
The review‑assignment subsystem in the reimplemented Expertiza supports both static and dynamic allocation of reviews within an assignment. The goal is to give instructors flexible control over how reviewers are matched to submissions, while also supporting on‑demand, fairness‑aware assignment for students who request reviews dynamically.
This page documents the behavior implemented in ReviewMappingsController and the strategy classes under ReviewMappingStrategies.
Static Review Assignment
Static assignment is initiated by the instructor and produces a complete set of review mappings in a single operation. Three static strategies are supported.
Round‑Robin Assignment
Controller action: assign_round_robin
This strategy cycles through all reviewers in order and assigns each team to the next reviewer in the cycle. Only participants who are eligible to review (can_review?) are included.
Characteristics:
- Deterministic ordering
- Each team receives exactly one reviewer
- Reviewers are reused as needed
- No workload balancing beyond the simple cycle
Random Assignment
Controller action: assign_random
This strategy assigns reviewers to teams randomly. The distribution is determined by the logic inside ReviewMappingHandler.
CSV‑Based Assignment
Controller action: assign_from_csv
This strategy imports reviewer–team pairs from an uploaded CSV file.
Expected CSV format:
reviewer_email,team_name
For each row:
- The reviewer is located by email and matched to an AssignmentParticipant.
- The team is located by name within the assignment.
- A mapping is created if both are found.
This allows instructors to pre‑compute assignments externally and import them directly.
Dynamic Review Assignment
Dynamic assignment is initiated by a reviewer requesting a new review. Instead of pre‑assigning all reviews, the system allocates reviews on demand, which helps avoid the common problem of students failing to complete their assigned reviews.
Two dynamic strategies are supported.
Least‑Reviewed Submission Strategy
Controller action: request_review_fewest Strategy: LeastReviewedSubmissionStrategy
This strategy assigns the reviewer to the team whose submission has received the fewest completed reviews so far.
Eligibility rules:
- A reviewer may not be assigned to their own team.
- A reviewer may not be assigned to a team they have already reviewed.
- Among eligible teams, the one with the lowest review count is selected.
This ensures a balanced distribution of reviews across submissions.
Topic‑Balanced Dynamic Assignment
Controller action: request_review_topic_balance Strategy: LeastReviewedTopicStrategy
When an assignment uses topics, reviewers may choose which topics they are willing to review. The system then assigns a team within an eligible topic, subject to fairness constraints.
Algorithm:
Review counts are aggregated per topic. Let min_count be the smallest number of reviews received by any topic. A topic is eligible if its review count is within k of min_count. The first eligible topic (deterministically) is selected. Within that topic, the reviewer is assigned to a team that:
- is not their own team,
- has not already been reviewed by them,
- has the fewest reviews among eligible teams.
This ensures:
- Reviewers can restrict themselves to topics they prefer.
- Topics remain approximately balanced, with fairness controlled by the threshold k.
Calibration Review Assignment
Calibration Round‑Robin
Controller action: assign_calibration_artifacts
This assigns calibration reviews to all reviewers using a round‑robin strategy. Calibration reviews are typically instructor‑provided artifacts used to train or benchmark reviewers.
Deleting Review Mappings
Delete a Single Mapping
Controller action: destroy
Deletes a specific review mapping by ID.
Delete All Reviews for a Reviewer
Controller action: delete_all_for_reviewer
Removes all review mappings associated with a given reviewer. Useful when resetting or reassigning work.
Instructor Grading of Reviews
Controller action: grade_review
Instructors may grade individual reviews. This action:
- Locates the review mapping
- Records the instructor’s grade and comment
- Returns a success message
Strategy Classes
All strategies inherit from ReviewMappingStrategies::BaseStrategy, which defines two abstract methods:
- each_review_pair — for static strategies
- assign_one — for dynamic strategies
Each concrete strategy implements one of these depending on its purpose.
CsvImportStrategy
Implements each_review_pair by reading reviewer–team pairs from a CSV file.
LeastReviewedSubmissionStrategy
Implements assign_one by selecting the eligible team with the fewest reviews.
LeastReviewedTopicStrategy
Implements assign_one with topic‑level fairness constraints and a threshold k.
RoundRobinStrategy
Implements each_review_pair by cycling through reviewers and pairing them with teams.
Common Eligibility Rules
Across all dynamic strategies, a reviewer is never assigned to:
- Their own team
- A team they have already reviewed
- A team outside the selected topic (for topic‑based assignment)