Reviewer Assignment implementation

Overview

The review‑assignment subsystem in the reimplemented Expertiza supports both static and dynamic allocation of reviews within an assignment. The goal is to give instructors flexible control over how reviewers are matched to submissions, while also supporting on‑demand, fairness‑aware assignment for students who request reviews dynamically.

This page documents the behavior implemented in ReviewMappingsController and the strategy classes under ReviewMappingStrategies.

Static Review Assignment

Static assignment is initiated by the instructor and produces a complete set of review mappings in a single operation. Three static strategies are supported.

Round‑Robin Assignment

Controller action: assign_round_robin

This strategy cycles through all reviewers in order and assigns each team to the next reviewer in the cycle. Only participants who are eligible to review (can_review?) are included.

Characteristics:

• Deterministic ordering
• Each team receives exactly one reviewer
• Reviewers are reused as needed
• No workload balancing beyond the simple cycle

Random Assignment

Controller action: assign_random

This strategy assigns reviewers to teams randomly. The distribution is determined by the logic inside ReviewMappingHandler.

CSV‑Based Assignment

Controller action: assign_from_csv

This strategy imports reviewer–team pairs from an uploaded CSV file.

Expected CSV format:

reviewer_email,team_name

For each row:

• The reviewer is located by email and matched to an AssignmentParticipant.
• The team is located by name within the assignment.
• A mapping is created if both are found.

This allows instructors to pre‑compute assignments externally and import them directly.

Dynamic Review Assignment

Dynamic assignment is initiated by a reviewer requesting a new review. Instead of pre‑assigning all reviews, the system allocates reviews on demand, which helps avoid the common problem of students failing to complete their assigned reviews.

Two dynamic strategies are supported.

Least‑Reviewed Submission Strategy

Controller action: request_review_fewest Strategy: LeastReviewedSubmissionStrategy

This strategy assigns the reviewer to the team whose submission has received the fewest completed reviews so far.

Eligibility rules:

• A reviewer may not be assigned to their own team.
• A reviewer may not be assigned to a team they have already reviewed.
• Among eligible teams, the one with the lowest review count is selected.

This ensures a balanced distribution of reviews across submissions.

Topic‑Balanced Dynamic Assignment

Controller action: request_review_topic_balance Strategy: LeastReviewedTopicStrategy

When an assignment uses topics, reviewers may choose which topics they are willing to review. The system then assigns a team within an eligible topic, subject to fairness constraints.

Algorithm:

Review counts are aggregated per topic. Let min_count be the smallest number of reviews received by any topic. A topic is eligible if its review count is within k of min_count. The first eligible topic (deterministically) is selected. Within that topic, the reviewer is assigned to a team that:

• is not their own team,
• has not already been reviewed by them,
• has the fewest reviews among eligible teams.

This ensures:

• Reviewers can restrict themselves to topics they prefer.
• Topics remain approximately balanced, with fairness controlled by the threshold k.

Calibration Review Assignment

Calibration Round‑Robin

Controller action: assign_calibration_artifacts

This assigns calibration reviews to all reviewers using a round‑robin strategy. Calibration reviews are typically instructor‑provided artifacts used to train or benchmark reviewers.

Deleting Review Mappings

Delete a Single Mapping

Controller action: destroy

Deletes a specific review mapping by ID.

Delete All Reviews for a Reviewer

Controller action: delete_all_for_reviewer

Removes all review mappings associated with a given reviewer. Useful when resetting or reassigning work.

Instructor Grading of Reviews

Controller action: grade_review

Instructors may grade individual reviews. This action:

• Locates the review mapping
• Records the instructor’s grade and comment
• Returns a success message

Strategy Classes

All strategies inherit from ReviewMappingStrategies::BaseStrategy, which defines two abstract methods:

• each_review_pair — for static strategies

• assign_one — for dynamic strategies

Each concrete strategy implements one of these depending on its purpose.

CsvImportStrategy

Implements each_review_pair by reading reviewer–team pairs from a CSV file.

LeastReviewedSubmissionStrategy

Implements assign_one by selecting the eligible team with the fewest reviews.

LeastReviewedTopicStrategy

Implements assign_one with topic‑level fairness constraints and a threshold k.

RoundRobinStrategy

Implements each_review_pair by cycling through reviewers and pairing them with teams.

Common Eligibility Rules

Across all dynamic strategies, a reviewer is never assigned to:

• Their own team
• A team they have already reviewed
• A team outside the selected topic (for topic‑based assignment)