CSC/ECE 517 Fall 2025 - E2563. Review calibration

This page contains information about E2563.Review calibration, which was a project in CSC517 Fall 2025.

Please see below for a description of the design of the project.

Background

Expertiza is a peer-review-based learning system where students submit their work and review the work of their peers. The quality and consistency of these peer reviews are critical for the system's effectiveness. Review calibration is a mechanism to train and evaluate students' reviewing capabilities. In a calibration assignment, students review pre-selected "gold-standard" submissions that have already been reviewed by the instructor or teaching staff. By comparing the student's review to the staff's review, the system can assess the student's competency as a reviewer before they begin reviewing their peers' work. This project involves implementing the calibration functionality within the new Expertiza reimplementation.

Problem Statement

The current reimplementation of Expertiza lacks a structured review calibration feature. Without this feature, instructors have no systematic way to ensure that students understand the review rubric and can provide consistent, high-quality feedback before they assess their classmates' submissions. This can lead to unreliable peer-review scores and diminish the educational value of the peer-review process. The goal of this project is to design and implement a comprehensive review calibration system integrated into the assignment workflow.

Requirements

Core Requirements

1. Calibration Tab: A new "Calibration" tab must be added to the instructor's "Create/Edit Assignment" view.

1. Participant Management: The calibration tab must allow the instructor to add and manage phantom participants (e.g., unctlt1, unctlt2) who will represent the "gold-standard" submissions.

1. Instructor Calibration Review: The instructor must be able to initiate a review process for these phantom submissions directly from the calibration tab, using a standard review interface to provide the benchmark reviews.

1. Calibration Report: A new report must be generated that displays, for each student reviewer: Their score on each calibration review.A comparison of their score against the instructor's score for the same submission.

1. Student Workflow Integration: The assignment workflow for students must be updated to include a calibration stage where they review the instructor-provided submissions before proceeding to the regular submission and peer-review phases.

Design Requirements

1. Front-End (TypeScript): The user interface for the Calibration tab and the Calibration Report must adhere to the provided design guidelines and be implemented in TypeScript as part of the reimplementation-front-end repository.

1. Back-End (Rails API): The backend logic for managing calibration data, phantom participants, and generating the report must be implemented as a Rails API in the reimplementation-back-end repository. New models, controllers, and routes must be designed to support these features.

1. Database: New database tables or modifications to existing ones will be required to store calibration-specific data, such as links between assignments, phantom participants, and benchmark reviews.

1. Code Quality: Code must not be copied from the old Expertiza repository without explicit permission. New code should follow Rails and TypeScript best practices.

Implementation

Will be updated later