CSC/ECE 517 Spring 2024/ OSS E2400 Allow Reviewers to Bid on What to Review
This wiki page provides information regarding the refactoring of E2400 review_mapping_controller.rb OSS project for Spring 2024, CSC/ECE 517
Introduction
The Expertiza project makes use of peer review to allow students to learn from one another. It's a Ruby on Rails-based open-source application. It is used by faculty and students for the management of courses and assignments for specific courses. Different screens in the manage content area of the application offer information about users, courses, assignments, questionnaires, and reviews.
A review bid is a system used in Expertiza, where students indicate their preferences for reviewing certain assignments or projects. Students bid on the tasks they prefer to review through ranking their interests. The allocation of review assignments is determined by a bidding algorithm, balancing student preferences with fair and efficient distribution.
Test Login Credentials
Problem Statement
- The process of reviewing plays a crucial role in the learning and improvement cycle, benefiting both the reviewer and the reviewee. Allowing reviewers to choose their topics freely can lead to more thorough reviews and increased engagement. However, the current system of assigning review topics on a first-come-first-served basis disadvantages students who are slower to sign up, preventing them from reviewing topics they are most interested in.
- An improved approach would be to introduce a bidding system for review topics, similar to the one used for topic selection. This system would enable students to express their preferences for topics they are keen to review. An effective bidding algorithm could ensure that all interested students receive a topic they are eager to review.
- While the prior work met the functional requirements, it could not be integrated due to numerous violations of the DRY (Don't Repeat Yourself) principle.
Background
When topics are opened up for bidding, students can see how “hot” each topic is by the color it has on their topic list. However, instructors have no way to view the bidding process except by impersonating students. Furthermore, when the bidding assignment algorithm is run, there is no way to verify that it did in fact assign teams to topics they had chosen.
Feature Overview
Current Topic Bidding Feature
Students may bid on topics they would like to work on. The following image shows the functionality of this feature.
Students are allowed to bid on several topics, ranking them by priority. When the bidding deadline passes, every student is allocated a topic based on their preferences.
This allocation process utilizes a modified Gale-Shapely Algorithm to ensure a stable match between students and topics, aiming to satisfy student preferences as much as possible. To resolve preference ties, topics prioritize students based on whether they are bidding as part of a group (with group members receiving higher priority) and the timing of their bid (earlier bids receive higher priority). After the bidding deadline:
- Students are initially paired with their top-choice topic.
- If a topic exceeds its capacity, it retains only the students with the highest priority.
- Topics that have reached capacity are then removed from the selection pool.
- Unmatched students repeat the process, selecting from the remaining topics.
- This cycle continues until every student has been assigned a topic.
- The algorithm checks for and corrects any 'unstable' matches, where swapping students between topics would more accurately fulfill their preferences.
This process, with a time complexity of O(n^2), ensures that all students are matched with a topic they prefer, respecting their choices as closely as possible. For further information on this algorithm and its application, one can refer to documentation from the previous year when this method was first implemented.
Preferred Features for Reviews
The following Use Case Diagram demonstrates the primary functionality explained in the previous section for different types of users.
The desired feature for reviews should have a nearly same layout as this, but with reviews rather than themes. Students should be presented a list of topics that have been worked on, and they should be able to bid on the ones they want to review the most. The Priority system might be implemented in the same way, with the exception that a student should not be able to review their own work.
This feature appears as an option when editing/creating a new assignment under the "Topics" area, as shown in the below.
When this option is enabled, students will be able to access a new view that lists of the topics from submitted assignments (except their own). The user can drag the topics from topics table from the left into the selection table. The user can rearrange the selected topics in the selection table based on the priority. The list of prioritized topics are used to run the bidding algorithm.
If the student has not selected any topic for review then the review_bidding algorithm will assign the required number of topics automatically after the given deadline to choose the topics to review.
Proposed Solutions
Review_bids_others_work is a DRY violation
The previously implemented code had DRY violation found here The below code is implemented to over come the DRY violation. The use of helper methods (display_topic_and_participant and display_assignment_reviews_info) modularizes the code. The code avoids repetition and enhances readability. These methods can be reused wherever similar functionality is required, reducing the need to duplicate logic. The code efficiently processes each review mapping by iterating over @review_mappings with a loop, eliminating the need for repeated markup or logic for each review's display. This approach is significantly more effective than individually coding the display for each review.
Future Plan of Action
Test Plan
- Manual Test
- UI testing of the implemented functionality to be done.
- No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.
- Automated Test
- TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
- When all the reviewers submit their preferences.
- When there are no preferences from any user.
- When few have their preferences and few don't.
- Edge cases
- Case 1: No reviewer submits a list of preferred topics to review : In this case all topics are assigned same bid score. Topics are assigned to users on the basis of "balanced assignment", first come first serve.
- Case 2: All reviewers submit exactly the same list of topics to review: It will be first come, first serve.
- Case 3: Number of topics exceed the number of teams: Some topics may not be reviewed, depending on the limits put by the conference. It is responsibility of administrator to add more teams in such a situation.
Things to implement
- Currently it doesn't work if some student does not bid. In this case, algorithm needs to be fixed to ignore anyone who didn’t bid, and let them choose from what’s left over.
- Make sure your code works on individual assignments, i.e., assignments where participants sign up for topics instead of teams. In this case, a team is created for each participant when they sign up. So the code should work for assignments to which either individuals or teams submit.
- To make the functionality more intuitive, include a message to say how many students are eligible to submit bids, how many have submitted their bids, when deadline for submitting bids is.
Demo Video
- We will create a video to demonstrate the feature.