CSC/ECE 517 Fall 2019 - E1986. Allow reviewers to bid on what to review
Why Should Participants bid for Reviews
If the choice of which submission to review is left to student itself, he/she would choose to review those topics which are related to their project or whatever they are highly interested in. This would benefit both the reviewer and reviewer because
From Student's Perspective:
- Because he/she does not have to spend a lot of time on understanding what is going on.
- The Student would be more focused on giving the review and hence judgement made tend to be more objective than subjective.
- This would improve the quality of reviews and will also be helpful for the reviewer.
- Also, Bidding in Fun !!!!!!!
From Instructor's Standpoint:
- Because opinions expressed in the reviews are a result of informed decisions, the inference made from the Machine Learning models trained on these data sets would make more sense.
Problem Statement
Current situation
When an assignment participant wants to review others’ work, they will be asked to choose an assignment topic. The participant will be allowed to choose from among those assignment topics that have not been already assigned to 10 other participants. If they choose the ‘I do not care about the topic I review’ option, a random topic will be chosen from all the available topics and will be assigned to them.
Previous Work on this issue
What’s Deeply wrong in those implementations
To understand what's wrong, first consider the differences between Teams Bidding for Assignments and Students Bidding for Reviews. Both of these are Matching or resource Allocation Problems.
The former can be modelled as a one-to-one matching problem. (i.e team and assignment has one-to-one correspondence).
However, Students and Reviews have many-to-many relationship (a student can choose multiple submissions for review and a submission can be given to multiple students for review)
The Mathematical formulation is itself wrong in E1856 and E1928 and they have used the below shown diagram to represent the relationship.
Why does the difference is representation matter
Since they have modeled the problem on the same lines, they have used the same version of Gale-Shapley or Top Trading Cycles Algorithm used for one-to-many or one-to-one approach. Famous Problems Dealt on these lines.
Other Implementation pitfalls
E1928
- Once the bidding for review topics is done, the selections needs to be saved to the database which is not happening currently that means when the page is refreshed, the UI does not retain the bids.
- The button responsible for running the algorithm cannot be checked and it is a hunk- like icon.
- The button appears multiple times on the page.
- The algorithm needs to be implemented in the web service which should be ideally be used from the lottery controller which is not they have tried to implement. The entire code is written in Ruby.
- The algorithm needs to skip the review topic that the user has worked on. This Code already exists for topic and again written in the new implementation that needs to be integrated into the existing one and hence needs refactoring.
- Proper tests need to be written.
What needs to be done
The participants should be able to bid for topics to review in the same way they bid for topics to work on.
Functional Requirements
- Each participant should be able to submit a list of preferences in decreasing preference order for the topics they would like to review. The list may contain any number of topics.
- During bidding, the topics should be color coded according to the number of participants who are contending for each topic. The colors should range from green to red with green representing a topic with a low number of bids and red representing a topic with a high number of bids.
- There should be a deadline before which all participants must submit their list of preferences.
- After the deadline, the course instructor should be able to run the bidding algorithm after which each participant will be assigned a list of topics to review. The number of topics assigned to a participant should be minimum(4,k) where ‘k’ is the number of topics in their preference list.
- Once the topics to review have been assigned, the participants should be able to see those topics they have been assigned and give feedback to the submission for each topic.
- If a participant has given feedback to less than 4 topics and choose to review a submission, the topic with the minimum number of current reviewers will be assigned to that participant.
Non-functional Requirements
- There must exist a web service which takes the participants and their respective preference lists as input, runs a bidding algorithm like top trading cycles and returns a map of the participants and the list of topics assigned to each participant.
- This web service must either be called from the LotteryController or a new controller can be created on the same (if Lottery Controller becomes too complex).
Bidding Algorithm
We have a many-to-many matching problem in our hand. The assignment topics each have a quota 10 of reviewers the topic can be assigned to at maximum. Each of the students have a quota 4 of topics they can be assigned at maximum. Gale and Shapely have proposed a mechanism called the Deferred Acceptance (DA) algorithm for finding a stable solution for one-to-one and many-to-one matching problems. The generalization of the DA algorithm other case of many-to-many matching with different preferences and quotas proposed by Freer et al. can be used as our bidding algorithm.
Test Plan
- Check whether the UI reflects the bids made by the user
- Check if a user is assigned a maximum of 4 reviews and not more than that even though he bids for more.
- Regression testing if the code is added in the lottery controller. If new model/controller is added then relevant tests will be included based on the proposal acceptance.
- Tests to ensure that each review topic gets minimum number of reviews.
- Tests for color coding
- Tests to check whether the web service functionality works as expected.
Important references
Many-to-Many-Matching-Problem-with-Quotas-by-Freer-and-Titova