CSC/ECE 517 Fall 2019 - E1986. Allow reviewers to bid on what to review

Why Should a Student bid for Reviewing Submissions?

If the choice of which submission to review is left to student itself, he/she would choose to review those topics which are related to their project or whatever they are highly interested in. This would benefit both the reviewer and reviewer because

From a Student's Perspective:

• Because he/she does not have to spend a lot of time understanding what is going on.
• The Student would be more focused on giving the review and hence judgment made tend to be more objective than subjective.
• This would improve the quality of reviews and will also be helpful for the reviewer.
• By implementing the color-coding feature, the student at any point knows which topic/s are more likely to get and he/she optimizes between "what they want" and "what they can get"
• Also, Bidding in Fun !!!!!!!

From the Instructor's Standpoint:

• Because opinions expressed in the reviews are a result of informed decisions, the inference made from the Machine Learning models trained on these data sets would make more sense.

Problem Statement

Currently Deployed Implementation

When an assignment participant wants to review others’ work, they will be asked to choose an assignment topic. The participant will be allowed to choose from among those assignment topics that have not been already assigned to 10 other participants. If they choose the ‘I do not care about the topic I review’ option, a random topic will be chosen from all the available topics and will be assigned to them. That is, the reviews are being allocated on a First Come First Serve basis.

Previous Work on this issue

E1856

E1928

What’s Deeply wrong in those implementations?

To understand what's wrong, first consider the differences between Teams Bidding for Assignments and Students Bidding for Reviews. Both of these are Matching or resource Allocation Problems.

The former can be modelled as a one-to-one matching problem. (i.e team and assignment has one-to-one correspondence).

However, Students and Reviews have many-to-many relationship (a student can choose multiple submissions for review and a submission can be given to multiple students for review)

The Mathematical formulation is itself wrong in E1856 and E1928 and they have used the below shown diagram to represent the relationship.

Why does the difference is the representation matter?

Since they have modeled the problem on the same lines, they have used the same version of the Gale-Shapley or Top Trading Cycles Algorithm used for one-to-many or one-to-one approach. Famous Problems Dealt on these lines.

Stable Marriage Problem (One-to-One)

School Choice Problem (One-to-Many)

Other Implementation pitfalls observed

E1928

• Once the bidding for review topics is done, the selections need to be saved to the database which is not happening and when the page is refreshed, the UI does not retain the bids.
• The button responsible for running the algorithm cannot be checked and it is a hunk-like icon.
• The button appears multiple times on the page.
• The algorithm needs to be implemented in the web service which should be ideally be used from the lottery controller which is not they have tried to implement. The entire code is written in Ruby.
• The algorithm needs to skip the review topic that the user has worked on. This Code already exists for topic and again written in the new implementation that needs to be integrated into the existing one and hence needs refactoring.
• Proper tests need to be written.

What needs to be done?

The participants should be able to bid for topics to review in the same way they bid for topics to work on.

Approach for matching students with topics

• Gale-Shapely is Proven to be a stable solution in one-to-one and one-to-many scenarios. While the literature on using Gale-Shapely for Many-to-Many scenarios is not very well defined as a generalized framework, they are a few resources similar to our case.

• The assignment topics each have a quota 10 of reviewers the topic can be assigned to at maximum. Each of the students have a quota 4 of topics they can be assigned at maximum. Gale and Shapely have proposed a mechanism called the Deferred Acceptance (DA) algorithm for finding a stable solution for one-to-one and many-to-one matching problems. The generalization of the DA algorithm other case of many-to-many matching with different preferences and quotas proposed by Freer et al. can be used as our bidding algorithm. Details can be found in the reference below.

Use Case on applying Gale-Shapely to a Many-to-Many scenario

• Although, we were asked to implement Gale-Shapely as a part of this project, the liberty of using any other alternative and a suitable algorithm is in discussion.

Functional Requirements

• Each participant should be able to submit a list of preferences in decreasing preference order for the topics they would like to review. The list may contain any number of topics.
• During bidding, the topics should be color-coded according to the number of participants who are contending for each topic. The colors should range from green to red with green representing a topic with a low number of bids and red representing a topic with a high number of bids.
• There should be a deadline before which all participants must submit their list of preferences.
• After the deadline, the course instructor should be able to run the bidding algorithm after which each participant will be assigned a list of topics to review. The number of topics assigned to a participant should be minimum(4,k) where ‘k’ is the number of topics in their preference list.
• Once the topics to review have been assigned, the participants should be able to see those topics they have been assigned and give feedback to the submission for each topic.
• If a participant has given feedback to less than 4 topics and chooses to review a submission, the topic with the minimum number of current reviewers will be assigned to that participant.

Non-functional Requirements

• There must exist a web service that takes the participants and their respective preference lists as input runs a bidding algorithm like top trading cycles and returns a map of the participants and the list of topics assigned to each participant.
• This web service must either be called from the LotteryController or a new controller can be created on the same (if Lottery Controller becomes too complex).

Implementing the Web Service

• 1) Python
• 2) Flask (Minimal MVC Framework for Python Backend)
• 3) JSON as the Data Serialization Format.
• 4) Deployment (AWS or VCL)
• 5) Apache Web Server or GUnicorn or Default Flask Server.

(Potential) Addition/Modification of files in expertiza

• 1) Lottery Controller
• 2) Review Mapping Controller
• 3) A New model to store the bidding information entered by the student.
• 4) Unit Test Files Associated
• 5) Views/Routes Associated

Test Plan

• Ensuring that the algorithm results in stable matchings. The concept of stability is defined above.
• Check whether the UI reflects the bids made by the user.
• Testing that the color-coding feature is working and appropriate.
• Check if a user is assigned 4 topics for review, even though he might have bid for any number of topics.
• Regression testing if the code is added in the lottery controller. If new model/controller is added then relevant tests will be included based on the proposal acceptance.
• Tests to ensure that each review topic gets minimum number of reviews.
• Tests to check whether the web service works as expected.

Project Mentor

Ramya Vijayakumar (rvijaya4@ncsu.edu)

Team Members

Yaswanth Soodini (ysoodin@ncsu.edu)

Sai Shruthi Madhuri Kara (skara2@ncsu.edu)

Ayushi Rungta (arungta@ncsu.edu)

Vivek Karri (vkarri@ncsu.edu)

Important references

InteligentAssignment

Many-to-Many-Matching-Problem-with-Quotas-by-Freer-and-Titova

Project_E1928._Allow_reviewers_to_bid_on_what_to_review

E1856_Allow_reviewers_to_bid_on_what_to_review