CSC/ECE 517 Spring 2020 - E2021. Allow reviewers to bid on what to review

Motivation

Currently, the projects to review are assigned to the reviewers on a first come first serve basis. This policy does not let the reviewers investigate and learn from the projects they are specifically interested in. Therefore, the idea here is to let the students decide on what projects they wish to review in a fair manner by placing a bidding policy.

Problem Statement

The reviews performed by the students right now are being assigned on a first-come-first-serve basis. The task at hand is to implement a procedure so that the students can bid on the reviews that they would like to review, similar to the procedure of the assignment of the topics. The difference between the two procedures is that the assignment of the topic is a one-one mapping, i.e. a team is assigned to a single topic, whereas the assignment of the reviews is many-many mapping with each review being assigned to different students and each student assigned to different reviews.

The need for a bidding system for reviews

The bidding process, when implemented for reviews of projects, will hold several advantages.

1. The users will be able to bid for reviews that interest them. This will incentivize them to get a better understanding of the project requirements and provide the necessary feedback in an informed manner.

2. Bidding provides the users receiving feedback a very informed report on how to enhance their project. This will help them get better results for their respective projects.

3. The bidding also gives an advantage to the instructor because the instructor will now be able to receive more well-written, useful and meaningful feedback that will be more useful in their academic projects.

What needs to be done

The following are the tasks that are to be done-

1. Implement top trading cycles on a web service as mentioned in the paper [1].

The top trading cycles is an intelligent algorithm that allows users to achieve the review they will need. For example, let us consider a situation where user 'A' wants the review assigned to user 'B' and user 'B' wants the review assigned to user 'A'. Considering that it will motivate the students if they get the project that they would like to review, this mutual exchange between them would be beneficial. It can be solved using the top trading cycles algorithm.

Algorithm Steps

1) Let the user bid, these bids will point to the reviews they are requesting for in the graph.

2) There will be cycles links of length 1 between the person who owns ever review currently, a directed acyclic graph.

3) We will need to break all the cycles in the graph and assign the reviews to which the user is pointing at. The top is represent as [ TOP(i) ]

4) We can repeat this order until we have exhausted our preference list where all the assigned houses are constantly deleted.

5) we will repeat this bidding process for all slot of the bidding until the user gets all this review slots filled, making it many to many.

From the diagram we can see that the three users are represented as u1, u2, u3 and the the three reviews are r1,r2,r3.

In phase 1 : all users want the review 1 so all of them point to r1 and we see r1 is owned by u1 and r2 is owned by u2 and r3 by u3.

In phase 2 : u1 is assigned r1 and now u2 points to r3 which is now its highest priority because r1 is removed and similarly u3 points to r2.

In phase 3: r2 is assigned to u3 and and r3 is assigned to u2, thus satisfying all.

2. Add the front end code to allow the bidding on topics and call the appropriate web services from the lottery controller. This is similar to the already implemented bidding process on topics [2].

flow chart for this us functionality would involve :

Previous Work

The latest implementation of the problem was E1986 [3][4].

We can see that they have figured out the issues with the bidding assignement being one to one and not many to one and have succesfully used the Gale-Shapely version for many-to-many matching algorithm to solve the issues. They have also fixed a few problems with the UI that were being faced with the buttons and writing proper test cases.

From the previous implementations we can see that the following files were modified in order to get the required implementation they have modified the following files :

Modified Files

routes.rb

app/views/student_task/view.html.erb

app/views/assignments/edit.html.erb

app/controllers/student_task_controller

New Files

app/controllers/review_bidding.rb

app/models/review_bidding_controller.rb

app/views/sign_up_sheet/review_bid.html.erb

Source Code for the Web Service

Problems found in previous work

The problems in the previous work were given-

1. The link to the run bidding assignment is on the Review Strategy tab instead of the actions associated with an assignment like the topic assignment. This is an inconsistency in the UI that needs to be remedied.

2. All of the implementation code was put in the controller which is inappropriate for a controller. The code should probably be in the corresponding model file.

3. The tests that they have written are not adequate.

Suggested Improvements

1. Change the location of the running bidding assignment to the appropriate location. This is because it is inconsistent with the UI and will require changes.

The part of the code that requires modification is : app/views/assignments/edit.html.erb

As we can see from the code attached we need to move this code away from the Review Strategy tab.

2. Write the code where it is suitable, following the DRY principles.

3. Write suitable tests to ensure that the implementation is working.

4. Make sure the reviewer can't bid for his own project.

Plan for Testing

As of now, and as far as we understood the feature.

There are a few cases that need to be tested.

1) The first case would be to test for the scenario, where no reviewer bids i.e. no one has given a preference of any sort. Then the bidding algorithm should assign reviews randomly.

2) The other case would be when all reviewers have selected a different project as their preference, at different times. This is an ideal scenario given that the reviewer cannot bid for his own project.

3) The other scenario would be when everyone selects the same project at the same or different times and doesn't provide any other choice or preference. This can be the scenario if one project is relatively interesting or easy to review in comparison to the others in the roster.

4) Another scenario would be if some projects haven't been selected at all for reviewing, this case arises if the particular project may require additional effort on the part of the reviewing to successfully complete the review.

5) A usual scenario is that the reviewers select all or most of the projects, and each project is on at least one reviewers biding list.

In all of the above cases, all the projects have to be assigned to at least one reviewer so that every project has been peer-reviewed.

References

[1] School Choice: A Mechanism Design Approach by Atila Abdulkadiroğlu and Tayfun Sönmez.

[2] Bidding Interface Implementation.

[3] E1986 Pull Request.

[4] E1986 Wiki page.