CSC/ECE 517 Spring 2020 - E2021. Allow reviewers to bid on what to review

Problem Statement

The reviews performed by the students right now are being assigned on a first-come-first-serve basis. The task at hand is to implement a procedure so that the students can bid on the reviews that they would like to review, similar to the procedure of the assignment of the topics. The difference between the two procedures is that the assignment of the topic is a one-one mapping, i.e. a team is assigned to a single topic, whereas the assignment of the reviews is many-many mapping with each review being assigned to different students and each student assigned to different reviews.

The need for a bidding system for reviews

The bidding process, when implemented for reviews of projects, will hold several advantages.

1. The users will be able to bid for reviewing the projects that would interest them. This will incentivize them to get a better understanding of the project requirements and provide the necessary feedback in a very informed manner.

2. With the help of color coding, the students will be able to decide on the probability of being assigned to a particular review, that could help them in bidding wisely.

3. It provides the team members working on the project, a very informed feedback on how to enhance their project. Because,the feedback is from people who really are interested on the project.

4. It also helps the instructor in receiving more well-written, useful and meaningful feedback on what projects are more in demand among the students, which would be further useful in crafting the academic projects.

What needs to be done

The following are the tasks that are to be done-

1. Implement Gale-Shapely version for many-to-many matching situations on a web service as mentioned in the paper [1].

The Gale-Shapely version for many-to-many matching is an intelligent algorithm that allows users to achieve the review they will need. For example, let us consider a situation where user 'A' wants the review assigned to user 'B' and user 'B' wants the review assigned to user 'A'. Considering that it will motivate the students if they get the project that they would like to review, this mutual exchange between them would be beneficial. This classical problem can be solved using the following algorithm.

Motivation

From a set R of available review topics, the user u from set U have to be matched with a pre-configured number of topics r1,r2,r3.....rk. So,the idea is to record the user priority and time-stamp at which r is bid. This information can then be used to change the color of the review in real time, which could further let the users know that their bid for this particular bis cannot be given a top priority. Once the bidding timeline is past, the recorded information for each bis is then fed to the algorithm, which would then match user u to review r1,r2,r3...rk.

How should the algorithm match students S to the Review set R  ?

The algorithm has to initially append additional reviews randomly in case, the user did not bid for the required number of reviews. Then based on the input data of the list for each student, which contains the topic,time-stamp and priority it has to allot the review to a student with earliest time stamp and highest priority.

Algorithm Steps

1) Let the user bid, these bids will have time-stamps and priority(determined by the color of the bid at that time stamp).

2) There will be cycles links of length 1 between the person who owns every review currently,which generates a directed acyclic graph.

3) We the will need to break all the cycles in the graph and assign the reviews to which the user is pointing at. The top is represent as [ TOP(i) ]

4) We can repeat this order until we have exhausted our preference list where all the assigned reviews are constantly deleted.

5) We will repeat this process for all the slots of the bidding until the users gets all these review slots filled, making it many to many.

From the diagram we can see that the three users are represented as u1, u2, u3 and the the three reviews are r1,r2,r3.

In phase 1 : All users want the review 1 so all of them point to r1 and we see r1 is owned by u1 and r2 is owned by u2 and r3 by u3.

In phase 2 : u1 is assigned r1 and now u2 points to r3 which is now its highest priority because r1 is removed and similarly u3 points to r2.

In phase 3 : r2 is assigned to u3 and and r3 is assigned to u2, thus satisfying all.

2. Add the front end code to allow the bidding on topics and call the appropriate web services from the lottery controller. This is similar to the already implemented bidding process on topics [2].

Flow chart for this us functionality would involve :

Previous Work

The latest implementation of the problem was E1986 [3][4].

We can see that they have figured out the issues with the bidding assignement being one to one and not many to one and have succesfully used the Gale-Shapely version for many-to-many matching algorithm to solve the issues. They have also fixed a few problems with the UI that were being faced with the buttons and writing proper test cases.

From the previous implementations we can see that the following files were modified in order to get the required implementation they have modified the following files :

Modified Files

routes.rb

app/views/student_task/view.html.erb

app/views/assignments/edit.html.erb

app/controllers/student_task_controller

New Files

app/controllers/review_bidding.rb

app/models/review_bidding_controller.rb

app/views/sign_up_sheet/review_bid.html.erb

Source Code for the Web Service

Problems found in previous work

The problems in the previous work were given-

1. The link to the run bidding assignment is on the Review Strategy tab instead of the actions associated with an assignment like the topic assignment. This is an inconsistency in the UI that needs to be remedied.

2. All of the implementation code was put in the controller which is inappropriate for a controller. The code should probably be in the corresponding model file.

3. The tests that they have written are not adequate.

4. It does not allow the Instructor, to decide to whether allow the bidding for reviews or not.

Suggested Improvements

1. Change the location of the running bidding assignment to the appropriate location. This is because it is inconsistent with the UI and will require changes.

The part of the code that requires modification is- app/views/assignments/edit.html.erb

As we can see from the code attached we need to move this code away from the Review Strategy tab.

File:Locationchange.PNG

2. Write the code where it is suitable, following the DRY principles.

The first file where we want to apply the principle to is the: app/controllers/review_bidding_controller.rb

The code is shown below:

From this, we can see the entire method can be moved to the model to reduce the stress on the model. The assignment participant code is moved to the controller thereby making it easier to perform seperations of the model,view and contoller. Also in the below code attached we try to move the entire reviewer_bidding_data to the model as well.

The implemented changes by moving the code to the model is shown in the code attached below. WE can see that both the method have been successfully moved to the model and thereby reducing the stress on the controller.

In the same file, we notice the method: get_quartiles(topic_id) which assigns colors to the topics based on how much in demand they are.

We would like to again reduce the assignment statements and move the entire if structure to the model to return the colors reducing stress on the controller.

The next function in this file is set_priority, where we want to rewrite the if structure because we fell its too long and has a very convoluted structure to it. The code shown below clearly shows the multiple nested if structs that we want to remove. Having multiple if structs could fail the code at unexpected edge points.

3. Write suitable tests to ensure that the implementation is working.

The previous implementation gave their try on implementing the required test but all of them are commented out. So, they are required to be redone during this implementation.

4. Make sure the reviewer cannot bid for his own project.

This was done in the previous implementation in both using the set_priority and review_bid methods, but the structure is pretty complex to these and we would like to work on reducing the complexity of these functions and also moving parts of these codes to the model.

Plan for Testing

As of now, and as far as we understood the feature.

Test Cases

There are a few cases that need to be tested.

1. The first case would be to test for the scenario, where no reviewer bids i.e. no one has given a preference of any sort. Then the bidding algorithm should assign reviews randomly.

2. The other case would be when all reviewers have selected a different project as their preference, at different times. This is an ideal scenario given that the reviewer cannot bid for his own project.

3. The other scenario would be when everyone selects the same project at the same or different times and does not provide any other choice or preference. This can be the scenario if one project is relatively interesting or easy to review in comparison to the others in the roster.

4. Another scenario would be if some projects haven't been selected at all for reviewing, this case arises if the particular project may require additional effort on the part of the reviewing to successfully complete the review.

5. A usual scenario is that the reviewers select all or most of the projects, and each project is on at least one reviewer's biding list.

In all of the above cases, all the projects have to be assigned to at least one reviewer so that every project has been peer-reviewed.

References

[1] School Choice: A Mechanism Design Approach by Atila Abdulkadiroğlu and Tayfun Sönmez.

[2] Bidding Interface Implementation.

[3] E1986 Pull Request.

[4] E1986 Wiki page.