CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

Project Overview

Background

Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

Objectives

1. Implement top trading cycles on a web service.
2. Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
3. Add working test cases to test the new changes.
4. Code should be inclined to the Beta branch
5. Link to run bidding assignment should be on the page associated with the assignment actions.
6. Code such as getter methods should be added to the model and not the controller.
7. The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

Previous Work

There have been two previous implementations:

• Fall 2019 here
• Spring 2020 here

Relevant Links

Github Repository: https://github.com/uahamedncsu/expertiza/tree/beta
Pull Request: https://github.com/expertiza/expertiza/pull/1822
Server: http://152.7.99.70:8081/
Video Demo: [Not Created]

Files Modified

This is a list of expected modifications to the code and will be updated as the implementation progresses.

Changed:
routes.rb

app/views/student_task/view.html.erb

app/controllers/student_task_controller

Added:
app/controllers/review_bidding.rb

app/models/review_bidding.rb

app/views/sign_up_sheet/review_bidding.html.erb

spec/controllers/review_bidding_controller_spec.rb

spec/models/review_bidding_model_spec.rb

Web Services files

Design

User Interface

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.

As discussed in the Project Overview section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking Has teams? and Has topics? in the General tab of an assignment then checking Allow reviewer to choose which topic to review? in the Topics tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for Allow reviewer to bid on topics to review?.

Analyzing the problem

Currently, the student can either pick a topic or is randomly assigned a topic. There is no Algorithm to do this selection.
In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select. This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.

Type of problem
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review.
This requires a Many to Many mapping.

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm.

In the real world, there are cases where we face this problem:

• Student choosing their future universities School Choice Problem (One-to-Many)
• Men and women seeking each other to create families Stable Marriage Problem (One-to-One)
• Internet services that need to be connected with users with the smallest amount of time

Algorithm Implementation for matching students and topics

The Gale Shapely Algorithm has steps to make sure it is a Stable marriage/allocation. An unstable allocation is when the participant does not like the topic assigned to them. But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1. This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

Assignment of priority

Every student bids on a number of topics of their choice with the highest priority topic on the top of their list. The time stamp of when the student had placed the bid is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority.

The recorded timestamp can be used to change the color of the Topic so that the participant can find out from the color that the topic is already chosen by someone and highest priority cannot be assigned for it. Any topics that were not selected by the participant are added to their preference list in a random preference order to create a situation where the number of students and number of topics is equal. When the algorithm starts, it matches each topic to students who have the topic as the highest priority and also based on the earliest timestamps so that the early bidder gets a preference.

Algorithm Steps

For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)

If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.

As this goes on, all students are matched even if all matches are not stable at this point.

In the previous round, if the participant did not get a topic and was assigned a different topic, In current round, an exchange can be done to get a desired topic if the other participant also needs the topic assigned to this participant.
In this case, a mutual exchange is done and this creates a stable matching.

The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

The time complexity of this algorithm is O(n^2)

Simulation

Implementation

This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

Overview

In order to create a successful review bidding system, the code will need its own model, view, and controller which will need to be created. The model and controller will contain the algorithm for the bidding process mentioned in the Algorithm section of Design and will include many of the new methods needed to implement the algorithm. The new model and controller will need related rspec testing files to be added to the code. A web service will be used carry out the top trading cycles which will generate new files. The 'routes.rb' file will need to be modified to include a route to this new bidding system. Likewise, the 'student_task' view and 'student_task' controller will need to be modified to account for the new bidding system's implementation. The full list of files expected to be changed or created can be found in the Files Modified section of the Project Overview.

Test Plan

This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

Testing Scenarios

This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either Manual Testing or Automatic Testing.

Basic Bidding Scenario

• Reviewers bid on multiple projects at different times
  • Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

Edge Cases

• All reviewers bid on the same project
  • If different time stamps: bidding algorithm should give priority to reviewers that bid first
  • If same time stamps: bidding algorithm should assign reviews randomly
• None of the reviews bid on any projects
  • Bidding algorithm should assign reviews randomly
• Reviewer bids on their own project
  • Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
• None of the reviewers bid on a specific project
  • Bidding algorithm should still assign reviewers to this project

Manual Testing

Manual testing should be preformed on an Expertiza server.

Manual Testing Plan

Manual testing should (at a minimum):

• test that the UI is implemented as expected
• test that a participant can bid on projects to review
  • test that a participant can move projects into the bidding column
  • test that a participant can reorder projects in the bidding column
  • test that a participant's bidding preferences save

Directions for Manual Testing

1. Go to testing server
2. Log in as username: instructor6 password: password
3. Go to Assignments
4. Two tasks should be listed in the main assignments box

a. Go to E2085 Manual Testing- Instructor6 Reviewer review

1. This UI is based on the user being a reviewer (i.e. user was not in a team and did not submit anything)
2. The UI should allow the user to bid on any project

b. Go to E2085 Manual Testing- Instructor6 Participant review

1. This UI is based on the user being a participant (i.e. user was in a team and assigned a topic)
2. The UI blocks user from bidding on their own project (instructor6 was assigned topic4)

Manual Testing Demo Video

Video demonstration of manual testing: link to be added

Automatic/RSpec Testing

RSpec tests will need to be written to related controller(s) and model(s).

RSpec Testing Plan

These RSpec tests should (at a minimum):

• test basic functionality of controller(s)
• test validation of the model(s)
• test the scenarios (basic case & edge cases) mentioned above

Directions for RSpec Testing

1. Set up Expertiza on personal device using forked repo
2. Run the following commands:

# related controllers: 
rspec spec/controllers/

# related models: 
rspec spec/models/

RSpec Testing Demo Video

Video demonstration of RSpec tests passing: link to be added

Team

Ummu Kolusum Yasmin Ahamed Adam (uahamed)
Ryan Grainger (rpgraing)
Luis Delossantos (lgdeloss)
Colleen "Bria" Engen (ceengen)
Mentor: Saurabh Shingte (svshingt)

Resources

Previous Implementations: [1], [2]
Bidding Interface Implementation Pull Request: [3]