CSC/ECE 517 Fall 2020 - E2085. Allow reviewers to bid on what to review

Project Overview

Background

Assigning reviews to users is a complicated process. Currently, reviews are still assigned using a first-come-first-served basis. However, reviews could be bid on like how topics are bid on. This would involve matching multiple students to review a submission up to the maximum reviewers for a submission.The manual selection process can be replaced with an automated bidding approach for this purpose.

Objectives

1. Implement top trading cycles on a web service.
2. Add front end code to allow bidding on topics and call the appropriate web service from the lottery controller.
3. Add working test cases to test the new changes.
4. Code should be inclined to the Beta branch
5. Link to run bidding assignment should be on the page associated with the assignment actions.
6. Code such as getter methods should be added to the model and not the controller.
7. The number of reviews created by default needs to be checked. One way of achieving this is creating minimum reviews and then allowing users to request more reviews if required.

Previous Work

There have been two previous implementations:

• Fall 2019 here
• Spring 2020 here

Relevant Links

Github Repository for Expertiza: https://github.com/uahamedncsu/expertiza/tree/beta
Github Repository for Web service: https://github.com/uahamedncsu/IntelligentAssignment
Pull Request: https://github.com/expertiza/expertiza/pull/1822
Expertiza Server: http://152.7.99.70:8081/
Web Service Server: https://app-csc517.herokuapp.com/match_topics
Video Demo: https://www.screencast.com/t/GJNOcfa9Ln

Files Modified

This is a list of all the files added or changed in completing this project.

Changed:

• config/routes.rb
  

• config/webservices.yml

• app/controllers/student_review_controller.rb
  

• app/views/assignments/edit/_topics.html.erb
  

• app/views/student_task/list.html.erb
  

• app/views/student_task/view.html.erb
  

• spec/factories/factories.rb
  

Added:

• app/controllers/review_bids_controller.rb
  

• app/helpers/review_bids_helper.rb
  

• app/models/review_bid.rb
  

• app/views/review_bids/_all_actions.html.erb
  

• app/views/review_bids/_table_header.html.erb
  

• app/views/review_bids/_table_line.html.erb
  

• app/views/review_bids/_topic_names.html.erb
  

• app/views/review_bids/others_work.html.erb
  

• app/views/review_bids/show.html.erb
  

• db/migrate/20201107203204_create_review_bids.rb
  

• spec/controllers/review_bids_controller_spec.rb
  

• spec/models/review_bid_spec.rb
  

Design

User Interface

Currently, the users are offered the entire list of topics to review and are asked to select a topic to begin a review. Topics that cannot be reviewed or have already been reviewed the maximum number of times are greyed out and cannot be selected. There is also a check box in which the student can select "I don't care which topic I review" and will be randomly assigned a topic from the list. The current User Interface (UI) for this is shown below.

As discussed in the Project Overview section, a bidding system similar to the bidding system used in topic selection is preferable to the current review selection system. The UI for this process will mirror the UI for the topic bidding system. The new UI will allow users to drag a topic from a list of all the topics on the left, to their priority list on the right. The ordering of the right list indicates the priority of the users preference. The colors indicate the likelihood of being assigned that topic. By utilizing the same UI, the consistency of the user experience and the DRY-ness of the code are increased. An example of the topic selection bidding UI is shown below.

Currently, an instructor has the option while creating or editing an assignment to allow a reviewers to select the topics they want to review. This is done by checking Has teams? and Has topics? in the General tab of an assignment then checking Allow reviewer to choose which topic to review? in the Topics tab. This interface can be used for allowing review bidding either by editing the current check box or adding an additional check box for Allow reviewer to bid on topics to review?.

Analyzing the problem

Currently, the student can either pick a topic or is randomly assigned a topic, from among the ones that are currently eligible to be reviewed.
In this scenario, it is possible that some good topics are already picked by a lot of students and if someone is late to do the selecting, they miss out on it since it is grayed out.
To solve the above problem of not getting the topics they like, the participants must be assigned topics based on a list of topics they pre-select. This is where an algorithm will come in handy. For picking a suitable algorithm, we should first identify the type of the problem.

Type of problem
Each student should be able to bid on multiple assignments of their choice and the same assignment can be assigned to multiple participants for review.
This requires a Many to Many mapping.

The algorithm of choice for this type of stable allocations is Gale-Shapely algorithm.

In the real world, there are cases where we face this problem:

• Student choosing their future universities School Choice Problem (One-to-Many)
• Men and women seeking each other to create families Stable Marriage Problem (One-to-One)
• Internet services that need to be connected with users with the smallest amount of time

Algorithm Implementation for matching students and topics

The Gale-Shapely Algorithm has steps to make sure it is a Stable marriage/allocation. An unstable allocation is when the participant does not like the topic assigned to them. But it can happen that the participant 1 might like the review topic assigned to participant 2 and participant 2 likes the topic assigned to participant 1. This is the unstable case. To make it stable, the topics need to be exchanged between these 2 participants so that both of them get the topic they wanted.

Source: Stable Marriage Algorithm

Assignment of priority

When a student places a bid, the timestamp is recorded. Priority will be given to the earlier bidder. If the student had chosen only very few bids, then this person will be given a higher priority. Any topics that were not selected by the participant are added to bottom of their preference list in a random preference order to create a situation where the number of students and number of topics is equal.

Algorithm Steps

For the first round, the Algorithm will assign topics to each student based on the Priority and each topic is assigned to a limited number of students based on the Limit. It picks one topic and begins assigning students to it. (Note: The Algorithm makes sure the participant is not in a group that worked on this particular topic.)

If the one topic is assigned to enough participants already, then it is removed from the queue and the same process is repeated for the next topic.

As this goes on, all students are matched even if all matches are not stable at this point.

The Algorithm terminates when all Topics that have not reached the maximum assignment level have tried to accept every student.

The time complexity of this algorithm is O(n^2)

Simulation

In the simulation above, we have equal number of Students (S1, S2, S3) and Review topics to assign (R1, R2, R3). The students preference of topics is to left of student. The Review topics does not have any preferences. To have preferences, the timestamps are used to determine which student bid on each topic first to create such a Preference list for topics.

1. Phase 1: Both Students S1 and S2 have selected R2 as their highest preference. Since there is a conflict, The preference of R2 is checked which shows that S2 had bid earlier than S1. Thus, the topic R2 is assigned to student S2.
2. Phase 2: Now, S1 and S3 both want to review R1 but R1 was first bid on by S1 as shown in the preference. Hence, R1 is assigned to S1.
3. Phase 3: The only topic remaining in this case is R3 which is also on the preference list of Student S3. Therefore S3 is assigned the topic R3.
4. By the end of the 3 phases, All students are assigned to a Review. The same process needs to be repeated till the required number of reviews are assigned to each student. In each iteration, if a review is assigned to max number of students already, the review is removed from consideration.

Implementation

This section details the implementation of the code and will highlight each component of the design in the code and its intended purpose, as well as any modifications or testing. As the implementation progresses this section will be updated.

Overview

This project was split into 2 objectives, the Expertiza MVC implementation and the webservice algorithm. The Expertiza MVC follows the MVC standard such that the controller handles directing the users to the correct views, as well as handling the communication with the webservice. The model performs all necessary computations and data handling like the creation and storing of bids on reviews and assigning reviews to students. The views render the corresponding webpage based on the actions of the user. The webservice contains the algorithm outlined in the Algorithm Implementation section of Design. Our Expertiza controller sends this webservice bidding information and the algorithm returns the matched topics to be assigned.

Controller

The controller contains 6 methods that perform all the necessary functions. The first method, "action_allowed?" performs authentication checks The second, "index", is used to redirect the user to a list of assigned reviews after the algorithm has been run. The third, "show", is used in conjunction with fourth, "set_priority", to update and store the bids when users make changes in the bidding page. The fifth, "assign_bidding", utilizes the sixth, "run_bidding_algorithm", to connect to the webservice, send and receive the correct bidding information for the algorithm.

Method: action_allowed?

This method performs authentication by allowing only certain types of users to access controller methods. The first portion of this method allows users 'Instructor', 'Teaching Assistant', 'Administrator', 'Super-Administrator', and 'Student' to access the methods 'show', 'set_priority', and 'index'. These are the methods that render the bidding pages and review listings. The other methods are restricted for 'Student' users.

Method: index

This method collects all the necessary information about the user, Assignment details, and Review Mapping details to render the 'review_bids/others_work', which lists the reviews that are assigned to the user. This page also includes a button to request more reviews which will add reviews up to the maximum number of reviews. To see the view rendered by this method, please go to "others_work".

Method: show

This method renders the bidding page in which participants can select from a table of topics the topics they would prefer to review. It collects the necessary information about the user, Assignment, Topics, and current Review Bids to render this page. It is used in conjunction with set_priority where the users bidding information is stored, and the page updated. To see the bidding page that this function renders, visit the view "show"

Method: set_priority

In the view file "show" there exists a line the calls this method when the page is updated or accessed. This method stores the user's bidding information in the schema database. This allows the model to access this information when collecting all the bidding information, as well as allowing users to refresh or return to this page with their selections staying in the bid table.

Method: assign_bidding

This method utilizes methods in the model, as well as the method "run_bidding_algorithm" to collect the bidding information of all students of a particular Assignment, send it to the webservice algorithm, store the matched topics that are returned, and pass these matched topics to the model to be assigned. After this method is run, the students should have assigned reviews and bidding should be turned off.

Method: run_bidding_algorithm

This method connects to the webservice using RestClient API, which is part of the 'require's at the top of the Controller. It will return a false if the connection fails for ay reason such as connection is not established or the algorithm has an internal error, otherwise it returns the matched topics from the algorithm as a 'json'.

Model

The model contains 5 methods that are utilized by the controller to save bidding information per user or all users, retrieve bidding information per user or for all users, and assign topics to review to participants.

Method: get_bidding_data

This method is called by the "assign_bidding" controller method. It receives the Assignment ID and a list of reviewers, which are student IDs. It creates a Hash to store bidding data into and iterates through all of the reviewers, calling "reviewer_bidding_data" to get each students bidding data and store it in the Hash. It returns a Hash with all the bidding data needed for the Webservice Algorithm to run.

Method: assign_review_topics

This method is called by the "assign_bidding" controller method. Given the Assignment ID, list of reviewers, and the matched topics returned by the webservice algorithm, it first clears any previously assigned reviews for that Assignment to prevent multiple assigns, then iterates through each reviewer and assigns them to the topics that the algorithm selected for them.

Method: assign_topic_to_reviewer

This method is called by the method "assign_review_topics" and assigns a single topic to a single user to review given an Assignment ID.

Method: reviewer_self_topic

This method returns the topic id of the reviewer that is passed in. This method is utilized to retrieve the reviewers' Topic ID, which is needed for the webservice algorithm to prevent users from being assigned to review their own topic. This is also prevented in views, but this is an additional safety precaution.

Method: reviewer_bidding_data

This method is called by "get_bidding_data". It creates a hash to store all the reviewer's bidding information. It iterates through all the bids for a reviewer given the Assignment ID and it returns a Hash of the bidding data.

Views

There are two main views for this project, they are for bidding and for showing assigned reviews after the bidding algorithm has been run.

View: others_work.html.erb

This file renders the view where students see a list of the reviews they've been assigned, with appropriate links and related messages. It includes a "Request Additional Review" button that will allow students to view more reviews up to the max number of reviews allowed.





View rendered by above:

View: show.html.erb

This view file renders the Review Bidding page where participants can select which topics to review. It renders partials which are included below, and includes coloring effects from helper methods indicating likelihood of receiving that particular topic to review.




Page rendered by above: (This rendering is an instructor view, the student view is the same without the "Edit Topics" column.)

Webservice

app

The entry point to the Web service is the app.py file where we have the POST Method match_topics. This receives the input in a json format from Expertiza. This json will have a list of all topics, list of all students and the topics each student selected and also the maximum number of review topics per student. Also, it has the timestamp information of the bidding time to give preference to early bidders. The input json is passed to the JsonParser class to create a Dictionary.

json parser

The json parser converts the json into a Dictionary. The parser also creates a preferences dictionary for the Student to store their topic preferences and one for Topics to store list of topics which have been selected by students and also the priority for the selection using the Timestamps. The json parser does one more task of finding Popular topics. This is needed because topics that are in demand should be preserved for students who bid on it and to avoid assigning it to students who did not bid on anything. High demand topics are assigned a lower priority for such students so that there are other topics they can be assigned.

Class Topic

The Topic class is the one that does the proposing to the students during the runtime. There are 2 cases. In case 1 when the particular topic is asked by more students than the limit of number of students the one topic can be assigned to. In this case, it uses the priorities it already has in the dictionary to make sure the topic proposes only to those students who chose it and those who bid earlier. In Case 2, There are less students who asked for the topic. Hence, the topic can propose to all the students who asked for it and still be under the limit. In the end of this step. All topics make their proposals to be assigned to some students.

Class Student

Once the Topics propose to students, the Student has to accept some of the proposals. First, each student sorts the proposals based on the students preferences. Now, the student accepts all the proposals they can upto the number of slots remaining for them. The max slots for the student is determined by the max_proposals_accepted field in the json. Once the acceptance is complete, the Topics are all updated so that the topics can adjust their remaining slots accordingly.

Topic Matcher

Next, The Topics Matcher class is called which is the main class that runs the Algorithm. Before running the algorithm, it creates a list of Student and Topic objects by using instances of Student and Topic class. Each student and Topic now has its own class so that it can be used to make proposals for assignment of topics.

The get_student_topic_matches() method kicks off the Algorithm. Each topic proposes to students based on the topic priorities as covered in the Topics section. It uses a max_topic_assignment_limit (say 4 for example) to make sure the same topic is not assigned to too many students. Once the proposals are made, we iterate over students so that they can accept proposals as long as they have slots remaining to be assigned to.

The round is now complete. To determine if we need a next round, we ask the topics if they are done proposing using is_topics_done_proposing() method. As long as each topic has not reached its assignment limit and there are still students to propose to, the proposing continues for the next round. Once every topic is done, it breaks out of the loop and we now have the list of students with equal number of assigned topics ( 3-4 for example) and this list is returned back to Expertiza as the Stable matching to do the assignment.

Test Input

Sample Test case:
Case: There are a few very popular topics and the students who bid for them and bid early should receive it. Popular Topics: 3969, 3971, 3972

{

 "tid": [
   3969,3970,3971,3972,3973,3974,3975,3976,3977
 ],
 "users": {
   "36239": {
     "tid": [
       3970, 3972
     ],
     "otid": 3977,
     "priority": [
       2,1
     ],
     "time": [
       "Thu, 12 Nov 2020 12:01:06 EST -05:00","Thu, 12 Nov 2020 12:01:07 EST -05:00"
     ]
   },
   "36240": {
     "tid": [],
     "otid": 3976,
     "priority": [],
     "time": []
   },
   "36241": {
     "tid": [
       3969,3971,3972
     ],
     "otid": 3975,
     "priority": [
       1,3,2
     ],
     "time": [
       "Thu, 12 Nov 2020 12:00:22 EST -05:00","Thu, 12 Nov 2020 12:00:25 EST -05:00","Thu, 12 Nov 2020 12:00:27 EST -05:00"
     ]
   },
   "36242": {
     "tid": [
       3969,3971,3973
     ],
     "otid": 3974,
     "priority": [
       3,2,1
     ],
     "time": [
       "Wed, 11 Nov 2020 12:15:43 EST -05:00","Thu, 12 Nov 2020 11:59:40 EST -05:00","Thu, 12 Nov 2020 13:07:53 EST -05:00"
     ]
   },
   "36243": {
     "tid": [
       3971,3969,3970,3976
     ],
     "otid": 3972,
     "priority": [
       4,3,2,1,5
     ],
     "time": [
       "Wed, 11 Nov 2020 11:34:50 EST -05:00","Wed, 11 Nov 2020 12:30:16 EST -05:00","Wed, 11 Nov 2020 12:30:19 EST -05:00","Thu, 12 Nov 2020 13:02:02 EST -05:00"
     ]
   }
 },
 "max_accepted_proposals": 3

}

Output

Since there is competition for these topics, the algorithm recognizes this and makes sure to assign them only to the students who bid for it.

Other students who picked other topics or did not pick anything are assigned different topics.

{

 "36239": [
   3970,
   3972,
   3975
 ],
 "36240": [
   3973,
   3974,
   3972
 ],
 "36241": [
   3969,
   3971,
   3972
 ],
 "36242": [
   3969,
   3971,
   3973
 ],
 "36243": [
   3969,
   3970,
   3971
 ]

}

Test Plan

This section outlines the test plan for this project including testing scenarios/edge cases, the manual testing plan, and the automatic/RSpec testing plan.

Testing Scenarios

This section outlines the various bidding scenarios that could occur. These scenarios should be tested in either Manual Testing or Automatic Testing.

Basic Bidding Scenario

• Reviewers bid on multiple projects at different times
  • Bidding algorithm should give priority to reviewers that bid first and should assign based on reviewers ranking

Edge Cases

• All reviewers bid on the same project
  • If different time stamps: bidding algorithm should give priority to reviewers that bid first
  • If same time stamps: bidding algorithm should assign reviews randomly
• None of the reviews bid on any projects
  • Bidding algorithm should assign reviews randomly
• Reviewer bids on their own project
  • Bidding algorithm should not allow a reviewer to bid on their own project and should have validation to prevent assigning a reviewer their own project
• None of the reviewers bid on a specific project
  • Bidding algorithm should still assign reviewers to this project

Manual Testing

Manual testing should be preformed on an Expertiza server and webserver platform

Manual Testing Plan for Expertiza Server

• test that an instructor can allow review bidding for an assignment
• test that server routes correctly when review bidding is allowed
  • test that task box links to bidding page before algorithm is run
  • test that assignment->others work links to bidding page before algorithm is run
  • test that task box links to review page with bidding information after algorithm is run
  • test that assignment->others work links to review page with bidding information after algorithm is run
• test that the bidding UI is implemented as expected
• test that a participant can bid on projects to review
  • test that a participant can move projects into the bidding column
  • test that a participant can reorder projects in the bidding column
  • test that a participant's bidding preferences save
  • test that a participant can't bid for their own topic
• test that run algorithm button calls webserver with correct information and returns review assignments for each user

Directions for Manual Testing of Expertiza Server

1. Go to testing server
2. Log in as username: instructor6 password: password
3. Go to Manage -> Assignments -> E2085 Manual Testing -> edit (pencil symbol)

1. In the General tab make sure Has teams? and Has topics? are checked
2. In the Topics tab check Allow review to choose which topic to review? then click save
3. In the Topics tab select Bidding then click save

4. Go to Assignments

1. E2085 Manual Testing review task should be listed in the main assignments box

1. This UI is based on instructor6 being a participant assigned to topic Topic4
2. The UI blocks user from bidding on their own project
3. Check that user can bid on topics and reorder bids
4. Check that users topic preferences save on page refresh
5. Impersonate other participants and repeat bidding UI tests ('student7601, student7602, student7603, student7604, student7605, student7606, student7607, student7608, student7609)

5. Go to Manage -> Assignments -> E2085 Manual Testing -> edit (pencil symbol)

1. In the Topics tab click Run Review Algorithm button

6. Go to Assignments

1. E2085 Manual Testing review task should be listed in the main assignments box

1. E2085 Manual Testing review should now link to the reviewing page
2. The number of required reviews should already be displayed
3. A Request Another Review button should be present unless reviews listed is equal to the number of reviews allowed
4. Begin a review and test save and submit

Manual Testing Demo Video for Expertiza Server

Video demonstration of manual testing: here

Automatic/RSpec Testing

RSpec tests will need to be written to related controller(s) and model(s).

RSpec Testing Plan

These RSpec tests:

• test basic functionality of review bids controller
• test basic functionality of review bid model

Directions for RSpec Testing

1. Set up Expertiza on personal device using forked repo
2. Run the following commands:

# review bids controller: 
bundle exec rspec spec/controllers/review_bids_controller_spec.rb

# review bid model: 
bundle exec rspec spec/models/review_bid_spec.rb

RSpec Testing Demo Video

Video demonstration of RSpec tests passing: here

Team

Ummu Kolusum Yasmin Ahamed Adam (uahamed)
Ryan Grainger (rpgraing)
Luis Delossantos (lgdeloss)
Colleen "Bria" Engen (ceengen)
Mentor: Saurabh Shingte (svshingt)

Resources

Previous Implementations: [1], [2]
Bidding Interface Implementation Pull Request: [3]

Relevant Links

Github Repository for Expertiza: https://github.com/uahamedncsu/expertiza/tree/beta
Github Repository for Web service: https://github.com/uahamedncsu/IntelligentAssignment
Pull Request: https://github.com/expertiza/expertiza/pull/1822
Expertiza Server: http://152.7.99.70:8081/
Web Service Server: https://app-csc517.herokuapp.com/match_topics
Video Demo: https://www.screencast.com/t/GJNOcfa9Ln