CSC/ECE 517 Fall 2018/E1856 Allow reviewers to bid on what to review
Description
Students currently are able to “bid” for the projects that they want to do as an assignment. On the other hand, for reviewing others’ work, the policy that’s currently in use is called “first-come-first-serve”.
In this project, we update the “first-come-first-serve” policy for assigning projects to reviewers, to the “bidding” policy. In the backend, we implement the “top trading cycles” algorithm. And we modify the controller to meet the needs of the algorithm. At the front end, we create an UI design to present to the students their choices to select.
Introduction
The bidding policy for project topics that students want to work on is already implemented. Students can currently bid on project topics for their team assignment. This reduces a lot of possible conflicts in assigning topics to each team.
There’s also reviewing work for each assignment. Currently, the policy to assign the project to the reviewer is “first-come-first-serve”. Students that choose to review a project first will get that project.
However, this policy creates an issue — while reviewing students’ work, sometimes the same project is requested to be reviewed a lot of times, while some other projects are not requested as much. A similar bidding policy for assigning projects to reviewers can help students who want to review a project the most to be most likely to receive that project. The completion of this project will allow students to also bid on what projects they are interested in reviewing.
We need to alter the “first-come-first-serve” to the “bidding” policy. In this policy, students will be matched to review a submission up to the maximum reviewers.
The projects includes implementation of the top trading cycles algorithm on Expertiza. A UI design for the bidding front-end will also be developed for the end-users.
Current System Analysis
Regarding to the functionality of review mapping, it's necessary to describe some of the related models and controllers in expertiza and how they are organized to fully implement the functionality.
Models
1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.
2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams(Model Team).
3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.
4. After assignment submission, participants choose their preferred topics and response mappings between assignment(reviewed object), assignment team(reviewee) and assignment participant(reviewer) are created.
5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.
Workflow
For now, expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, method "add_reviewer" is invoked when students are trying to request a peer review. When it is done, a new mapping is created, that is why review assignment is in FIFO style.
If reviewers are allowed to bid on what to review, the procedure of topics bidding implemented by LotteryController is a good reference. Here is the basic workflow of topics bidding:
First of all, teams have different preference towards the topics. For example, there are three topics within an assignment, namely A, B and C. Therefore 4 teams W, X, Y, Z. Teams would have their own preference towards different topics:
W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C
This data structure is passed to backend service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to have a fair bidding over the topics and teams.
A similar bid can take place on peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.
Some discrepancies need additional attention.
1. Slots can be left unoccupied if no team is willing to response. However, every response needs to have at least some reviewers.
2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.
3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.
Design
Almost all of the functionality for our project has already been implemented in the review portions of the Expertiza system. Because of this, we will approach this project by first using delegation pattern to add biding capability to the review mapping controller for choosing topics. Then we will duplicate the view for choosing project topics and modify it so that it interacts with the review mapping controller. We understand that this is not DRY code, so we will try to address this problem by seeing if there is any way we can modify the choose project topics page in order to reuse its code, yet still fulfill the functionality of review mapping.
The main idea of this design to a find a delegator to complete the review bidding so that no modification is needs for the original system and reuse the capability of it. Therefore, we need a new set of model, controller and view for this delegator.
New Model
To maintain the original functionality of Expertiza as well as to add review bidding, we decide to add a new model to handle the data of the bidding. Let's call it ReviewBid. ReviewBid contains the information of bidding assignment and its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participant to bid on assignment teams rather than topics for simplicity.
Here is the definition of ReviewBid:
| Field Name | Type | Description |
|---|---|---|
| id | int(11) | unique identifier for the record |
| team_id | int(11) | the work of team to be bid on |
| participant_id | int(11) | participant id |
| priority | text | participant's preference towards the team |
| created_at | timestamp | |
| updated_at | timestamp |
New Controller
Construct new new Controller ReviewBidController. This controller handles the following functionality:
1. Trigger the bidding by sending request to PeerLogic backend with proper parameters retrieved from ReviewBid records.
2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.
New Views
Here we try to refer to the pull request #778 to generate the view code for the review bidding.
Improve Original System
1. StudentReviewController: provides a topic-team bidding list to participants; and receives bidding requests from participants so that they can modify their preference.
2. View of Assignment: add a new option of review strategy so that the instructor is able to allow participants to bid on what to review; add a new icon on tree list of assignments to allow instructor to start bidding.
Bidding Algorithm
For each submission that has received bidding, we keep a record of the interested reviewers, with their preference level. When the bidding is finished, we use the bidding algorithm to assign each reviewer with a submission, according to their preference level, which are stored in each record. Each record is sorted according to their preferences. The most interested reviewer is ordered to be the first, the second most interested to be the next, etc.
For example, if there are three submissions A, B, C, and three reviewers D, E and F. Each reviewer chooses two submissions for their preferred review subject. After the bidding period is finished, we collect the following record list:
A: [D0, F0]
B: [E0, D1]
C: [E1, F1]
For each record that we have kept, we use the following method to match the submission with the reviewer:
1. We generally assign the submission to the most interested reviewer.
2. If there are several reviewers that are equally interested in this submission, we use random shuffling to assign the submission to randomly one of them. Each of the equally interested individual get equal chances to be given the submission for reviewing.
3. If an individual in the record has been assigned a submission from the previous record-matching, we exclude them from the current matching.
So in this example, for assignment A, D and F are equally interested in the submission. So both of them get into the pool for random shuffling. If D gets the chance to review A, then in choosing reviewers for record of B, D is excluded. And E, as the only and the most highest ranked reviewer, gets to review B. Similarly, In C, since E has been assigned a submission to review, F is the only one left, thus gets the submission.
Similarly, we have designed the topic bidding algorithm, for reviewers to bid for the topics that they are interested in reviewing. This serves as an alternative for the submission bidding, as one topic sometimes have several different submissions from different teams.
Implementation
Allow Instructor to set review strategy to be bidding:
After setting the review strategy to be bidding, participants are assigned with a default bidding list, ordered by topic's name.
Participants can drag the items up and own to alter the priority.
Instructors can start bidding by clicking the following icon.
After the bidding is done, participants can login and see what they are assigned with.
Challenges
Test Plan
1. When a participant hasn't reordered the bid, a default list is provided.
2. When a participant has reordered the bid, a priority list is provided.
3. When a participant is ask for a comprehensive bidding list include topics, teams, a bidding items list is provided.
Important Files for Reference
1. Top Trading Cycles algorithm: [1]
2. Top Trading Cycles implementation: [2]
3. Peerlogic service: [3]
4. LotteryController: [4]
5. ReviewMappingController: [5]