CSC/ECE 517 Fall 2018/E1856 Allow reviewers to bid on what to review
Description
Students currently are able to “bid” for the projects that they want to do as an assignment. On the other hand, for reviewing others’ work, the policy that’s currently in use is called “first-come-first-serve”.
In this project, we update the “first-come-first-serve” policy for assigning projects to reviewers, to the “bidding” policy. In the backend, we implement the “top trading cycles” algorithm. And we modify the controller to meet the needs of the algorithm. At the front end, we create an UI design to present to the students their choices to select.
Introduction
The bidding policy for project topics that students want to work on is already implemented. Students can currently bid on project topics for their team assignment. This reduces a lot of possible conflicts in assigning topics to each team.
There’s also reviewing work for each assignment. Currently, the policy to assign the project to the reviewer is “first-come-first-serve”. Students that choose to review a project first will get that project.
However, this policy creates an issue — while reviewing students’ work, sometimes the same project is requested to be reviewed a lot of times, while some other projects are not requested as much. A similar bidding policy for assigning projects to reviewers can help students who want to review a project the most to be most likely to receive that project. The completion of this project will allow students to also bid on what projects they are interested in reviewing.
We need to alter the “first-come-first-serve” to the “bidding” policy. In this policy, students will be matched to review a submission up to the maximum reviewers.
The projects includes implementation of the top trading cycles algorithm on Expertiza. A UI design for the bidding front-end will also be developed for the end-users.
Current System Analysis
Regarding to the functionality of review mapping, it's necessary to describe some of the related models and controllers in expertiza and how they are organized to fully implement the functionality.
Models
1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.
2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams(Model Team).
3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.
4. After assignment submission, participants choose their preferred topics and response mappings between assignment(reviewed object), assignment team(reviewee) and assignment participant(reviewer) are created.
5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.
Workflow
For now, expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, method "add_reviewer" is invoked when students are trying to request a peer review. When it is done, a new mapping is created, that is why review assignment is in FIFO style.
If reviewers are allowed to bid on what to review, the procedure of topics bidding implemented by LotteryController is a good reference. Here is the basic workflow of topics bidding:
First of all, teams have different preference towards the topics. For example, there are three topics within an assignment, namely A, B and C. Therefore 4 teams W, X, Y, Z. Teams would have their own preference towards different topics:
W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C
This data structure is passed to backend service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to have a fair bidding over the topics and teams.
A similar bid can take place on peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.
Some discrepancies need additional attention.
1. Slots can be left unoccupied if no team is willing to response. However, every response needs to have at least some reviewers.
2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.
3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.
Important Files for Reference
1. Top Trading Cycles algorithm: [1]
2. Top Trading Cycles implementation: [2]
3. Peerlogic service: [3]
4. LotteryController: [4]
5. ReviewMappingController: [5]
Design
Almost all of the functionality for our project has already been implemented in the review portions of the Expertiza system. Because of this, we will approach this project by first using delegation pattern to add biding capability to the review mapping controller for choosing topics. Then we will duplicate the view for choosing project topics and modify it so that it interacts with the review mapping controller. We understand that this is not DRY code, so we will try to address this problem by seeing if there is any way we can modify the choose project topics page in order to reuse its code, yet still fulfill the functionality of review mapping.
New Model
To maintain the original functionality of Expertiza as well as to add review bidding, we decide to add a new model to handle the data of the bidding. Let's call it ReviewBid. ReviewBid contains the information of bidding assignment and its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm.
Here is the definition of ReviewBid:
| Field Name | Type | Description |
|---|---|---|
| id | int(11) | unique identifier for the record |
| reviewed_object_id | int(11) | this is foreign key to assignment |
| reviewer_id | int(11) | participant id |
| reviewed_topics | text | the list of topic ids that participants want to bid. The order of ids represents the preference of the participant |
| created_at | timestamp | |
| updated_at | timestamp |
Implementation
Challenges
Testing
Design Patterns
Model View Controller - Expertiza uses the Ruby on Rails framework which is set up to use the Model View Controller software pattern.
DRY Principle - We will be attempting to write our code in a dry way, that being that code within the system is not duplicated.