CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review
About Expertiza
Expertiza is a web application through which students can submit and peer-review learning objects (articles, code, web sites, etc). The Expertiza project is supported by the National Science Foundation. It is used in select courses at NC State and by professors at several other colleges and universities.
Similarity to previous implementation
A similar implementation was provided in the last semester was by a team who had implemented the TTC and the required review bidding functionality.
Link: CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review
However, their implementation had a few problems, notably:
- The color-coding feature was not implemented.
- The implementation entailed ordering by IDs of teams who did the topics (not a reasonable ordering).
- The content of some newly-added file are very similar with existing ones.
- The UI was not clean, it appeared to be disarrayed.
- There is only one list while topic bidding interface has two lists, one for the the available list of topics and the other list for topics that are selected for the bidding.
- There was no "add link to submission" in the bidding list.
- This implementation altered too many lines of existing Expertiza code.
- Many irrelevant tests were included.
While we would not be writing this implementation from scratch, we would instead be using their Pull Request and remove the possible defects that they had.
Problem statement
Students currently are able to “bid” for the projects that they want to do as an assignment. On the other hand, for reviewing others’ work, the policy that’s currently in use is “first-come-first-serve”. We would try to implement the Top Trading Cycles algorithm that assigns review topics to students in a priority.
The bidding policy for project topics that students want to work on is already implemented. Students can currently bid on project topics for their team assignment. This reduces the conflicts in assigning topics to each team. But to contradict the desired implementation, we will make use of student user instead of teams, since a single student bids on a multiple topics.
There’s also reviewing work for each assignment. Currently, the policy to assign the project to the reviewer is “first-come-first-serve”. Student who chooses to review a project, will get the same project based on priority.
However, this policy creates an issue — while reviewing a student's work, sometimes the same project is requested to be reviewed by many students, while some other projects are not in demand. A similar bidding policy for assigning projects to reviewers can help students who want to review a project the most to be most likely to receive that project. The completion of this project will allow students to also bid on what projects they are interested in reviewing.
The project includes implementation of the top trading cycles algorithm on Expertiza.
Current Implementation of the bidding
Regarding to the functionality of review mapping, it's necessary to describe some of the related models and controllers in expertiza and how they are organized to fully implement the functionality. It should be noted here that this is an implementation of the bidding for assignment topics.
Models
1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.
2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams(Model Team).
3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.
4. After assignment submission, participants choose their preferred topics and response mappings between assignment(reviewed object), assignment team(reviewee) and assignment participant(reviewer) are created.
5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.
Workflow
For now, expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, method "add_reviewer" is invoked when students are trying to request a peer review. When it is done, a new mapping is created, that is why review assignment is in FIFO style.
If reviewers are allowed to bid on what to review, the procedure of topics bidding implemented by LotteryController is a good reference. Here is the basic workflow of topics bidding:
First of all, teams have different preference towards the topics. For example, there are three topics within an assignment, namely A, B and C. Therefore 4 teams W, X, Y, Z. Teams would have their own preference towards different topics:
W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C
This data structure is passed to backend service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to have a fair bidding over the topics and teams.
A similar bid can take place on peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.
Some discrepancies need additional attention.
1. Slots can be left unoccupied if no team is willing to response. However, every response needs to have at least some reviewers.
2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.
3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.
Top Trading Cycles Mechanism
In the paper School Choice: A Mechanism Design Approach, author Atila Abdulkadiroglu and Tayfun Sonmez have described Top
1. Top Trading Cycles algorithm: [1]
Bidding policy (Stable marriage problem)
As part of this assignment, we have to analyse the implementation of the stable marriage problem which is a stable sorting algorithm. The explanation for this algorithm is as follows which as been taken from "College Admissions and the Stability of Marriage (1962)" by D. Gale and L. S. Shapley.
A certain community consists of n men and n women. Each person ranks those of the opposite sex in accordance with his or her preferences for a marriage partner. We
seek a satisfactory way of marrying off all members of the community. Imitating our earlier definition, we call a set of marriages unstable (and here the suitability of the
term is quite clear) if under it there are a man and a woman who are not married to each other but prefer each other to their actual mates.
Definition: An assignment of couples will be called unstable if there are two men α and β who are married to women A and B, respectively, although β prefers A to B and A prefers β to α.
So, we can ask the question: For any pattern of preferences is it possible to find a stable set of marriages?
Before giving the answer let us look at some examples.
Example 1. The following is the “ranking matrix” of three men, α, β, and γ , and three women, A, B, and C.
| Man\Woman | A | B | C |
|---|---|---|---|
| α | 1,3 | 2,2 | 3,1 |
| β | 3,1 | 1,3 | 2,2 |
| γ | 2,2 | 3,1 | 1,3 |
The first number of each pair in the matrix gives the ranking of women by the men, the second number is the ranking of the men by the women. Thus, α ranks A first, B second, C third, while A ranks β first, γ second, and α third, etc.
There are six possible sets of marriages; of these, three are stable. One of these is realized by giving each man his first choice, thus α marries A, β marries B, and γ marries C. Note that although each woman gets her last choice, the arrangement is nevertheless stable. Alternatively one may let the women have their first choices and marry α to C, β to A, and γ to B. The third stable arrangement is to give everyone his or her second choice and have α marry B, β marry C, and γ marry A. The reader will easily verify that all other arrangements are unstable.
In the existing implementation, we have match_new_teams_to_topics method in lottery_controller that performs this stable sorting algorithm for teams that have bid for an assignment. Our objective would be to implement a similar strategy, but we would use students instead of teams since there are no "team reviews". Every student selects their own assignment to review
Design strategy
Almost all of the functionality for our project has already been implemented in the review portion of the Expertiza system - where the teams are allowed to bid on topics they want to work for their project. We plan to reuse the code to keep the implementation DRY but we need to add some additional features to make it work for review bidding functionality. Delegation is a way to make composition as powerful for reuse as an inheritance. Delegation pattern will help us to reduce the coupling of methods to their original class as we have components that seem to behave identically, but we realize that this situation can change in the future.
Controller
Because of this, we plan to approach this project by first using delegation pattern to add biding capability to the review mapping controller for choosing topics. Apart from this, a new controller,review bids controller is implemented to handle interactions similar to the one done in lottery controller. It handles the following implementations.
1. Trigger the bidding by sending a request to PeerLogic backend with proper parameters retrieved from ReviewBid records.
2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.
View
The bidding interface for reviews is followed from similar implementation for topic bidding. We need to modify the code to implement the heat-map showing us demand density for each project to be reviewed and list of all available topics to review and selected topics. After we allow the student users to bid for their topic of interest we need to provide access to instructors to start the bidding algorithm.
Model
To maintain the original functionality of Expertiza as well as to add review bidding, a new model to handle the data of the bidding called ReviewBid is created. ReviewBid contains the information of bidding assignment and its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participant to bid on assignment teams rather than topics for simplicity.
Here is the definition of ReviewBid:
| Field Name | Type | Description |
|---|---|---|
| id | int(11) | unique identifier for the record |
| team_id | int(11) | the work of team to be bid on |
| participant_id | int(11) | participant id |
| priority | text | participant's preference towards the team |
| created_at | timestamp | |
| updated_at | timestamp |