CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review
About Expertiza
Expertiza is a web application through which students can submit and peer-review learning objects (articles, code, web sites, etc). The Expertiza project is supported by the National Science Foundation. It is used in select courses at NC State and by professors at several other colleges and universities.
Similarity to previous implementation
A similar review bidding feature was implemented in the previous semester, by making use of the Top Trading Cycles (TTC) algorithm.
Link: CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review
However, there were a few issues with that implementation, notably:
- The lack of color-coding feature.
Solution: We implmented the color coding feature in our bidding list and the selection list. Following is the screenshot for that implementation.
- Ordering was done by team IDs, which is not a reasonable ordering.
- The content of some newly-added file are very similar with existing ones.
- The UI could have been cleaner.
Solution: Some of their existing code was not DRY and they had some smelly code in their implementation. We have tried to remove most of these inconsistencies in the code.
- There is only one list while topic bidding interface has two lists, one for the the available list of topics and the other list for topics that are selected for the bidding.
Solution: We have implemented the two lists as seen below:
<div style="display:block">
<div style="display:inline-block" >
<!--Topics table-->
<table style="width: 48%; float: left" class="general">
<tr><%= render :partial => 'table_header' %></tr>
<br/><br/>
<% i=1 %>
<tbody id="topics" data-update-url=<%= "/student_review/set_priority?participant_id=#{participant.id}" %> class="connectedSortable">
<tr class="sort-disabled"><td></td><td></td></tr>
<% for bid in @bids %>
<%= get_intelligent_review_row(bid) %>
<%= render :partial => 'table_line', :locals => {:i=>i, :bid=>bid} %>
</tr>
<% i=i+1 %>
<% end %>
</tbody>
</table>
<!--Selections table-->
<table style="width: 48%; display:block" class="general" align="right">
<% i=1 %>
<tr><%= render :partial => 'table_header' %></tr>
<tbody id="selections" data-update-url=<%= "/student_review/set_priority?participant_id=#{@participant.id}" %> class="connectedSortable">
<tr class="sort-disabled"><td></td><td></td></tr>
<% for topic in @selected_topics %>
<tr style= "background-color:<%= get_topic_bg_color_by_review(topic, @max_team_size) %>" id="topic_<%= topic.id %>">
<%= render :partial => 'table_line', :locals => {:i=>i, :topic=>topic} %>
</tr>
<% i=i+1 %>
<% end %>
</tbody>
</table>
</div>
</div>
- There was no "add link to submission" in the bidding list.
Solution: After much deliberation, this feature is not required.
- This implementation altered too many lines of existing Expertiza code.
Solution: The project required us to implement review_bids and student_review models, controllers and view for student_review. We also applied migrations to the existing database to incorporate this functionality.
- Many irrelevant tests were included.
Solution: We have not included any of the team's irrelevant tests.
While we would not be writing this implementation from scratch, we would instead be using their Pull Request and remove the possible defects that they had.
Problem statement
Students currently are able to "bid" for topics that they want to do as an assignment. We plan to implement the same bidding feature for reviewing others' work, which is currently assigned on a "first-come-first-serve" basis.
The current first-come-first-serve approach to assigning reviews to students is not efficient since sometimes the same project is requested to be reviewed by multiple students, while other projects are not in demand. By letting the students bid on topics they're interested in, the reviews are assigned more efficiently based on user interest.
In this project we will make use of the Top Trading Cycles algorithm on Expertiza, to implement the review-bidding feature.
Current Implementation of the bidding
Before talking about the functionality of review mapping, it's necessary to describe some of the related models and controllers in Expertiza and how they are organized to fully implement the functionality. It should be noted here that this is an implementation of the bidding feature for assignment topics.
Models
1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.
2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams (Model Team).
3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.
4. After assignment submission, participants choose their preferred topics and response mappings between assignment (reviewed object), assignment team (reviewee) and assignment participant (reviewer) are created.
5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.
Use Case Diagram
Workflow
Currently, Expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, the method "add_reviewer" is invoked when students try to request a peer review. When it's done, a new mapping is created, that is why review assignment is in FIFO style.
To understand the basic workflow of bidding on topics, we can look at the LotteryController:
Firstly, teams have different preferences towards available topics. For example, let's say there are three topics within an assignment, namely A, B and C and there are 4 teams W, X, Y, Z. Let's assume each teams preferred order of topics is as follows:
W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C
This data structure is passed to the back-end service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to let the teams have a fair bidding over the topics.
Now, consider a similar bidding approach to peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.
Some discrepancies need additional attention.
1. Slots can be left unoccupied if no team is willing to respond. However, every response needs to have at least some reviewers.
2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.
3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.
Top Trading Cycles Mechanism
In the paper School Choice: A Mechanism Design Approach, author Atila Abdulkadiroglu, and Tayfun Sonmez have described Top Trading Cycles Mechanism. The intuition of the mechanism is students who have the highest priorities are allowed to trade the schools. Students who are assigned to schools are removed and other students who have the highest priority will compete for schools. The mechanism is as follows:
Step1: Each school has a counter (No. of seats/ capacity). Each student has a priority list. Each school points to the student who has the highest priority to the school. A cycle is formed which is an ordered list (s1, i1, s2, ...., sk,ik). Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.
Step k: Remaining students point to their favorite school among the remaining schools and each remaining school points to the student with the highest priority among the remaining students. There is at least one student. Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.
The algorithm terminates when all students are assigned a seat.
1. Top Trading Cycles algorithm: [1]
Bidding policy (Stable marriage problem)
As part of this assignment, we have to analyse the implementation of the stable marriage problem which is a stable sorting algorithm. The following summary of how the algorithm works, is taken from the "College Admissions and the Stability of Marriage (1962)" by D. Gale and L. S. Shapley.
A certain community consists of n men and n women. Each person ranks those of the opposite sex in accordance with his or her preferences for a marriage partner. We
seek a satisfactory way of marrying off all members of the community. Imitating our earlier definition, we call a set of marriages unstable (and here the suitability of the
term is quite clear) if under it there are a man and a woman who are not married to each other but prefer each other to their actual mates.
Definition: An assignment of couples will be called unstable if there are two men α and β who are married to women A and B, respectively, although β prefers A to B and A prefers β to α.
So, we can ask the question: For any pattern of preferences is it possible to find a stable set of marriages?
Before giving the answer let us look at some examples.
Example 1. The following is the “ranking matrix” of three men, α, β, and γ , and three women, A, B, and C.
| Man\Woman | A | B | C |
|---|---|---|---|
| α | 1,3 | 2,2 | 3,1 |
| β | 3,1 | 1,3 | 2,2 |
| γ | 2,2 | 3,1 | 1,3 |
The first number of each pair in the matrix gives the ranking of women by the men, the second number is the ranking of the men by the women. Thus, α ranks A first, B second, C third, while A ranks β first, γ second, and α third, etc.
There are six possible sets of marriages; of these, three are stable. One of these is realized by giving each man his first choice, thus α marries A, β marries B, and γ marries C. Note that although each woman gets her last choice, the arrangement is nevertheless stable. Alternatively one may let the women have their first choices and marry α to C, β to A, and γ to B. The third stable arrangement is to give everyone his or her second choice and have α marry B, β marry C, and γ marry A. The reader will easily verify that all other arrangements are unstable.
Based on this approach, in the existing implementation of Expertiza, we have match_new_teams_to_topics method in the lottery_controller that performs this stable sorting algorithm for teams that have bid for an assignment. Our objective would be to implement a similar strategy, but we would use students instead of teams since there are no "team reviews". Every student selects their own assignment to review.
Design strategy
Almost all of the functionality for our project has already been implemented in the review portion of the Expertiza system - where the teams are allowed to bid on topics they want to work for their project. We plan to reuse the code to keep the implementation DRY but we need to add some additional features to make it work for the review bidding functionality. Delegation is a way to make composition as powerful for reuse as inheritance. The Delegation pattern will help us reduce the coupling of methods to their original class as we have components that seem to behave identically, but we realize that this situation can change in the future.
Controller
Because of this, we plan to approach this project by first using delegation pattern to add biding capability to the review mapping controller for choosing topics. Apart from this, a new controller,review bids controller is implemented to handle interactions similar to the one done in lottery controller. It handles the following implementations.
1. Trigger the bidding by sending a request to PeerLogic backend with proper parameters retrieved from ReviewBid records.
2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.
View
The bidding interface for reviews is followed from similar implementation for topic bidding. We need to modify the code to implement the heat-map showing us the demand density for each project to be reviewed, the list of all available topics to review and the list of topics selected already.
Color Coding Scheme:
The color coding scheme would range from Red for the most in-demand topics to green for the least contested topics.
The top 10% of in-demand topics will be coded in Red, the next 30% of the topics will be in orange, the next 30% in yellow and the last 30% in the green. We are using percentages instead of hard-coding the color coding scheme so that it works dynamically with changing number of students.
After we allow the student users to bid for their topic of interest we need to provide access to instructors to start the bidding algorithm.
Model
To maintain the original functionality of Expertiza as well as to add review bidding, a new model to handle bidding data called ReviewBid is created. ReviewBid contains bidding assignment information, its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participants bid on assignment teams rather than topics for simplicity.
Here is the definition of ReviewBid:
| Field Name | Type | Description |
|---|---|---|
| id | int(11) | unique identifier for the record |
| team_id | int(11) | the work of team to be bid on |
| participant_id | int(11) | participant id |
| priority | text | participant's preference towards the team |
| created_at | timestamp | |
| updated_at | timestamp |
Implementation
Allow Instructor to set review strategy to bidding:
After setting the review strategy to bidding, participants are assigned a default bidding list, ordered by topic's name.
Participants can drag the items up or down to alter the priority.
Instructors can start bidding by clicking the following icon.
After the bidding is done, participants can login and see what they are assigned with.
Test Plan
Automated tests are to be written for model ReviewBid, controller ReviewBidController. We also plan to write basic tests to check the correctness of Gale Shapley algorithm.
UI Tests:
In addition to automated tests we will also perform manual testing of the newly added features which include the following:
1. The student should be able to bid the reviews when Professor enables bid option for students on that particular assignment.
2. The reviews prioritized by the student should show colors i.e. green, orange and red indicating how likely he is going to get the review that he bid.
3. At maximum, a student should be able to four assignments.