Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-05-06T22:45:22Z

Dde:

== About Expertiza ==
Expertiza is a web application through which students can submit and peer-review learning objects (articles, code, web sites, etc). The Expertiza project is supported by the National Science Foundation. It is used in select courses at NC State and by professors at several other colleges and universities.

== Similarity to previous implementation ==
A similar review bidding feature was implemented in the previous semester, by making use of the '''Top Trading Cycles''' (TTC) algorithm. <br>
Link: [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review] <br>
However, there were a few issues with that implementation, notably:
*The lack of color-coding feature.
Solution: We implmented the color coding feature in our bidding list and the selection list. Following is the screenshot for that implementation.
* Ordering was done by team IDs, which is not a reasonable ordering.
* The content of some newly-added file are very similar with existing ones.
* The UI could have been cleaner.
Solution: Some of their existing code was not DRY and they had some smelly code in their implementation. We have tried to remove most of these inconsistencies in the code.
* There is only one list while topic bidding interface has two lists, one for the the available list of topics and the other list for topics that are selected for the bidding.
Solution: We have implemented the two lists as seen below:
<pre>
<div style="display:block">
<div style="display:inline-block" >

<table style="width: 48%; float: left" class="general">
<tr><%= render :partial => 'table_header' %></tr>
<br/><br/>
<% i=1 %>
<tbody id="topics" data-update-url=<%= "/student_review/set_priority?participant_id=#{participant.id}" %> class="connectedSortable">
<tr class="sort-disabled"><td></td><td></td></tr>
<% for bid in @bids %>
<%= get_intelligent_review_row(bid) %>
<%= render :partial => 'table_line', :locals => {:i=>i, :bid=>bid} %>
</tr>
<% i=i+1 %>
<% end %>
</tbody>
</table>


<table style="width: 48%; display:block" class="general" align="right">
<% i=1 %>
<tr><%= render :partial => 'table_header' %></tr>
<tbody id="selections" data-update-url=<%= "/student_review/set_priority?participant_id=#{@participant.id}" %> class="connectedSortable">
<tr class="sort-disabled"><td></td><td></td></tr>
<% for topic in @selected_topics %>
<tr style= "background-color:<%= get_topic_bg_color_by_review(topic, @max_team_size) %>" id="topic_<%= topic.id %>">
<%= render :partial => 'table_line', :locals => {:i=>i, :topic=>topic} %>
</tr>
<% i=i+1 %>
<% end %>
</tbody>
</table>

</div>
</div>
</pre>
[[File:sri4.JPG]]
[[File:sri5.JPG]]

* There was no "add link to submission" in the bidding list.
Solution: After much deliberation, this feature is not required.
* This implementation altered too many lines of existing Expertiza code.
Solution: The project required us to implement review_bids and student_review models, controllers and view for student_review. We also applied migrations to the existing database to incorporate this functionality.
* Many irrelevant tests were included.
Solution: We have not included any of the team's irrelevant tests.

While we would not be writing this implementation from scratch, we would instead be using their [https://github.com/expertiza/expertiza/pull/1322 Pull Request] and remove the possible defects that they had.

== Problem statement ==
Students currently are able to "bid" for topics that they want to do as an assignment. We plan to implement the same bidding feature for reviewing others' work, which is currently assigned on a "first-come-first-serve" basis.

The current first-come-first-serve approach to assigning reviews to students is not efficient since sometimes the same project is requested to be reviewed by multiple students, while other projects are not in demand. By letting the students bid on topics they're interested in, the reviews are assigned more efficiently based on user interest.

In this project we will make use of the Top Trading Cycles algorithm on Expertiza, to implement the review-bidding feature.

== Current Implementation of the bidding ==
Before talking about the functionality of review mapping, it's necessary to describe some of the related models and controllers in Expertiza and how they are organized to fully implement the functionality. It should be noted here that this is an implementation of the bidding feature for assignment topics.

===Models===

[[File:Current_design_review_bidding.jpg]]

1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.

2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams (Model Team).

3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.

4. After assignment submission, participants choose their preferred topics and response mappings between assignment (reviewed object), assignment team (reviewee) and assignment participant (reviewer) are created.

5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.

===Use Case Diagram===
[[File:Review_use_case.PNG‎]]
===Workflow===

Currently, Expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, the method "add_reviewer" is invoked when students try to request a peer review. When it's done, a new mapping is created, that is why review assignment is in FIFO style.

To understand the basic workflow of bidding on topics, we can look at the LotteryController:

Firstly, teams have different preferences towards available topics. For example, let's say there are three topics within an assignment, namely A, B and C and there are 4 teams W, X, Y, Z. Let's assume each teams preferred order of topics is as follows:

W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C

This data structure is passed to the back-end service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to let the teams have a fair bidding over the topics.

Now, consider a similar bidding approach to peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
[[File:Contrast_topic_review_bidding.jpg]]

As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.

Some discrepancies need additional attention.

1. Slots can be left unoccupied if no team is willing to respond. However, every response needs to have at least some reviewers.

2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.

3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.

== Top Trading Cycles Mechanism ==

In the paper School Choice: A Mechanism Design Approach, author Atila Abdulkadiroglu, and Tayfun Sonmez have described Top Trading Cycles Mechanism. The intuition of the mechanism is students who have the highest priorities are allowed to trade the schools. Students who are assigned to schools are removed and other students who have the highest priority will compete for schools. The mechanism is as follows:

Step1: Each school has a counter (No. of seats/ capacity). Each student has a priority list. Each school points to the student who has the highest priority to the school. A cycle is formed which is an ordered
list (s1, i1, s2, ...., sk,ik). Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

Step k: Remaining students point to their favorite school among the remaining schools and each remaining school points to the student with the highest priority among the remaining students. There is at least one student. Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

The algorithm terminates when all students are assigned a seat.

1. Top Trading Cycles algorithm: [https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

== Bidding policy (Stable marriage problem) ==
As part of this assignment, we have to analyse the implementation of the stable marriage problem which is a stable sorting algorithm. The following summary of how the algorithm works, is taken from the "College Admissions and the Stability of Marriage (1962)" by D. Gale and L. S. Shapley. <br><br>
A certain community consists of n men and n women. Each person ranks those of the opposite sex in accordance with his or her preferences for a marriage partner. We
seek a satisfactory way of marrying off all members of the community. Imitating our earlier deﬁnition, we call a set of marriages unstable (and here the suitability of the
term is quite clear) if under it there are a man and a woman who are not married to each other but prefer each other to their actual mates.

'''Deﬁnition:''' An assignment of couples will be called unstable if there are two men α and β who are married to women A and B, respectively, although β prefers A to B and A prefers β to α.

So, we can ask the question: For any pattern of preferences is it possible to ﬁnd a stable set of marriages?

Before giving the answer let us look at some examples.

Example 1. The following is the “ranking matrix” of three men, α, β, and γ , and three women, A, B, and C.

{| class="wikitable"
|-
! Man\Woman !! A !! B !! C
|-
| α || 1,3 || 2,2 || 3,1
|-
| β || 3,1 || 1,3 || 2,2
|-
| γ || 2,2 || 3,1 || 1,3
|}

The ﬁrst number of each pair in the matrix gives the ranking of women by the men, the second number is the ranking of the men by the women. Thus, α ranks A ﬁrst, B second, C third, while A ranks β ﬁrst, γ second, and α third, etc.

There are six possible sets of marriages; of these, three are stable. One of these is realized by giving each man his ﬁrst choice, thus α marries A, β marries B, and γ marries C. Note that although each woman gets her last choice, the arrangement is nevertheless stable. Alternatively one may let the women have their ﬁrst choices and marry α to C, β to A, and γ to B. The third stable arrangement is to give everyone his or her second choice and have α marry B, β marry C, and γ marry A. The reader will easily verify that all other arrangements are unstable.

Based on this approach, in the existing implementation of Expertiza, we have match_new_teams_to_topics method in the lottery_controller that performs this stable sorting algorithm for teams that have bid for an assignment. Our objective would be to implement a similar strategy, but we would use students instead of teams since there are no "team reviews". Every student selects their own assignment to review.

== Design strategy ==
Almost all of the functionality for our project has already been implemented in the review portion of the Expertiza system - where the teams are allowed to bid on topics they want to work for their project. We plan to reuse the code to keep the implementation DRY but we need to add some additional features to make it work for the review bidding functionality. '''Delegation''' is a way to make composition as powerful for reuse as inheritance. The Delegation pattern will help us reduce the coupling of methods to their original class as we have components that seem to behave identically, but we realize that this situation can change in the future.

[[File:Untitled_Diagram.png]]

====Controller====
Because of this, we plan to approach this project by first using delegation pattern to add biding capability to the [https://github.com/expertiza/expertiza/blob/master/app/controllers/review_mapping_controller.rb review mapping controller] for choosing topics. Apart from this, a new controller,[https://github.com/expertiza/expertiza/pull/1322/files#diff-e064188effa2297543b98ac353bba4e3 review bids controller] is implemented to handle interactions similar to the one done in [https://github.com/expertiza/expertiza/blob/master/app/controllers/lottery_controller.rb lottery controller]. It handles the following implementations.

1. Trigger the bidding by sending a request to PeerLogic backend with proper parameters retrieved from ReviewBid records.

2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.

====View====
The bidding interface for reviews is followed from [https://github.com/expertiza/expertiza/pull/778/commits similar] implementation for topic bidding. We need to modify the ''code'' to implement the heat-map showing us the demand density for each project to be reviewed, the list of all available topics to review and the list of topics selected already. <br>

'''Color Coding Scheme:''' <br>
The color coding scheme would range from Red for the most in-demand topics to green for the least contested topics. <br>
The top 10% of in-demand topics will be coded in Red, the next 30% of the topics will be in orange, the next 30% in yellow and the last 30% in the green. We are using percentages instead of hard-coding the color coding scheme so that it works dynamically with changing number of students. <br>
After we allow the student users to bid for their topic of interest we need to provide access to instructors to start the bidding algorithm.

====Model====
To maintain the original functionality of Expertiza as well as to add review bidding, a new model to handle bidding data called ReviewBid is created. ReviewBid contains bidding assignment information, its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participants bid on assignment teams rather than topics for simplicity.

Here is the definition of ReviewBid:

{| class="wikitable"
!Field Name !!Type !!Description
|-
!id
|int(11)
|unique identifier for the record
|-
!team_id
|int(11)
|the work of team to be bid on
|-
!participant_id
|int(11)
|participant id
|-
!priority
|text
|participant's preference towards the team
|-
!created_at
|timestamp
|
|-
!updated_at
|timestamp
|
|-
|}

== Implementation ==

Allow Instructor to set review strategy to '''bidding''': <br> <br>
[[File:sri1.JPG]] <br> <br>

After setting the review strategy to '''bidding''', participants are assigned a default bidding list, ordered by topic's name.<br> <br>
[[File:Default_bidding_list.png]]<br> <br>

Participants can drag the items up or down to alter the priority.<br> <br>
[[File:Altered_bidding_list.png]]<br> <br>

Instructors can start bidding by clicking the following icon.<br> <br>
[[File:sri3.JPG]]<br> <br>

After the bidding is done, participants can login and see what they are assigned with.<br> <br>
[[File:Bidding_result.png]]<br> <br>

== Test Plan ==

Automated tests are to be written for model ReviewBid, controller ReviewBidController. We also plan to write basic tests to check the correctness of Gale Shapley algorithm.

UI Tests:

In addition to automated tests we will also perform manual testing of the newly added features which include the following:

1. The student should be able to bid the reviews when Professor enables bid option for students on that particular assignment.

2. The reviews prioritized by the student should show colors i.e. green, orange and red indicating how likely he is going to get the review that he bid.

3. At maximum, a student should be able to four assignments.

== Pointers for future teams working on this project ==
1. The selections table in two column view does not retain the data
After we drag and drop the projects in the second table ‘selections’, the data does not persist or get saved in the DB. So when we refresh the page the selections table is empty again.
Code: topic_list code

2. Color-coding
The color coding implementation should be tested.
As the selections table is not getting saved, the color-coding feature could not be tested though it is implemented in the below file.
Code: def bg_color_coding_review

3. The Gale Shaply implementation needs to be checked.
After making sure the data gets saved in the DB from selection table, we need to make sure that it is passed to the implemented Gale shaply algorithm in the required format and gives the required output.
Code: def gale_shaply

4. The Gale Shapley button icon needs to be changed.
Right now, we are reusing the info.png which is the icon for intelligent topic assignment. We should find a better suited name for the file and also find an appropriate image for it.

5. DRY the code as it is vastly similar to the intelligent assignment implementation.

6. Testing
We need to write proper tests to verify our Gale shaply algorithm and the UI. Some possible Rspec tests that we can add are:
If a person gives a topic as his first preference and no one else has that topic as their first preference, the aforementioned person should get that topic to review after the Gale shaply algorithm is run.
The topic that a participant is assigned cannot be reviewed by the same participant.

== References ==

# College Admissions and the Stability of Marriage (1962), D. Gale and L. S. Shapley [https://www-jstor-org.prox.lib.ncsu.edu/stable/2312726?Search=yes&resultItemClick=true&searchText=College&searchText=Admissions&searchText=and&searchText=the&searchText=Stability&searchText=of&searchText=Marriage&searchUri=%2Faction%2FdoBasicSearch%3Fgroup%3Dnone%26amp%3Bfc%3Doff%26amp%3BQuery%3DCollege%2BAdmissions%2Band%2Bthe%2BStability%2Bof%2BMarriage%26amp%3Bwc%3Don%26amp%3Bacc%3Don&ab_segments=0%2Fdefault-2%2Fcontrol&refreqid=search%3A6c17f9ae92b22bc404fc3b7641ac908a&seq=1#metadata_info_tab_contents ]
# CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review]
# Pull Request #1322 (E1856. Allow reviewers to bid on what to review) [https://github.com/expertiza/expertiza/pull/1322 ]
# Top Trading Cycles Algorithm[https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-04-27T01:55:05Z

Dde: /* Similarity to previous implementation */

== About Expertiza ==
Expertiza is a web application through which students can submit and peer-review learning objects (articles, code, web sites, etc). The Expertiza project is supported by the National Science Foundation. It is used in select courses at NC State and by professors at several other colleges and universities.

== Similarity to previous implementation ==
A similar review bidding feature was implemented in the previous semester, by making use of the '''Top Trading Cycles'''(TTC) algorithm. <br>
Link: [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review] <br>
However, there were a few issues with that implementation, notably:
*The lack of color-coding feature.
Solution: We implmented the color coding feature in our bidding list and the selection list. Following is the screenshot for that implementation.
* Ordering was done by team IDs, which is not a reasonable ordering.
* The content of some newly-added file are very similar with existing ones.
* The UI could have been cleaner.
Solution: Some of their existing code was not DRY and they had some smelly code in their implementation. We have tried to remove most of these inconsistencies in the code.
* There is only one list while topic bidding interface has two lists, one for the the available list of topics and the other list for topics that are selected for the bidding.
Solution: We have implemented the two lists as seen below:
[[File:Dde1.png]]
[[File:Dde2.png]]

* There was no "add link to submission" in the bidding list.
Solution: After much deliberation, this feature is not required.
* This implementation altered too many lines of existing Expertiza code.
Solution: The project required us to implement review_bids and student_review models, controllers and view for student_review. We also applied migrations to the existing database to incorporate this functionality.
* Many irrelevant tests were included.

While we would not be writing this implementation from scratch, we would instead be using their [https://github.com/expertiza/expertiza/pull/1322 Pull Request] and remove the possible defects that they had.

== Problem statement ==
Students currently are able to "bid" for topics that they want to do as an assignment. We plan to implement the same bidding feature for reviewing others' work, which is currently assigned on a "first-come-first-serve" basis.

The current first-come-first-serve approach to assigning reviews to students is not efficient since sometimes the same project is requested to be reviewed by multiple students, while other projects are not in demand. By letting the students bid on topics they're interested in, the reviews are assigned more efficiently based on user interest.

In this project we will make use of the Top Trading Cycles algorithm on Expertiza, to implement the review-bidding feature.

== Current Implementation of the bidding ==
Before talking about the functionality of review mapping, it's necessary to describe some of the related models and controllers in Expertiza and how they are organized to fully implement the functionality. It should be noted here that this is an implementation of the bidding feature for assignment topics.

===Models===

[[File:Current_design_review_bidding.jpg]]

1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.

2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams(Model Team).

3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.

4. After assignment submission, participants choose their preferred topics and response mappings between assignment(reviewed object), assignment team(reviewee) and assignment participant(reviewer) are created.

5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.

===Use Case Diagram===
[[File:Review_use_case.PNG‎]]
===Workflow===

Currently, Expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, the method "add_reviewer" is invoked when students try to request a peer review. When it's done, a new mapping is created, that is why review assignment is in FIFO style.

To understand the basic workflow of bidding on topics, we can look at the LotteryController:

Firstly, teams have different preferences towards available topics. For example, let's say there are three topics within an assignment, namely A, B and C and there are 4 teams W, X, Y, Z. Let's assume each teams preferred order of topics is as follows:

W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C

This data structure is passed to the back-end service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to let the teams have a fair bidding over the topics.

Now, consider a similar bidding approach to peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
[[File:Contrast_topic_review_bidding.jpg]]

As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.

Some discrepancies need additional attention.

1. Slots can be left unoccupied if no team is willing to respond. However, every response needs to have at least some reviewers.

2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.

3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.

== Top Trading Cycles Mechanism ==

In the paper School Choice: A Mechanism Design Approach, author Atila Abdulkadiroglu, and Tayfun Sonmez have described Top Trading Cycles Mechanism. The intuition of the mechanism is students who have the highest priorities are allowed to trade the schools. Students who are assigned to schools are removed and other students who have the highest priority will compete for schools. The mechanism is as follows:

Step1: Each school has a counter (No. of seats/ capacity). Each student has a priority list. Each school points to the student who has the highest priority to the school. A cycle is formed which is an ordered
list (s1, i1, s2, ...., sk,ik). Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

Step k: Remaining students point to their favorite school among the remaining schools and each remaining school points to the student with the highest priority among the remaining students. There is at least one student. Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

The algorithm terminates when all students are assigned a seat.

1. Top Trading Cycles algorithm: [https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

== Bidding policy (Stable marriage problem) ==
As part of this assignment, we have to analyse the implementation of the stable marriage problem which is a stable sorting algorithm. The following summary of how the algorithm works, is taken from the "College Admissions and the Stability of Marriage (1962)" by D. Gale and L. S. Shapley. <br><br>
A certain community consists of n men and n women. Each person ranks those of the opposite sex in accordance with his or her preferences for a marriage partner. We
seek a satisfactory way of marrying off all members of the community. Imitating our earlier deﬁnition, we call a set of marriages unstable (and here the suitability of the
term is quite clear) if under it there are a man and a woman who are not married to each other but prefer each other to their actual mates.

'''Deﬁnition:''' An assignment of couples will be called unstable if there are two men α and β who are married to women A and B, respectively, although β prefers A to B and A prefers β to α.

So, we can ask the question: For any pattern of preferences is it possible to ﬁnd a stable set of marriages?

Before giving the answer let us look at some examples.

Example 1. The following is the “ranking matrix” of three men, α, β, and γ , and three women, A, B, and C.

{| class="wikitable"
|-
! Man\Woman !! A !! B !! C
|-
| α || 1,3 || 2,2 || 3,1
|-
| β || 3,1 || 1,3 || 2,2
|-
| γ || 2,2 || 3,1 || 1,3
|}

The ﬁrst number of each pair in the matrix gives the ranking of women by the men, the second number is the ranking of the men by the women. Thus, α ranks A ﬁrst, B second, C third, while A ranks β ﬁrst, γ second, and α third, etc.

There are six possible sets of marriages; of these, three are stable. One of these is realized by giving each man his ﬁrst choice, thus α marries A, β marries B, and γ marries C. Note that although each woman gets her last choice, the arrangement is nevertheless stable. Alternatively one may let the women have their ﬁrst choices and marry α to C, β to A, and γ to B. The third stable arrangement is to give everyone his or her second choice and have α marry B, β marry C, and γ marry A. The reader will easily verify that all other arrangements are unstable.

Based on this approach, in the existing implementation of Expertiza, we have match_new_teams_to_topics method in the lottery_controller that performs this stable sorting algorithm for teams that have bid for an assignment. Our objective would be to implement a similar strategy, but we would use students instead of teams since there are no "team reviews". Every student selects their own assignment to review.

== Design strategy ==
Almost all of the functionality for our project has already been implemented in the review portion of the Expertiza system - where the teams are allowed to bid on topics they want to work for their project. We plan to reuse the code to keep the implementation DRY but we need to add some additional features to make it work for the review bidding functionality. '''Delegation''' is a way to make composition as powerful for reuse as inheritance. The Delegation pattern will help us reduce the coupling of methods to their original class as we have components that seem to behave identically, but we realize that this situation can change in the future.

[[File:Untitled_Diagram.png]]

====Controller====
Because of this, we plan to approach this project by first using delegation pattern to add biding capability to the [https://github.com/expertiza/expertiza/blob/master/app/controllers/review_mapping_controller.rb review mapping controller] for choosing topics. Apart from this, a new controller,[https://github.com/expertiza/expertiza/pull/1322/files#diff-e064188effa2297543b98ac353bba4e3 review bids controller] is implemented to handle interactions similar to the one done in [https://github.com/expertiza/expertiza/blob/master/app/controllers/lottery_controller.rb lottery controller]. It handles the following implementations.

1. Trigger the bidding by sending a request to PeerLogic backend with proper parameters retrieved from ReviewBid records.

2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.

====View====
The bidding interface for reviews is followed from [https://github.com/expertiza/expertiza/pull/778/commits similar] implementation for topic bidding. We need to modify the ''code'' to implement the heat-map showing us the demand density for each project to be reviewed, the list of all available topics to review and the list of topics selected already. <br>

'''Color Coding Scheme:''' <br>
The color coding scheme would range from Red for the most in-demand topics to green for the least contested topics. <br>
The top 10% of in-demand topics will be coded in Red, the next 30% of the topics will be in orange, the next 30% in yellow and the last 30% in the green. We are using percentages instead of hard-coding the color coding scheme so that it works dynamically with changing number of students. <br>
After we allow the student users to bid for their topic of interest we need to provide access to instructors to start the bidding algorithm.

====Model====
To maintain the original functionality of Expertiza as well as to add review bidding, a new model to handle bidding data called ReviewBid is created. ReviewBid contains bidding assignment information, its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participants bid on assignment teams rather than topics for simplicity.

Here is the definition of ReviewBid:

{| class="wikitable"
!Field Name !!Type !!Description
|-
!id
|int(11)
|unique identifier for the record
|-
!team_id
|int(11)
|the work of team to be bid on
|-
!participant_id
|int(11)
|participant id
|-
!priority
|text
|participant's preference towards the team
|-
!created_at
|timestamp
|
|-
!updated_at
|timestamp
|
|-
|}

== Implementation ==

Allow Instructor to set review strategy to '''bidding''': <br> <br>
[[File:Review_strategy.png]] <br> <br>

After setting the review strategy to '''bidding''', participants are assigned a default bidding list, ordered by topic's name.<br> <br>
[[File:Default_bidding_list.png]]<br> <br>

Participants can drag the items up or down to alter the priority.<br> <br>
[[File:Altered_bidding_list.png]]<br> <br>

Instructors can start bidding by clicking the following icon.<br> <br>
[[File:Bidding_icon.png]]<br> <br>

After the bidding is done, participants can login and see what they are assigned with.<br> <br>
[[File:Bidding_result.png]]<br> <br>
== Test Plan ==

Automated tests are to be written for model ReviewBid, controller ReviewBidController. We also plan to write basic tests to check the correctness of Gale Shapley algorithm.

UI Tests:

In addition to automated tests we will also perform manual testing of the newly added features which include the following:

1. The student should be able to bid the reviews when Professor enables bid option for students on that particular assignment.

2. The reviews prioritized by the student should show colors i.e. green, orange and red indicating how likely he is going to get the review that he bid.

3. At maximum, a student should be able to four assignments.

== References ==

# College Admissions and the Stability of Marriage (1962), D. Gale and L. S. Shapley [https://www-jstor-org.prox.lib.ncsu.edu/stable/2312726?Search=yes&resultItemClick=true&searchText=College&searchText=Admissions&searchText=and&searchText=the&searchText=Stability&searchText=of&searchText=Marriage&searchUri=%2Faction%2FdoBasicSearch%3Fgroup%3Dnone%26amp%3Bfc%3Doff%26amp%3BQuery%3DCollege%2BAdmissions%2Band%2Bthe%2BStability%2Bof%2BMarriage%26amp%3Bwc%3Don%26amp%3Bacc%3Don&ab_segments=0%2Fdefault-2%2Fcontrol&refreqid=search%3A6c17f9ae92b22bc404fc3b7641ac908a&seq=1#metadata_info_tab_contents ]
# CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review]
# Pull Request #1322 (E1856. Allow reviewers to bid on what to review) [https://github.com/expertiza/expertiza/pull/1322 ]
# Top Trading Cycles Algorithm[https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

File:Dde1.png

2019-04-27T01:52:17Z

Dde:

File:Dde2.png

2019-04-27T01:51:56Z

Dde:

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-04-12T20:34:12Z

Dde: /* View */

== About Expertiza ==
Expertiza is a web application through which students can submit and peer-review learning objects (articles, code, web sites, etc). The Expertiza project is supported by the National Science Foundation. It is used in select courses at NC State and by professors at several other colleges and universities.

== Similarity to previous implementation ==
A similar review bidding feature was implemented in the previous semester, by making use of the '''Top Trading Cycles'''(TTC) algorithm. <br>
Link: [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review] <br>
However, there were a few issues with that implementation, notably:
# The lack of color-coding feature.
# Ordering was done by team IDs, which is not a reasonable ordering.
# The content of some newly-added file are very similar with existing ones.
# The UI could have been cleaner.
# There is only one list while topic bidding interface has two lists, one for the the available list of topics and the other list for topics that are selected for the bidding.
# There was no "add link to submission" in the bidding list.
# This implementation altered too many lines of existing Expertiza code.
# Many irrelevant tests were included.

While we would not be writing this implementation from scratch, we would instead be using their [https://github.com/expertiza/expertiza/pull/1322 Pull Request] and remove the possible defects that they had.

== Problem statement ==
Students currently are able to "bid" for topics that they want to do as an assignment. We plan to implement the same bidding feature for reviewing others' work, which is currently assigned on a "first-come-first-serve" basis.

The current first-come-first-serve approach to assigning reviews to students is not efficient since sometimes the same project is requested to be reviewed by multiple students, while other projects are not in demand. By letting the students bid on topics they're interested in, the reviews are assigned more efficiently based on user interest.

In this project we will make use of the Top Trading Cycles algorithm on Expertiza, to implement the review-bidding feature.

== Current Implementation of the bidding ==
Before talking about the functionality of review mapping, it's necessary to describe some of the related models and controllers in Expertiza and how they are organized to fully implement the functionality. It should be noted here that this is an implementation of the bidding feature for assignment topics.

===Models===

[[File:Current_design_review_bidding.jpg]]

1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.

2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams(Model Team).

3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.

4. After assignment submission, participants choose their preferred topics and response mappings between assignment(reviewed object), assignment team(reviewee) and assignment participant(reviewer) are created.

5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.

===Use Case Diagram===
[[File:Review_use_case.PNG‎]]
===Workflow===

Currently, Expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, the method "add_reviewer" is invoked when students try to request a peer review. When it's done, a new mapping is created, that is why review assignment is in FIFO style.

To understand the basic workflow of bidding on topics, we can look at the LotteryController:

Firstly, teams have different preferences towards available topics. For example, let's say there are three topics within an assignment, namely A, B and C and there are 4 teams W, X, Y, Z. Let's assume each teams preferred order of topics is as follows:

W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C

This data structure is passed to the back-end service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to let the teams have a fair bidding over the topics.

Now, consider a similar bidding approach to peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
[[File:Contrast_topic_review_bidding.jpg]]

As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.

Some discrepancies need additional attention.

1. Slots can be left unoccupied if no team is willing to respond. However, every response needs to have at least some reviewers.

2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.

3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.

== Top Trading Cycles Mechanism ==

In the paper School Choice: A Mechanism Design Approach, author Atila Abdulkadiroglu, and Tayfun Sonmez have described Top Trading Cycles Mechanism. The intuition of the mechanism is students who have the highest priorities are allowed to trade the schools. Students who are assigned to schools are removed and other students who have the highest priority will compete for schools. The mechanism is as follows:

Step1: Each school has a counter (No. of seats/ capacity). Each student has a priority list. Each school points to the student who has the highest priority to the school. A cycle is formed which is an ordered
list (s1, i1, s2, ...., sk,ik). Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

Step k: Remaining students point to their favorite school among the remaining schools and each remaining school points to the student with the highest priority among the remaining students. There is at least one student. Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

The algorithm terminates when all students are assigned a seat.

1. Top Trading Cycles algorithm: [https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

== Bidding policy (Stable marriage problem) ==
As part of this assignment, we have to analyse the implementation of the stable marriage problem which is a stable sorting algorithm. The following summary of how the algorithm works, is taken from the "College Admissions and the Stability of Marriage (1962)" by D. Gale and L. S. Shapley. <br><br>
A certain community consists of n men and n women. Each person ranks those of the opposite sex in accordance with his or her preferences for a marriage partner. We
seek a satisfactory way of marrying off all members of the community. Imitating our earlier deﬁnition, we call a set of marriages unstable (and here the suitability of the
term is quite clear) if under it there are a man and a woman who are not married to each other but prefer each other to their actual mates.

'''Deﬁnition:''' An assignment of couples will be called unstable if there are two men α and β who are married to women A and B, respectively, although β prefers A to B and A prefers β to α.

So, we can ask the question: For any pattern of preferences is it possible to ﬁnd a stable set of marriages?

Before giving the answer let us look at some examples.

Example 1. The following is the “ranking matrix” of three men, α, β, and γ , and three women, A, B, and C.

{| class="wikitable"
|-
! Man\Woman !! A !! B !! C
|-
| α || 1,3 || 2,2 || 3,1
|-
| β || 3,1 || 1,3 || 2,2
|-
| γ || 2,2 || 3,1 || 1,3
|}

The ﬁrst number of each pair in the matrix gives the ranking of women by the men, the second number is the ranking of the men by the women. Thus, α ranks A ﬁrst, B second, C third, while A ranks β ﬁrst, γ second, and α third, etc.

There are six possible sets of marriages; of these, three are stable. One of these is realized by giving each man his ﬁrst choice, thus α marries A, β marries B, and γ marries C. Note that although each woman gets her last choice, the arrangement is nevertheless stable. Alternatively one may let the women have their ﬁrst choices and marry α to C, β to A, and γ to B. The third stable arrangement is to give everyone his or her second choice and have α marry B, β marry C, and γ marry A. The reader will easily verify that all other arrangements are unstable.

Based on this approach, in the existing implementation of Expertiza, we have match_new_teams_to_topics method in the lottery_controller that performs this stable sorting algorithm for teams that have bid for an assignment. Our objective would be to implement a similar strategy, but we would use students instead of teams since there are no "team reviews". Every student selects their own assignment to review.

== Design strategy ==
Almost all of the functionality for our project has already been implemented in the review portion of the Expertiza system - where the teams are allowed to bid on topics they want to work for their project. We plan to reuse the code to keep the implementation DRY but we need to add some additional features to make it work for the review bidding functionality. '''Delegation''' is a way to make composition as powerful for reuse as inheritance. The Delegation pattern will help us reduce the coupling of methods to their original class as we have components that seem to behave identically, but we realize that this situation can change in the future.

[[File:Untitled_Diagram.png]]

====Controller====
Because of this, we plan to approach this project by first using delegation pattern to add biding capability to the [https://github.com/expertiza/expertiza/blob/master/app/controllers/review_mapping_controller.rb review mapping controller] for choosing topics. Apart from this, a new controller,[https://github.com/expertiza/expertiza/pull/1322/files#diff-e064188effa2297543b98ac353bba4e3 review bids controller] is implemented to handle interactions similar to the one done in [https://github.com/expertiza/expertiza/blob/master/app/controllers/lottery_controller.rb lottery controller]. It handles the following implementations.

1. Trigger the bidding by sending a request to PeerLogic backend with proper parameters retrieved from ReviewBid records.

2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.

====View====
The bidding interface for reviews is followed from [https://github.com/expertiza/expertiza/pull/778/commits similar] implementation for topic bidding. We need to modify the ''code'' to implement the heat-map showing us the demand density for each project to be reviewed, the list of all available topics to review and the list of topics selected already. <br>

'''Color Coding Scheme:''' <br>
The color coding scheme would range from Red for the most in-demand topics to green for the least contested topics. <br>
The top 10% of in-demand topics will be coded in Red, the next 30% of the topics will be in orange, the next 30% in yellow and the last 30% in the green. We are using percentages instead of hard-coding the color coding scheme so that it works dynamically with changing number of students. <br>
After we allow the student users to bid for their topic of interest we need to provide access to instructors to start the bidding algorithm.

====Model====
To maintain the original functionality of Expertiza as well as to add review bidding, a new model to handle bidding data called ReviewBid is created. ReviewBid contains bidding assignment information, its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participants bid on assignment teams rather than topics for simplicity.

Here is the definition of ReviewBid:

{| class="wikitable"
!Field Name !!Type !!Description
|-
!id
|int(11)
|unique identifier for the record
|-
!team_id
|int(11)
|the work of team to be bid on
|-
!participant_id
|int(11)
|participant id
|-
!priority
|text
|participant's preference towards the team
|-
!created_at
|timestamp
|
|-
!updated_at
|timestamp
|
|-
|}

== Implementation ==

Allow Instructor to set review strategy to '''bidding''': <br> <br>
[[File:Review_strategy.png]] <br> <br>

After setting the review strategy to '''bidding''', participants are assigned a default bidding list, ordered by topic's name.<br> <br>
[[File:Default_bidding_list.png]]<br> <br>

Participants can drag the items up or down to alter the priority.<br> <br>
[[File:Altered_bidding_list.png]]<br> <br>

Instructors can start bidding by clicking the following icon.<br> <br>
[[File:Bidding_icon.png]]<br> <br>

After the bidding is done, participants can login and see what they are assigned with.<br> <br>
[[File:Bidding_result.png]]<br> <br>

== References ==

# College Admissions and the Stability of Marriage (1962), D. Gale and L. S. Shapley [https://www-jstor-org.prox.lib.ncsu.edu/stable/2312726?Search=yes&resultItemClick=true&searchText=College&searchText=Admissions&searchText=and&searchText=the&searchText=Stability&searchText=of&searchText=Marriage&searchUri=%2Faction%2FdoBasicSearch%3Fgroup%3Dnone%26amp%3Bfc%3Doff%26amp%3BQuery%3DCollege%2BAdmissions%2Band%2Bthe%2BStability%2Bof%2BMarriage%26amp%3Bwc%3Don%26amp%3Bacc%3Don&ab_segments=0%2Fdefault-2%2Fcontrol&refreqid=search%3A6c17f9ae92b22bc404fc3b7641ac908a&seq=1#metadata_info_tab_contents ]
# CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review]
# Pull Request #1322 (E1856. Allow reviewers to bid on what to review) [https://github.com/expertiza/expertiza/pull/1322 ]
# Top Trading Cycles Algorithm[https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-04-06T17:36:59Z

Dde: /* Design strategy */

== About Expertiza ==
Expertiza is a web application through which students can submit and peer-review learning objects (articles, code, web sites, etc). The Expertiza project is supported by the National Science Foundation. It is used in select courses at NC State and by professors at several other colleges and universities.

== Similarity to previous implementation ==
A similar implementation was provided in the last semester was by a team who had implemented the TTC and the required review bidding functionality. <br>
Link: [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review] <br>
However, their implementation had a few problems, notably:
# The color-coding feature was not implemented.
# The implementation entailed ordering by IDs of teams who did the topics (not a reasonable ordering).
# The content of some newly-added file are very similar with existing ones.
# The UI was not clean, it appeared to be disarrayed.
# There is only one list while topic bidding interface has two lists, one for the the available list of topics and the other list for topics that are selected for the bidding.
# There was no "add link to submission" in the bidding list.
# This implementation altered too many lines of existing Expertiza code.
# Many irrelevant tests were included.

While we would not be writing this implementation from scratch, we would instead be using their [https://github.com/expertiza/expertiza/pull/1322 Pull Request] and remove the possible defects that they had.

== Problem statement ==
Students currently are able to “bid” for the projects that they want to do as an assignment. On the other hand, for reviewing others’ work, the policy that’s currently in use is “first-come-first-serve”. We would try to implement the Top Trading Cycles algorithm that assigns review topics to students in a priority.

The bidding policy for project topics that students want to work on is already implemented. Students can currently bid on project topics for their team assignment. This reduces the conflicts in assigning topics to each team. But to contradict the desired implementation, we will make use of student user instead of teams, since a single student bids on a multiple topics.

There’s also reviewing work for each assignment. Currently, the policy to assign the project to the reviewer is “first-come-first-serve”. Student who chooses to review a project, will get the same project based on priority.

However, this policy creates an issue — while reviewing a student's work, sometimes the same project is requested to be reviewed by many students, while some other projects are not in demand. A similar bidding policy for assigning projects to reviewers can help students who want to review a project the most to be most likely to receive that project. The completion of this project will allow students to also bid on what projects they are interested in reviewing.

The project includes implementation of the top trading cycles algorithm on Expertiza.

== Current Implementation of the bidding ==
Regarding to the functionality of review mapping, it's necessary to describe some of the related models and controllers in expertiza and how they are organized to fully implement the functionality. It should be noted here that this is an implementation of the bidding for assignment topics.

===Models===

[[File:Current_design_review_bidding.jpg]]

1. When an assignment is released, an instance of Assignment is created and corresponding topic instances of SignUpTopics are also imported, indicating that different topics are available for students to choose from within this assignment.

2. Students are assigned as instances of AssignmentParticipant to this assignment and form up their teams(Model Team).

3. Each team registers for several topics and becomes a candidate instance of SignUpTeam. After topics bidding, some of the candidate teams become an official signed-up team of a particular topic.

4. After assignment submission, participants choose their preferred topics and response mappings between assignment(reviewed object), assignment team(reviewee) and assignment participant(reviewer) are created.

5. After the mapping, participants can submit their response and the afterwards is out of discussion range of this document.

===Workflow===

For now, expertiza employs FIFO strategy on review assignment. In the ReviewMappingController, method "add_reviewer" is invoked when students are trying to request a peer review. When it is done, a new mapping is created, that is why review assignment is in FIFO style.

If reviewers are allowed to bid on what to review, the procedure of topics bidding implemented by LotteryController is a good reference. Here is the basic workflow of topics bidding:

First of all, teams have different preference towards the topics. For example, there are three topics within an assignment, namely A, B and C. Therefore 4 teams W, X, Y, Z. Teams would have their own preference towards different topics:

W: B, A, C | X: B, C | Y: C, A, B | Z: A, B, C

This data structure is passed to backend service http://peerlogic.csc.ncsu.edu/intelligent_assignment/merge_teams by method "run_intelligent_assignment" of LotteryController. The peerlogic service runs top_trading_cycles algorithm to have a fair bidding over the topics and teams.

A similar bid can take place on peer review assignment. Why is it workable? The following diagram illustrates the similarity of the two bidding model.
[[File:Contrast_topic_review_bidding.jpg]]

As for topics bidding, a few available slots of a particular topic are open for teams contending. In the left part of the diagram, there are 4 slots so only 4 teams can successfully contend for it. Slot is described as "max_choosers" of a sign_up topic in Expertiza. Consequently, there will be 4 different responses after the submission, and this time it is the participants that contend for the responses.

Some discrepancies need additional attention.

1. Slots can be left unoccupied if no team is willing to response. However, every response needs to have at least some reviewers.

2. The size of bidding data is different, since the number of teams are usually 1/2, 1/3 or 1/4 of the participants.

3. The policy of bidding may be slightly different. Participants are not allowed to review a response submitted by themselves. However, the topic bidding doesn't have this constraint.

== Top Trading Cycles Mechanism ==

In the paper School Choice: A Mechanism Design Approach, author Atila Abdulkadiroglu, and Tayfun Sonmez have described Top Trading Cycles Mechanism. The intuition of the mechanism is students who have the highest priorities are allowed to trade the schools. Students who are assigned to schools are removed and other students who have the highest priority will compete for schools. The mechanism is as follows:

step1: Each school has a counter (no of seats/ capacity). Each student has a priority list. Each school points to the student who has the highest priority to the school. A cycle is formed which is an ordered
list (s1, i1, s2, ...., sk,ik). Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

step k: Remaining students point to their favorite school among the remaining schools and each remaining school points to the student with the highest priority among the remaining students. There is at least one student. Every student in the cycle points to the school he/she is admitted and removed from the list. The counter of school is reduced by one and when it becomes zero it is removed from the list.

The algorithm terminates when all students are assigned a seat.

1. Top Trading Cycles algorithm: [https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

== Bidding policy (Stable marriage problem) ==
As part of this assignment, we have to analyse the implementation of the stable marriage problem which is a stable sorting algorithm. The explanation for this algorithm is as follows which as been taken from "College Admissions and the Stability of Marriage (1962)" by D. Gale and L. S. Shapley. <br><br>
A certain community consists of n men and n women. Each person ranks those of the opposite sex in accordance with his or her preferences for a marriage partner. We
seek a satisfactory way of marrying off all members of the community. Imitating our earlier deﬁnition, we call a set of marriages unstable (and here the suitability of the
term is quite clear) if under it there are a man and a woman who are not married to each other but prefer each other to their actual mates.

'''Deﬁnition:''' An assignment of couples will be called unstable if there are two men α and β who are married to women A and B, respectively, although β prefers A to B and A prefers β to α.

So, we can ask the question: For any pattern of preferences is it possible to ﬁnd a stable set of marriages?

Before giving the answer let us look at some examples.

Example 1. The following is the “ranking matrix” of three men, α, β, and γ , and three women, A, B, and C.

{| class="wikitable"
|-
! Man\Woman !! A !! B !! C
|-
| α || 1,3 || 2,2 || 3,1
|-
| β || 3,1 || 1,3 || 2,2
|-
| γ || 2,2 || 3,1 || 1,3
|}

The ﬁrst number of each pair in the matrix gives the ranking of women by the men, the second number is the ranking of the men by the women. Thus, α ranks A ﬁrst, B second, C third, while A ranks β ﬁrst, γ second, and α third, etc.

There are six possible sets of marriages; of these, three are stable. One of these is realized by giving each man his ﬁrst choice, thus α marries A, β marries B, and γ marries C. Note that although each woman gets her last choice, the arrangement is nevertheless stable. Alternatively one may let the women have their ﬁrst choices and marry α to C, β to A, and γ to B. The third stable arrangement is to give everyone his or her second choice and have α marry B, β marry C, and γ marry A. The reader will easily verify that all other arrangements are unstable.

In the existing implementation, we have match_new_teams_to_topics method in lottery_controller that performs this stable sorting algorithm for teams that have bid for an assignment. Our objective would be to implement a similar strategy, but we would use students instead of teams since there are no "team reviews". Every student selects their own assignment to review

== Design strategy ==
Almost all of the functionality for our project has already been implemented in the review portion of the Expertiza system - where the teams are allowed to bid on topics they want to work for their project. We plan to reuse the code to keep the implementation DRY but we need to add some additional features to make it work for review bidding functionality. '''Delegation''' is a way to make composition as powerful for reuse as an inheritance. Delegation pattern will help us to reduce the coupling of methods to their original class as we have components that seem to behave identically, but we realize that this situation can change in the future.

[[File:Untitled_Diagram.png]]

====Controller====
Because of this, we plan to approach this project by first using delegation pattern to add biding capability to the [https://github.com/expertiza/expertiza/blob/master/app/controllers/review_mapping_controller.rb review mapping controller] for choosing topics. Apart from this, a new controller,[https://github.com/expertiza/expertiza/pull/1322/files#diff-e064188effa2297543b98ac353bba4e3 review bids controller] is implemented to handle interactions similar to the one done in [https://github.com/expertiza/expertiza/blob/master/app/controllers/lottery_controller.rb lottery controller]. It handles the following implementations.

1. Trigger the bidding by sending a request to PeerLogic backend with proper parameters retrieved from ReviewBid records.

2. Process the response from PeerLogic and transform it into records of ReviewResponseMap.

====View====
The bidding interface for reviews is followed from [https://github.com/expertiza/expertiza/pull/778/commits similar] implementation for topic bidding. We need to modify the ''code'' to implement the heat-map showing us demand density for each project to be reviewed and list of all available topics to review and selected topics. After we allow the student users to bid for their topic of interest we need to provide access to instructors to start the bidding algorithm.

====Model====
To maintain the original functionality of Expertiza as well as to add review bidding, a new model to handle the data of the bidding called ReviewBid is created. ReviewBid contains the information of bidding assignment and its biddable topics as well as the participants' preference. ReviewBid is served as the input of the bidding algorithm. We decided to let participant to bid on assignment teams rather than topics for simplicity.

Here is the definition of ReviewBid:

{| class="wikitable"
!Field Name !!Type !!Description
|-
!id
|int(11)
|unique identifier for the record
|-
!team_id
|int(11)
|the work of team to be bid on
|-
!participant_id
|int(11)
|participant id
|-
!priority
|text
|participant's preference towards the team
|-
!created_at
|timestamp
|
|-
!updated_at
|timestamp
|
|-
|}

== Implementation ==

Allow Instructor to set review strategy to be bidding:
[[File:Review_strategy.png]]

After setting the review strategy to be bidding, participants are assigned with a default bidding list, ordered by topic's name.
[[File:Default_bidding_list.png]]

Participants can drag the items up and own to alter the priority.
[[File:Altered_bidding_list.png]]

Instructors can start bidding by clicking the following icon.
[[File:Bidding_icon.png]]

After the bidding is done, participants can login and see what they are assigned with.
[[File:Bidding_result.png]]

== References ==

# [https://www-jstor-org.prox.lib.ncsu.edu/stable/2312726?Search=yes&resultItemClick=true&searchText=College&searchText=Admissions&searchText=and&searchText=the&searchText=Stability&searchText=of&searchText=Marriage&searchUri=%2Faction%2FdoBasicSearch%3Fgroup%3Dnone%26amp%3Bfc%3Doff%26amp%3BQuery%3DCollege%2BAdmissions%2Band%2Bthe%2BStability%2Bof%2BMarriage%26amp%3Bwc%3Don%26amp%3Bacc%3Don&ab_segments=0%2Fdefault-2%2Fcontrol&refreqid=search%3A6c17f9ae92b22bc404fc3b7641ac908a&seq=1#metadata_info_tab_contents College Admissions and the Stability of Marriage (1962), D. Gale and L. S. Shapley]
# [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review CSC/ECE_517_Fall_2018/E1856_Allow_reviewers_to_bid_on_what_to_review]
# [https://github.com/expertiza/expertiza/pull/1322 Pull Request #1322 (E1856. Allow reviewers to bid on what to review)]
# Top Trading Cycles Algorithm[https://www.jstor.org/stable/pdf/3132114.pdf?casa_token=Bp8Syf8Q0yYAAAAA:f7zDJdWurp5cYlHtSHaUxRbDc1o4sL0Qx8UTnk7C5eeXPhgyiTOaWqgftX-nNJ48s6SGyPBzH3_U3Yv6Pfapo1OJaj-estvNyRl60LNQi5QgGW3LmAW8pQ]

File:Untitled Diagram.png

2019-04-06T17:36:06Z

Dde: uploaded a new version of "File:Untitled Diagram.png"

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-04-06T16:08:58Z

Dde: /* Model */

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-04-06T16:01:34Z

Dde: /* Design strategy */

CSC/ECE 517 Spring 2019 - Project E1928. Allow reviewers to bid on what to review

2019-04-05T14:03:40Z

Dde:

== About Expertiza ==
It is a web server.
== Top Trading Cycles Algorithm==

== Design strategy ==

== References ==

E1912 Export Tags

2019-04-02T00:06:41Z

Dde: /* Implementation */

==Introduction==
===About Expertiza===
Expertiza is an Open Source Web Application Software managed by National Science Foundation. Expertiza is used by many courses including CSC 517 for assignment management. It has functionalities such as peer reviews, teammate reviews and tagging reviews in which students can provide feedback on other's work which helps peer in better developing the project.

===Motivation===
The Expertiza team currently had to manually fire queries in the database in order to get the tag submissions made by an individual user(student) on each review for a particular assignment. In this way, the team got the data from the entire class, which was then used to feed the Machine Learning algorithms. But many times it used to happen that students made random tags and the ML algorithm was not able to make good predictions out of it, hence in order to solve this issue a new idea was proposed as to select on the students which did not create outliers in the predictions and hence, therefore, it is a good proposal.

===Tasks Identified===
In order to perform this task following files were identified where the code hasbeen added.
* In _answer_tagging_report.html.erb view, a button is added for exporting the student's tagged values
* A checkbox for each row has been added in order to select students whose tags are to be exported(By default all will be selected).
* A method is written in export_file_controller.rb controller named export_tags that exports the CSV file.

==Implementation==
The workflow diagram of our implementation is shown here:

[[File:E1912flow.png]]

We have added a default select all checkbox under ''Report tags done by each user'' which enables student selection for exporting tags in the CSV file. We have added a column for the checkbox and a button named '''Export''' which on click generates the CSV file.

[[File:Eb1912Export.png]]

Below is the snippet of the same view ''_answer_tagging_report.html.erb ''

<pre>
<%= form_tag(export_tags, method: 'put',:id => 'tst') do %>
<div id="tab-summary">
<h4 align="center">Summary for each user</h4>
<table class="table table-striped sortable">
<thead>
<tr>
<th class="sorter-true">Username</th>
<th class="sorter-true">Name</th>
<th class="sorter-true">Percentage of tags done</th>
<th class="sorter-true"># tags done</th>
<th class="sorter-true"># tags not done</th>
<th class="sorter-true"># possible tags to do<img src='/assets/info.png' title='Number of answers that fulfil the prerequisite setup for each
tags (Answer of certain question types and has a length of greater than the threshold).'/></th>
<th>Tags export</th>
</tr>
</thead>
<% @user_tagging_report.each do |key, val| %>
<tr>
<td><%= val.user.name(session[:ip]) %></td>
<td><%= val.user.fullname(session[:ip]) %></td>
<td><%= val.percentage.to_s %>%</td>
<td><%= val.no_tagged.to_s %></td>
<td><%= val.no_not_tagged.to_s %></td>
<td><%= val.no_tagable %></td>
<td><%= check_box_tag 'names[]', val.user.name(session[:ip]), true %></td>
</tr>
<% end %>
</table>
</div>
<%= submit_tag "Export",:value => 'Export' ,:form => 'tst',method: 'put' %>
<% end %>
</pre>

==Screen Record==
# [https://drive.google.com/open?id=1B65_N3bWwPlSyiXMhXzXypKvKLN09g0r Here] is the link to a screen recording on how to access the functionality and export the CSV file.

==Code and Snapshots==

We have written a method inside the ''export_file_controller.rb''. Once this method is called it queries on the ''name'' array passed from the view and exports the CSV file with the required data.

<pre>
def export_tags
@user_ids = User.where("name IN (?)", params[:names])
@students = AnswerTag.select('answers.*, answer_tags.*').joins(:answer).where("answer_tags.answer_id = answers.id and answer_tags.user_id IN (?)", @user_ids.pluck(:id))
attributes = %w[user_id tag_prompt_deployment_id comments value]

csv_data = CSV.generate(col_sep: ",") do |csv|
csv << attributes
@students.each do |item|
csv << item.attributes.values_at(*attributes)
end
end
filename = "Tags"

send_data csv_data,
type: 'text/csv; charset=iso-8859-1; header=present',
disposition: "attachment; filename=#{filename}.csv"

end
</pre>

The output file has '''user_id, tag_prompt_deployment_id, comments and value'''. We can see that some comments are repeated it is because every review that is received has at least 1 tag for it. Hence we see the repetition. The value column has -1, 0 and 1 as No, not answered and Yes meaning respectively.

[[File:Eb1912csv.png]]

==Design==
The functionality for our project was to implement exporting of tagged comments from the Expertiza system. For this, we will approach the project by using the delegation pattern to add exporting capabilities to the export file controller for exporting tags.

==Teammates==
Mentor - Harsh Agrawal

* Aishwarya Subramanian (asubram7)
* Dyuti De (dde)
* Ankit Mody (amody)

==References==
# [https://github.com/expertiza/expertiza Github Repository for Expertiza]
# [http://wiki.expertiza.ncsu.edu/index.php/Documentation_on_Database_Tables Expertiza Documentation on Database Tables]
# [https://guides.rubyonrails.org/api_app.html Rails Guide Documentation]

File:E1912flow.png

2019-04-02T00:06:07Z

Dde:

E1912 Export Tags

2019-04-02T00:02:22Z

Dde:

==Introduction==
===About Expertiza===
Expertiza is an Open Source Web Application Software managed by National Science Foundation. Expertiza is used by many courses including CSC 517 for assignment management. It has functionalities such as peer reviews, teammate reviews and tagging reviews in which students can provide feedback on other's work which helps peer in better developing the project.

===Motivation===
The Expertiza team currently had to manually fire queries in the database in order to get the tag submissions made by an individual user(student) on each review for a particular assignment. In this way, the team got the data from the entire class, which was then used to feed the Machine Learning algorithms. But many times it used to happen that students made random tags and the ML algorithm was not able to make good predictions out of it, hence in order to solve this issue a new idea was proposed as to select on the students which did not create outliers in the predictions and hence, therefore, it is a good proposal.

===Tasks Identified===
In order to perform this task following files were identified where the code hasbeen added.
* In _answer_tagging_report.html.erb view, a button is added for exporting the student's tagged values
* A checkbox for each row has been added in order to select students whose tags are to be exported(By default all will be selected).
* A method is written in export_file_controller.rb controller named export_tags that exports the CSV file.

==Implementation==
The workflow diagram of our implementation is shown here:

[[File:E1912flowdia.png]]

We have added a default select all checkbox under ''Report tags done by each user'' which enables student selection for exporting tags in the CSV file. We have added a column for the checkbox and a button named '''Export''' which on click generates the CSV file.

[[File:Eb1912Export.png]]

Below is the snippet of the same view ''_answer_tagging_report.html.erb ''

<pre>
<%= form_tag(export_tags, method: 'put',:id => 'tst') do %>
<div id="tab-summary">
<h4 align="center">Summary for each user</h4>
<table class="table table-striped sortable">
<thead>
<tr>
<th class="sorter-true">Username</th>
<th class="sorter-true">Name</th>
<th class="sorter-true">Percentage of tags done</th>
<th class="sorter-true"># tags done</th>
<th class="sorter-true"># tags not done</th>
<th class="sorter-true"># possible tags to do<img src='/assets/info.png' title='Number of answers that fulfil the prerequisite setup for each
tags (Answer of certain question types and has a length of greater than the threshold).'/></th>
<th>Tags export</th>
</tr>
</thead>
<% @user_tagging_report.each do |key, val| %>
<tr>
<td><%= val.user.name(session[:ip]) %></td>
<td><%= val.user.fullname(session[:ip]) %></td>
<td><%= val.percentage.to_s %>%</td>
<td><%= val.no_tagged.to_s %></td>
<td><%= val.no_not_tagged.to_s %></td>
<td><%= val.no_tagable %></td>
<td><%= check_box_tag 'names[]', val.user.name(session[:ip]), true %></td>
</tr>
<% end %>
</table>
</div>
<%= submit_tag "Export",:value => 'Export' ,:form => 'tst',method: 'put' %>
<% end %>
</pre>

==Screen Record==
# [https://drive.google.com/open?id=1B65_N3bWwPlSyiXMhXzXypKvKLN09g0r Here] is the link to a screen recording on how to access the functionality and export the CSV file.

==Code and Snapshots==

We have written a method inside the ''export_file_controller.rb''. Once this method is called it queries on the ''name'' array passed from the view and exports the CSV file with the required data.

<pre>
def export_tags
@user_ids = User.where("name IN (?)", params[:names])
@students = AnswerTag.select('answers.*, answer_tags.*').joins(:answer).where("answer_tags.answer_id = answers.id and answer_tags.user_id IN (?)", @user_ids.pluck(:id))
attributes = %w[user_id tag_prompt_deployment_id comments value]

csv_data = CSV.generate(col_sep: ",") do |csv|
csv << attributes
@students.each do |item|
csv << item.attributes.values_at(*attributes)
end
end
filename = "Tags"

send_data csv_data,
type: 'text/csv; charset=iso-8859-1; header=present',
disposition: "attachment; filename=#{filename}.csv"

end
</pre>

The output file has '''user_id, tag_prompt_deployment_id, comments and value'''. We can see that some comments are repeated it is because every review that is received has at least 1 tag for it. Hence we see the repetition. The value column has -1, 0 and 1 as No, not answered and Yes meaning respectively.

[[File:Eb1912csv.png]]

==Design==
The functionality for our project was to implement exporting of tagged comments from the Expertiza system. For this, we will approach the project by using the delegation pattern to add exporting capabilities to the export file controller for exporting tags.

==Teammates==
Mentor - Harsh Agrawal

* Aishwarya Subramanian (asubram7)
* Dyuti De (dde)
* Ankit Mody (amody)

==References==
# [https://github.com/expertiza/expertiza Github Repository for Expertiza]
# [http://wiki.expertiza.ncsu.edu/index.php/Documentation_on_Database_Tables Expertiza Documentation on Database Tables]
# [https://guides.rubyonrails.org/api_app.html Rails Guide Documentation]

File:E1912flowdia.png

2019-04-01T23:58:11Z

Dde:

E1912 Export Tags

2019-04-01T18:10:19Z

Dde: /* Future work */

E1912 Export Tags

2019-04-01T17:39:46Z

Dde: /* References */

File:Eb1912Export.png

2019-04-01T17:06:33Z

Dde:

File:Eb1912csv.png

2019-04-01T17:05:56Z

Dde:

File:Eb1912Code3.png

2019-04-01T17:05:21Z

Dde:

File:E1912Code2.png

2019-04-01T17:01:20Z

Dde:

E1912 Export Tags

2019-03-25T06:54:01Z

Dde: Created page with "==Introduction== ===About Expertiza=== Expertiza is an Open Source Web Application Software managed by National Science Foundation. Expertiza is used by many courses including CS..."

File:Eb1912code1.png

2019-03-25T06:48:47Z

Dde:

File:Eb1912cb.png

2019-03-25T06:42:00Z

Dde:

File:Checkbox.png

2019-03-25T06:34:24Z

Dde: uploaded a new version of "File:Checkbox.png"