Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:11:29Z

Lguo7:

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:09:33Z

Lguo7: /* Data Flow Diagram */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:07:51Z

Lguo7: /* Use Case */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:05:41Z

Lguo7: /* Manual Testing */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:03:30Z

Lguo7: /* Manual Testing */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:02:07Z

Lguo7: /* Test Plan */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
====1. Log in as a participants/student====
====2. Go to Assignment====
====3. Go to Other's works and check preferred conference paper topics to bid on reviewing====
====4. Click submit and wait for bidding deadline====
====5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing====

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T16:01:13Z

Lguo7: /* Test Plan */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
1. Log in as a participants/student
2. Go to Assignment
3. Go to Other's works and check preferred conference paper topics to bid on reviewing
4. Click submit and wait for bidding deadline
5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T15:49:37Z

Lguo7: /* Use Case */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to
participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T15:48:30Z

Lguo7: /* Choose and submit preference to be reviewed */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T15:47:57Z

Lguo7: /* Use Case */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
Use Case Id: 1
Use Case Description: Participants choose the preference for conference review topics and submit it
Actors: Participants
Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it
====Saving bidder information into database====
Use Case Id: 2
Use Case Description: bidding preferences and related participants information are processed and saved to database
Actors: None
Triggered by: Use Case 1
Pre Conditions: participants preferences are submitted
Post Conditions: information can be retrieved and used by bidding algorithm
====View list of topics====
Use Case Id: 3
Use Case Description: participants can view list of topic available for conference paper topic
Actors: Participants
====Run bidding process====
Use Case Id: 4
Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
Actors: Conference Committee
Pre Conditions: preferences must be submitted by participants
Post Conditions: the bidding result can be used for paper assignment
====Choosing and assigning paper to bidders====
Use Case Id: 5
Use Case Description: System assigns participants to conference paper topics according to bidding result
Actors: None
Triggered by: Use Case 4
Pre Conditions: bidding algorithm has run and result has been returned
Post Conditions: Participants can view topics been assigned to them
====Change assignment of review using CRUD actions====
Use Case Id: 6
Use Case Description: Conference committee members can change assignment result manually
Actors: Conference Committee
Pre Conditions: topic assignment has been done
Post Conditions: changes on bidding result is visible to participants and other committee members

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T15:20:17Z

Lguo7: /* Data Flow Diagram */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin

Sathwick (sgopara@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

=== Data Flow Diagram ===
Below is the Data Flow Diagram for process flows of the project.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T03:57:46Z

Lguo7: /* Test Plan */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

=== Data Flow Diagram ===

[[File:Data Flow.png]]

= Design =

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

new_intelligent_property
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Additional Changes ===
The changes above were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition to the code above, we need to add new code.

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

=====Edge cases=====

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T03:47:38Z

Lguo7: /* Files to be changed */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

=== Data Flow Diagram ===

[[File:Data Flow.png]]

= Design =

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

anti_intelligent_property
topic_per_team
team_per_topic

anti_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic. Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if anti_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !anti_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);

=====Edge cases=====

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T03:43:41Z

Lguo7: /* Automated Test Cases */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Understanding the Flows=
Essentially, we have to design from scratch. Hence, we first describe the use cases, before going into the design phase.

===Use Case===
Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process.

[[File:UML_01.png]]

=== Data Flow Diagram ===

[[File:Data Flow.png]]

= Design =

=== Files to be changed ===
lottery_controller.rb

This is current code for bidding assignment. The code will basically get target assignment (conference paper assignment in our case) and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximem team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

To accommodate this code to fit our needs, we have proposed to add 3 new variable

anti_intelligent_property
topic_per_team
team_per_topic

anti_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic. Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if anti_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !anti_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);

=====Edge cases=====

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T03:32:21Z

Lguo7: /* Test Plan */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers (here M4 represents some constant)
** Each reviewer is assigned a maximum of M3 papers to review (here M3 represents some constant)
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Use Case Diagram=
Essentially, we have to design from scratch. Hence, we first describe some use cases, before going into the design phase.

[[File:UML_01.png]]

[[File:Data Flow.png]]

==Files to be Modified==

* lottery_controller.rb

This is current code for bidding assignment. The code will basically get target assignment (conference paper assignment in our case) and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximem team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

To accommodate this code to fit our needs, we have proposed to add 3 new variable

anti_intelligent_property
topic_per_team
team_per_topic

anti_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic. Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if anti_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !anti_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
**
**
**

=====Edge cases=====

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

* Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T03:26:26Z

Lguo7: /* Files to be Modified */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of M4 reviewers
** Each reviewer is assigned a maximum of M3 papers to review
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

In the algorithm above, the Calculate C(i,j) involves the following:

C(i,j) = 1/N * ∑ [ M1 -S(i) ]

where we do the summation from i = 1 to i = N

* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

Situation in case of a priority tie: In the case that the algorithm groups all students preferring specific topic together and finds that there are more students in the cluster than the maximum team size M1, it evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of the resulted evaluation for assigning topics.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

=Use Case Diagram=
Essentially, we have to design from scratch. Hence, we first describe some use cases, before going into the design phase.

[[File:UML_01.png]]

==Files to be Modified==

* lottery_controller.rb

This is current code for bidding assignment. The code will basically get target assignment (conference paper assignment in our case) and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximem team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

To accommodate this code to fit our needs, we have proposed to add 3 new variable

anti_intelligent_property
topic_per_team
team_per_topic

anti_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic. Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if anti_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !anti_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written.

=====Edge cases=====

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

* Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T02:39:20Z

Lguo7: /* Files to be Modified */

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

=Introduction=

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

===Background of the project===
As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

===Description of project===
As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers
** Each reviewer is assigned a maximum of P papers to review
** Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

=Project Requirements=

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

===Problem Statement===
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Existing Algorithm===
A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:
* Teacher puts out a list of available project topics (let number of topics be N)
* Making a Preference List with minimum 1 topic, maximum M2 topics:
** Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
** Or Every paired teams makes a (combined common) list.
* Bidding Priority Assignment is as follows:
'''for each''' ''topic j''':
'''for each''' ''position i'' '''that topic j has been placed on (in any preference list'''

''largest'' ← ''L''[0]
'''for each''' ''item'' '''in''' ''L'', '''do'''
'''if''' ''item'' > ''largest'', '''then'''
''largest'' ← ''item''
'''return''' ''largest''

For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
C(i,j) = 1/N * :<math>\sum_{i \mathop =1}^{N}(M1 -S(i) ).</math>
*i=1N
Weight(i,j) = B(i) / C(i,j)
* Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
* Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
** Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
** In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
** In each graph, only one team will be selected as per the above parameters

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

The changes related to this project are required for reviewing conference papers and journals. The users are able to submit their works as either individuals or teams to be scrutinized. The other participants are allowed to review the papers of their interest from the list of submitted papers. The participants are allowed to bid on the papers of their preference in a preference list with the top most entry taking the highest priority. The bidding algorithm already exists as called by a web service in Peerlogic. The goal of this project is to generalize the algorithm for both review and topic based bidding.

===Test
The existing bidding algorithm used by the application is only available for team assignment bidding. The algorithm itself groups all team members preferring specific topic together, then evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of resulted evaluation for assigning topics.

====Bidding Algorithm Overview====

Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

Our goal here is as follows:
* To implement a bidding algorithm for conference paper reviewer assignment functionality.
* To attempt to merge the existing and the new algorithm as much as possible

====Possible Extension====

A possible extension is to combine reviewer assignment by bidding with one of the other assignment methods--instructor assignment, or “automated” assignment, where each new reviewer is assigned one of the least-reviewed submissions so far. In this case, submissions that are not bid upon would be assigned to reviewers who “show up later,” as would reviews that are not completed by the reviewers who bid on them.

==Use Case Diagram==

[[File:UML_01.png]]

==Files to be Modified==

* lottery_controller.rb

This is current code for bidding assignment. The code will basically get target assignment (conference paper assignment in our case) and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximem team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

To accommodate this code to fit our needs, we have proposed to add a new variable

anti_intelligent_property

which will be true if bidding is on conference paper review and false if bidding is on team assignment. Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if anti_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !anti_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment)
end
redirect_to controller: 'tree_display', action: 'list'
end

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment function

end def

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written.

=====Edge cases=====

* Case 1: Users signing up after bidding is done need to be assigned the papers/journals with no or least reviews done.

* Case 2: Extending edge case 1, if somebody doesn't provide a list of topics s(he) wants to review, the topic with the least number of bids is assigned.

* Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T01:55:22Z

Lguo7: /* Files to be Modified */

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-15T01:50:49Z

Lguo7: /* Files to be Modified */

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-14T09:45:08Z

Lguo7: /* Problem Statement */

==Introduction==

====Description of project====

Expertiza provides a way for students to bid on different topics for proposed assignment topics. This allows the fair topic distribution among students when multiple students or groups are bidding on same topic or paper.

The changes related to this project are required for reviewing conference papers and journals. The users are able to submit their works as either individuals or teams to be scrutinized. The other participants are allowed to review the papers of their interest from the list of submitted papers. The participants are allowed to bid on the papers of their preference in a preference list with the top most entry taking the highest priority. The bidding algorithm already exists as called by a web service in Peerlogic. The goal of this project is to generalize the algorithm for both review and topic based bidding.

====Bidding Algorithm Overview====

Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

==Problem Statement==
* Reviews are often assigned by having reviewers bid on papers they want to review. This is very similar to the kind of bidding that Expertiza implements for topics, and it should be possible to call the same code (which will be modified to make it more general).

* The reviewer bidding for proposed conference papers has not been fully implemented. Currently the bidding process for assigning topics is either manual or automatic. Manual means the instructor assigns topics based on the preference provided by the students. Automatic assignment means that topics are assigned on the basis of an implemented algorithm, which takes into account some parameters, including intelligent-property.

* Incompatible bidding algorithm. The existing bidding algorithm used by the application is only available for team assignment bidding. The algorithm itself groups all team members preferring specific topic together, then evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of resulted evaluation for assigning topics. This algorithm will not work on our implementation for several reasons:
1. It is team-based where as our implementation is individual-based. Since the implementation is running on a web-service, we won't
be able to modify it as it is not an internal part of Expertiza application;
2. The current bidding implementation only returns one bidding result each time, whereas expected bidding implementation must be able
to return multiple bidding results every time.
3. The current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will
work for our implementation since now each bidding team would only have one person.

Our goal here is as follows:
* To implement a bidding algorithm for conference paper reviewer assignment functionality.
* To attempt to merge the existing and the new algorithm as much as possible

====Possible Extension====

A possible extension is to combine reviewer assignment by bidding with one of the other assignment methods--instructor assignment, or “automated” assignment, where each new reviewer is assigned one of the least-reviewed submissions so far. In this case, submissions that are not bid upon would be assigned to reviewers who “show up later,” as would reviews that are not completed by the reviewers who bid on them.

==Use Case Diagram==

[[File:UML_01.png]]

==Files to be Modified==

* lottery_controller.rb
* New Controller to be specified for review bidding

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written.

=====Edge cases=====

* Case 1: Users signing up after bidding is done need to be assigned the papers/journals with no or least reviews done.

* Case 2: Extending edge case 1, if somebody doesn't provide a list of topics s(he) wants to review, the topic with the least number of bids is assigned.

* Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-14T09:05:20Z

Lguo7: /* Bidding Algorithm Overview */

==Introduction==

====Description of project====

Expertiza provides a way for students to bid on different topics for proposed assignment topics. This allows the fair topic distribution among students when multiple students or groups are bidding on same topic or paper.

The changes related to this project are required for reviewing conference papers and journals. The users are able to submit their works as either individuals or teams to be scrutinized. The other participants are allowed to review the papers of their interest from the list of submitted papers. The participants are allowed to bid on the papers of their preference in a preference list with the top most entry taking the highest priority. The bidding algorithm already exists as called by a web service in Peerlogic. The goal of this project is to generalize the algorithm for both review and topic based bidding.

====Bidding Algorithm Overview====

Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

==Problem Statement==
* Reviews are often assigned by having reviewers bid on papers they want to review. This is very similar to the kind of bidding that Expertiza implements for topics, and it should be possible to call the same code (which will be modified to make it more general).

* The reviewer bidding for proposed conference papers has not been fully implemented. Currently the bidding process for assigning topics is either manual or automatic. Manual means the instructor assigns topics based on the preference provided by the students. Automatic assignment means that topics are assigned on the basis of an implemented algorithm, which takes into account some parameters, including intelligent-property.

Our goal here is as follows:
* To implement a bidding algorithm for conference paper reviewer assignment functionality.
* To attempt to merge the existing and the new algorithm as much as possible

====Possible Extension====

A possible extension is to combine reviewer assignment by bidding with one of the other assignment methods--instructor assignment, or “automated” assignment, where each new reviewer is assigned one of the least-reviewed submissions so far. In this case, submissions that are not bid upon would be assigned to reviewers who “show up later,” as would reviews that are not completed by the reviewers who bid on them.

==Use Case Diagram==

[[File:UML_01.png]]

==Files to be Modified==

* lottery_controller.rb
* New Controller to be specified for review bidding

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written.

=====Edge cases=====

* Case 1: Users signing up after bidding is done need to be assigned the papers/journals with no or least reviews done.

* Case 2: Extending edge case 1, if somebody doesn't provide a list of topics s(he) wants to review, the topic with the least number of bids is assigned.

* Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-11-14T09:04:15Z

Lguo7: /* Bidding Algorithm Overview */

==Introduction==

====Description of project====

Expertiza provides a way for students to bid on different topics for proposed assignment topics. This allows the fair topic distribution among students when multiple students or groups are bidding on same topic or paper.

The changes related to this project are required for reviewing conference papers and journals. The users are able to submit their works as either individuals or teams to be scrutinized. The other participants are allowed to review the papers of their interest from the list of submitted papers. The participants are allowed to bid on the papers of their preference in a preference list with the top most entry taking the highest priority. The bidding algorithm already exists as called by a web service in Peerlogic. The goal of this project is to generalize the algorithm for both review and topic based bidding.

====Bidding Algorithm Overview====
=====Current Bidding Algorithm=====
The Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

==Problem Statement==
* Reviews are often assigned by having reviewers bid on papers they want to review. This is very similar to the kind of bidding that Expertiza implements for topics, and it should be possible to call the same code (which will be modified to make it more general).

* The reviewer bidding for proposed conference papers has not been fully implemented. Currently the bidding process for assigning topics is either manual or automatic. Manual means the instructor assigns topics based on the preference provided by the students. Automatic assignment means that topics are assigned on the basis of an implemented algorithm, which takes into account some parameters, including intelligent-property.

Our goal here is as follows:
* To implement a bidding algorithm for conference paper reviewer assignment functionality.
* To attempt to merge the existing and the new algorithm as much as possible

====Possible Extension====

A possible extension is to combine reviewer assignment by bidding with one of the other assignment methods--instructor assignment, or “automated” assignment, where each new reviewer is assigned one of the least-reviewed submissions so far. In this case, submissions that are not bid upon would be assigned to reviewers who “show up later,” as would reviews that are not completed by the reviewers who bid on them.

==Use Case Diagram==

[[File:UML_01.png]]

==Files to be Modified==

* lottery_controller.rb
* New Controller to be specified for review bidding

==Test Plan==

====Manual Testing====
* UI testing of the implemented functionality to be done.

====Automated Test Cases====
* TDD and Feature Test cases to be written.

=====Edge cases=====

* Case 1: Users signing up after bidding is done need to be assigned the papers/journals with no or least reviews done.

* Case 2: Extending edge case 1, if somebody doesn't provide a list of topics s(he) wants to review, the topic with the least number of bids is assigned.

* Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-06T19:40:52Z

Lguo7: /* Testing */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
Git Original link
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

Git Revised link
*https://github.com/saiharsha1994/expertiza/tree/Bronzeproject

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

== Design Pattern Usage ==
We implemented using the MVC (Model View Controller) pattern. We felt it was the pattern used in expertiza , usage of same pattern would be much easier.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average (vm_q1 object);

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation (vm_q2 object).

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see in code for vm_q1 and vm_q3 objects). However, due to decrease in coverage for the test (after we added it, the coverage dropped by 0.4%), we have reverted it back to original 2 test cases. The additional test cases can be found below in snapshots, and original test cases with highest coverage in on github.

test file: vm_question_response_rows_spec.rb;

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where an extra parameter - score row - is needed during object initialization process. It would be a more efficient constructor to use when retrieving scores from database as it allows program to initialize a VmQuestionResponseRow object now with a list of AssignmentScore objects instead of having to initialize to an empty list and putting in AssignmentScore objects one by one through a loop. (Note: the implementation for using constructor with extra parameter in project is not implemented as it is originally not part of our requirement)

== Test implementation ==

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:rspec1787_1.png]]

[[File:rspec1787_2.png]]

[[File:rspec1787_3.png]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-06T19:39:48Z

Lguo7: /* Testing */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
Git Original link
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

Git Revised link
*https://github.com/saiharsha1994/expertiza/tree/Bronzeproject

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

== Design Pattern Usage ==
We implemented using the MVC (Model View Controller) pattern. We felt it was the pattern used in expertiza , usage of same pattern would be much easier.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average (vm_q1 object);

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation (vm_q2 object).

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see in code for vm_q1 and vm_q3 objects). However, due to decrease in coverage for the test (after we added it, the coverage dropped by 0.4%), we have reverted it back to original 2 test cases.

test file: vm_question_response_rows_spec.rb;

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where an extra parameter - score row - is needed during object initialization process. It would be a more efficient constructor to use when retrieving scores from database as it allows program to initialize a VmQuestionResponseRow object now with a list of AssignmentScore objects instead of having to initialize to an empty list and putting in AssignmentScore objects one by one through a loop. (Note: the implementation for using constructor with extra parameter in project is not implemented as it is originally not part of our requirement)

== Test implementation ==

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:rspec1787_1.png]]

[[File:rspec1787_2.png]]

[[File:rspec1787_3.png]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-06T19:25:49Z

Lguo7: /* Test implementation */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
Git Original link
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

Git Revised link
*https://github.com/saiharsha1994/expertiza/tree/Bronzeproject

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

== Design Pattern Usage ==
We implemented using the MVC (Model View Controller) pattern. We felt it was the pattern used in expertiza , usage of same pattern would be much easier.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average (vm_q1 object);

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation (vm_q2 object).

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see in code for vm_q1 and vm_q3 objects).

test file: vm_question_response_rows_spec.rb;

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where an extra parameter - score row - is needed during object initialization process. It would be a more efficient constructor to use when retrieving scores from database as it allows program to initialize a VmQuestionResponseRow object now with a list of AssignmentScore objects instead of having to initialize to an empty list and putting in AssignmentScore objects one by one through a loop. (Note: the implementation for using constructor with extra parameter in project is not implemented as it is originally not part of our requirement)

== Test implementation ==

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:rspec1787_1.png]]

[[File:rspec1787_2.png]]

[[File:rspec1787_3.png]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

File:Rspec1787 3.png

2017-11-06T19:25:07Z

Lguo7:

File:Rspec1787 2.png

2017-11-06T19:24:58Z

Lguo7:

File:Rspec1787 1.png

2017-11-06T19:24:07Z

Lguo7:

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-06T19:18:25Z

Lguo7: /* Testing */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
Git Original link
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

Git Revised link
*https://github.com/saiharsha1994/expertiza/tree/Bronzeproject

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

== Design Pattern Usage ==
We implemented using the MVC (Model View Controller) pattern. We felt it was the pattern used in expertiza , usage of same pattern would be much easier.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average (vm_q1 object);

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation (vm_q2 object).

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see in code for vm_q1 and vm_q3 objects).

test file: vm_question_response_rows_spec.rb;

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where an extra parameter - score row - is needed during object initialization process. It would be a more efficient constructor to use when retrieving scores from database as it allows program to initialize a VmQuestionResponseRow object now with a list of AssignmentScore objects instead of having to initialize to an empty list and putting in AssignmentScore objects one by one through a loop. (Note: the implementation for using constructor with extra parameter in project is not implemented as it is originally not part of our requirement)

== Test implementation ==

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:Rspec123.png|center]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-06T19:12:53Z

Lguo7: /* Our Work */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
Git Original link
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

Git Revised link
*https://github.com/saiharsha1994/expertiza/tree/Bronzeproject

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

== Design Pattern Usage ==
We implemented using the MVC (Model View Controller) pattern. We felt it was the pattern used in expertiza , usage of same pattern would be much easier.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average;

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation.

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see code).

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where an extra parameter - score row - is needed during object initialization process. It would be a more efficient constructor to use when retrieving scores from database as it allows program to initialize a VmQuestionResponseRow object now with a list of AssignmentScore objects instead of having to initialize to an empty list and putting in AssignmentScore objects one by one through a loop. (Note: the implementation for using constructor with extra parameter in project is not implemented as it is originally not part of our requirement)

== Test implementation ==

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:Rspec123.png|center]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-05T08:03:50Z

Lguo7: /* Github Link */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
Git Original link
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

Git Revised link
*https://github.com/saiharsha1994/expertiza/tree/Bronzeproject

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average;

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation.

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see code).

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where if specific score given is over certain upper-limit x or lower-limit y, the score will be automatically reset to x or y accordingly so that the score will alway be in between x and y.

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:Rspec123.png|center]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-05T00:17:39Z

Lguo7: /* Our Work */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average;

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation.

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see code).

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, a meeting once every week for 2 weeks with TA was proceeded. The main focus for the first one and half week involves setting up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, which were solved through file tracing, problem identifying, and debugging. For the rest of these a little over 2 weeks period, several issues were identified with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, it was identified that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where if specific score given is over certain upper-limit x or lower-limit y, the score will be automatically reset to x or y accordingly so that the score will alway be in between x and y.

Afterward, several tests were performed for the model class modifications using RSpec, with some of the code as shown below:

[[File:Rspec123.png|center]]

For this project, techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc, were used.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-05T00:11:59Z

Lguo7: /* Our Work */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average;

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation.

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see code).

==Our Work==

For the OSS project, the topic is fixing and modification on Expertize team score calculation mechanism. For this project, we have meet once every week for 2 weeks either with TA. For the first one and half week, each of us has set up the environment on the machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, and we worked together to identify the issues and finding ways to get around them. For the rest of these a little over 2 weeks period, we have identified several issues with the project. The main issue for program include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, we found that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

The issue can be solved by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: not_null_reviews.zero? check are used two times in order to pass the github check on coding, as combining these two lines into one (row_average_score = (row_average_score / not_null_reviews).round(2) unless not_null_reviews.zero?) would resulting in failing on github check, while the other solution which is putting both line into an unless block would not have any benefit execution efficiency wise).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where if specific score given is over certain upper-limit x or lower-limit y, the score will be automatically reset to x or y accordingly so that the score will alway be in between x and y.

Afterward, we have testing the model class modifications using RSpec, with code as shown below:

[[File:Rspec123.png|center]]

For this project, we have used techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-05T00:06:29Z

Lguo7: /* Our Work */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average;

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation.

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see code).

==Our Work==

For the OSS project, our topic is fixing and modification on Expertize team score calculation mechanism. For this project, we have meet once every week for 2 weeks either with TA. For the first one and half week, each of us has set up the environment on our machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, and we worked together to identify the issues and finding ways to get around them. For the rest of these a little over 2 weeks period, we have identified several issues with our project. The main issue for our part include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, we found that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

We proceeded to solve the issue by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below (note: we used: not_null_review.zero? check two times in order to pass the github check on coding, as combining these two lines into one would resulting in failing on github check).

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where if specific score given is over certain upper-limit x or lower-limit y, the score will be automatically reset to x or y accordingly so that the score will alway be in between x and y.

Afterward, we have testing the model class modifications using RSpec, with code as shown below:

[[File:Rspec123.png|center]]

For this project, we have used techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-11-05T00:01:19Z

Lguo7: /* Testing */

==Introduction==
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

==Scenario==

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Issue -

[[File:error_1787.png|center]]

==Solution==

Solution -

[[File:Grades.png|center]]

===Github Link===
*https://github.com/saiharsha1994/expertiza

Pull request link
*https://github.com/expertiza/expertiza/pull/1058

===Explanation===
====Average Calculation====
The scores are not calculated on dividing with total reviews given, instead calculated using total non null reviews given.

====Testing====
We have run unit test using RSpec and Carpybara. The test file path is spec/models/vm_question_response_row_spec.rb.

For the test, we have identified 2 scenarios. First one is regarding correctness on average score calculation, where we will create a VmQuestionResponseRow object with sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average;

Second one is regarding whether average score calculation includes nil values, where we will create a VmQuestionResponseRow object with one or more nil sample test scores inside and call its average_score_for_row method, then we match returned average to the expected average to check if nil is redefined and used in score calculation.

In addition, we have also tested the initialization of VmQuestionResponseRow object with 5 parameters and 6 parameters to ensure the initialization process works as intended (see code).

==Our Work==

For the OSS project, our topic is fixing and modification on Expertize team score calculation mechanism. For this project, we have meet once every week for 2 weeks either with TA. For the first one and half week, each of us has set up the environment on our machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, and we worked together to identify the issues and finding ways to get around them. For the rest of these a little over 2 weeks period, we have identified several issues with our project. The main issue for our part include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, we found that the reason for such issue involves average_score_for_row method in vm_question_response_row.rb in model as shown below in red - where the row_average_score dividend was not been used correctly.

Comparison of code changes can be seen in Pull request.

We proceeded to solve the issue by modify the calculation for dividend using a counter to exclude nil values and modified the constructor for VmQuestionResponseRow classs that it can dynamically accept 5 or 6 parameters, as shown below:

Code after first modification -

[[File:code.png|1047px|thumb|center]]

Code after second modification -

[[File:code_2.png|1047px|thumb|center]]

The first modification was done as it fix the issue while keep code at its most simplicity; (As shown in screen shot)

The second modification was done to accommodate the scenario where if specific score given is over certain upper-limit x or lower-limit y, the score will be automatically reset to x or y accordingly so that the score will alway be in between x and y.

Afterward, we have testing the model class modifications using RSpec, with code as shown below:

[[File:Rspec123.png|center]]

For this project, we have used techniques such as Unit testing, constructor overloading, and technologies such as bower, carpybara, RSpec, etc.

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-10-27T01:35:34Z

Lguo7:

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-10-27T01:32:34Z

Lguo7:

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-10-27T01:30:27Z

Lguo7:

[[Introduction]]
[[Scenario]]
[[Solution]]
[[Our Work]]
Introduction:
This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

Scenario:

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Solution:

Our Work:

For the OSS project, our topic is fixing and modification on Expertize team score calculation mechanism. For this project, we have meet once every week for 2 weeks either with TA. For the first one and half week, each of us has set up the environment on our machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, and we worked together to identify the issues and finding ways to get around them. For the rest of these a little over 2 weeks period, we have identified the issues with our project. The main issue for our part include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, we found that the reason for such issue involves … (code/file part that is not working correctly and why its behavior is not wanted) ... . Our proposed solution for issue is … (proposed solution here) … . This will involve skills we have learned in class such as unit testing using …(expand on skills) … .

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-10-27T01:29:16Z

Lguo7:

[[Introduction]]

This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

Scenario:

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Solution:

Our Work:

For the OSS project, our topic is fixing and modification on Expertize team score calculation mechanism. For this project, we have meet once every week for 2 weeks either with TA. For the first one and half week, each of us has set up the environment on our machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, and we worked together to identify the issues and finding ways to get around them. For the rest of these a little over 2 weeks period, we have identified the issues with our project. The main issue for our part include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, we found that the reason for such issue involves … (code/file part that is not working correctly and why its behavior is not wanted) ... . Our proposed solution for issue is … (proposed solution here) … . This will involve skills we have learned in class such as unit testing using …(expand on skills) … .

CSC/ECE 517 Fall 2017/E1787 OSS project Bronze Score calculations

2017-10-27T01:27:21Z

Lguo7: Created page with "Introduction: This project involves revision of score calculation bug of Expertiza homework reviewing mechanism. Scenario: Sometimes when a reviewee of a project or homework ..."

Introduction:

This project involves revision of score calculation bug of Expertiza homework reviewing mechanism.

Scenario:

Sometimes when a reviewee of a project or homework fills out the review form, he or she may leave certain review question blank. When taken into score calculation for the project or homework, the application instead fills in 0 on blank answers. This behavior is incorrect as the blank review answers should never be used when used to calculate final score.

Solution:

Our Work:

For the OSS project, our topic is fixing and modification on Expertize team score calculation mechanism. For this project, we have meet once every week for 2 weeks either with TA. For the first one and half week, each of us has set up the environment on our machine. Some of notable issues on setting up include theRubyRacer dependency problem on project, Node packages not recognized by rails (Windows), Font-Awesome-Rails path issues, etc, and we worked together to identify the issues and finding ways to get around them. For the rest of these a little over 2 weeks period, we have identified the issues with our project. The main issue for our part include the false score calculation mechanism by expertiza where empty scores are assumed automatically to 0 and included in calculation; this would in turn give a false score calculation result as empty scores are not supposed to be set to 0 and then included in for calculation. Through debugging and defining the problem, we found that the reason for such issue involves … (code/file part that is not working correctly and why its behavior is not wanted) ... . Our proposed solution for issue is … (proposed solution here) … . This will involve skills we have learned in class such as unit testing using …(expand on skills) … .

CSC/ECE 517 Fall 2017

2017-10-27T01:27:02Z

Lguo7:

'''Writing Assignment 2'''
*[[CSC/ECE 517 Fall 2017/E1779. Fix teammate advertisements and requests to join a team]]
*[[CSC/ECE_517_Fall_2017/E1773 Investigate and Fix Expertiza Production Version Runtime Exceptions.rb]]
*[[CSC/ECE_517_Fall_2017/E1774 Metareview fixes and improvements.rb]]
*[[CSC/ECE_517_Fall_2017/E1788 OSS project Maroon Heatmap fixes]]
*[[CSC/ECE 517 Fall 2017/E1781 Topic Management]]
*[[CSC/ECE_517_Fall_2017/E1745_Refactor_response_controller.rb]]
*[[CSC/ECE_517_Fall_2017/E1752 Refactor assignments controller]]
*[[CSC/ECE_517_Fall_2017/E1776_Enhance_Imports]]
*[[CSC/ECE_517_Fall_2017/E1756 TLD Refactor response.rb]]
*[[CSC/ECE_517_Fall_2017/E1772_Refactor reputation_web_service_controller.rb]]
*[[CSC/ECE_517_Fall_2017/M1754_Mutation Testing on Servo]]
*[[CSC/ECE_517_Fall_2017/E1753 OSS project bidding tests]]
*[[CSC/ECE_517_Fall_2017/E1766_Test team functionality]]
*[[CSC/ECE_517_Fall_2017/E1787_OSS project Bronze Score calculations]]

User:Lguo7

2017-10-27T01:22:29Z

Lguo7: Blanked the page

User:Lguo7

2017-10-27T01:22:03Z