Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T16:30:08Z

Bsingh8: /* Key Changes in Part B of the Design */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favours increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: D(i,j) = 1/N * ∑ [ M -U(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let U(i) = number of students who selected topic j at position i of their (individual) preference list
E(i) = 1/U(i)
Calculate D(i,j)
Weight(i,j) = E(i) / D(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the web service, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignment. Hence we completely change this section and implement it using a new algorithm.

== Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calculate m1 = floor (n1 * n3 / n2)
* Calculate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviewer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Re factoring the methods: The method that we are working on was very long. We will re factor it into several smaller methods of around 20 lines (altogether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common score list, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the score list belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from score list)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores corresponding to that topic from score list)
** Continue the above process until the score list is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readable. We divided it into several methods

==== New Assignment Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes : Given in the next sub section

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assigned topics such that each topic has more or less the same number of reviewers assigned.
* There can be two cases in which this does not happen: more number of teams via-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Use of Latest Expertiza Code Base ====
In the latest revision, we have used the latest expertiza code base for our project

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review : In this case all topics are assigned same bid score. Topics are assigned to users on the basis of "balanced assignment", first come first serve.

* Case 2: All reviewers submit exactly the same list of topics to review: It will be first come, first serve.

* Case 3: Number of topics exceed the number of teams: Some topics may not be reviewed,depending on the limits put by the conference. It is responsibility of administrator to add more teams in such a situation

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T16:29:08Z

Bsingh8: /* Use of Conference_slot variable */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favours increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: D(i,j) = 1/N * ∑ [ M -U(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let U(i) = number of students who selected topic j at position i of their (individual) preference list
E(i) = 1/U(i)
Calculate D(i,j)
Weight(i,j) = E(i) / D(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the web service, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignment. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calculate m1 = floor (n1 * n3 / n2)
* Calculate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviewer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Re factoring the methods: The method that we are working on was very long. We will re factor it into several smaller methods of around 20 lines (altogether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common score list, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the score list belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from score list)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores corresponding to that topic from score list)
** Continue the above process until the score list is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readable. We divided it into several methods

==== New Assignment Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes : Given in the next sub section

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assigned topics such that each topic has more or less the same number of reviewers assigned.
* There can be two cases in which this does not happen: more number of teams via-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Use of Latest Expertiza Code Base ====
In the latest revision, we have used the latest expertiza code base for our project

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review : In this case all topics are assigned same bid score. Topics are assigned to users on the basis of "balanced assignment", first come first serve.

* Case 2: All reviewers submit exactly the same list of topics to review: It will be first come, first serve.

* Case 3: Number of topics exceed the number of teams: Some topics may not be reviewed,depending on the limits put by the conference. It is responsibility of administrator to add more teams in such a situation

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T15:10:26Z

Bsingh8:

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favours increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: D(i,j) = 1/N * ∑ [ M -U(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let U(i) = number of students who selected topic j at position i of their (individual) preference list
E(i) = 1/U(i)
Calculate D(i,j)
Weight(i,j) = E(i) / D(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the web service, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignment. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calculate m1 = floor (n1 * n3 / n2)
* Calculate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviewer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Re factoring the methods: The method that we are working on was very long. We will re factor it into several smaller methods of around 20 lines (altogether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common score list, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the score list belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from score list)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores corresponding to that topic from score list)
** Continue the above process until the score list is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readable. We divided it into several methods

==== New Assignment Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :
all_topics.each do |topic|
#getting the max limit of each topic into a hash map
max_limit_of_topics.store(topic.id,topic.max_choosers)
max_topics_for_assignment =max_topics_for_assignment + topic.max_choosers
end
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#Here we are checking whether max limit of the particular topic reached or not. if not we then assign a team to that topic.
if((incomplete_topics[s.topic_id]<max_limit_of_topics[s.topic_id]) && (incomplete_teams[s.team_id]<assignment.max_reviews_per_submission))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assigned topics such that each topic has more or less the same number of reviewers assigned.
* There can be two cases in which this does not happen: more number of teams via-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Use of Latest Expertiza Code Base ====
In the latest revision, we have used the latest expertiza code base for our project

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review : In this case all topics are assigned same bid score. Topics are assigned to users on the basis of "balanced assignment", first come first serve.

* Case 2: All reviewers submit exactly the same list of topics to review: It will be first come, first serve.

* Case 3: Number of topics exceed the number of teams: Some topics may not be reviewed,depending on the limits put by the conference. It is responsibility of administrator to add more teams in such a situation

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T15:02:17Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :
all_topics.each do |topic|
#getting the max limit of each topic into a hash map
max_limit_of_topics.store(topic.id,topic.max_choosers)
max_topics_for_assignment =max_topics_for_assignment + topic.max_choosers
end
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#Here we are checking whether max limit of the particular topic reached or not. if not we then assign a team to that topic.
if((incomplete_topics[s.topic_id]<max_limit_of_topics[s.topic_id]) && (incomplete_teams[s.team_id]<assignment.max_reviews_per_submission))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assinged topics such that each topic has more or less the same number of reviwers assigned.
* There can be two cases in which this does not happen: more number of teams vis-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Use of Latest Expertiza Code Base ====
In the latest revision, we have used the latest expertiza code base for our project

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T15:00:58Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :
all_topics.each do |topic|
#getting the max limit of each topic into a hash map
max_limit_of_topics.store(topic.id,topic.max_choosers)
max_topics_for_assignment =max_topics_for_assignment + topic.max_choosers
end
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#Here we are checking whether max limit of the particular topic reached or not. if not we then assign a team to that topic.
if((incomplete_topics[s.topic_id]<max_limit_of_topics[s.topic_id]) && (incomplete_teams[s.team_id]<assignment.max_reviews_per_submission))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assinged topics such that each topic has more or less the same number of reviwers assigned.
* There can be two cases in which this does not happen: more number of teams vis-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Use of Latest Expertiza Code Base ====
In the latest revision, we have used the latest expertiza code base for our project

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T14:59:57Z

Bsingh8: /* Use of Conference_slot variable */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :
all_topics.each do |topic|
#getting the max limit of each topic into a hash map
max_limit_of_topics.store(topic.id,topic.max_choosers)
max_topics_for_assignment =max_topics_for_assignment + topic.max_choosers
end
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#Here we are checking whether max limit of the particular topic reached or not. if not we then assign a team to that topic.
if((incomplete_topics[s.topic_id]<max_limit_of_topics[s.topic_id]) && (incomplete_teams[s.team_id]<assignment.max_reviews_per_submission))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assinged topics such that each topic has more or less the same number of reviwers assigned.
* There can be two cases in which this does not happen: more number of teams vis-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T14:59:37Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :
all_topics.each do |topic|
#getting the max limit of each topic into a hash map
max_limit_of_topics.store(topic.id,topic.max_choosers)
max_topics_for_assignment =max_topics_for_assignment + topic.max_choosers
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
#Here we are checking whether max limit of the particular topic reached or not. if not we then assign a team to that topic.
if((incomplete_topics[s.topic_id]<max_limit_of_topics[s.topic_id]) && (incomplete_teams[s.team_id]<assignment.max_reviews_per_submission))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

==== Balancing of Assignment of Topics to Reviewers ====
* This was discussed above
* Basically, in the case none/some of the teams have no preferences, then these teams should be assinged topics such that each topic has more or less the same number of reviwers assigned.
* There can be two cases in which this does not happen: more number of teams vis-a-vis topics or vice versa.
* File Changed: lottery_controller.rb
* Code Changes:

#we are adjusting the max limit of teams with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_teams = ((all_topics.length * assignment.max_reviews_per_submission) / teams.length.to_f).ceil
balanced_max_limit_of_teams = [balanced_max_limit_of_teams , assignment.max_reviews_per_submission].min
#Assigning topics to teams based on highest score
sorted_list.each do |s|
#we are adjusting the max limit of topics with below formaula so that topic assignments is done in a balancing way
balanced_max_limit_of_topics = ((teams.length * max_limit_of_topics[s.topic_id]) / all_topics.length.to_f).ceil
balanced_max_limit_of_topics = [balanced_max_limit_of_topics , max_limit_of_topics[s.topic_id]].min
if((incomplete_topics[s.topic_id]< (s.score < base ? max_limit_of_topics[s.topic_id] : balanced_max_limit_of_topics) ) && (incomplete_teams[s.team_id]< (s.score < base ? assignment.max_reviews_per_submission : balanced_max_limit_of_teams) ))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T08:15:10Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

==== Use of Conference_slot variable ====
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :
all_topics.each do |topic|
#getting the max limit of each topic into a hash map
max_limit_of_topics.store(topic.id,topic.max_choosers)
max_topics_for_assignment =max_topics_for_assignment + topic.max_choosers
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
#Here we are checking whether max limit of the particular topic reached or not. if not we then assign a team to that topic.
if((incomplete_topics[s.topic_id]<max_limit_of_topics[s.topic_id]) && (incomplete_teams[s.team_id]<assignment.max_reviews_per_submission))
SignedUpTeam.create(team_id: s.team_id, topic_id: s.topic_id)
incomplete_teams[s.team_id]+= 1
incomplete_topics[s.topic_id]+= 1
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T08:05:56Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

**** Use of Conference_slot variable ****
While we defined some of our own variables when creating the new functionality, we also have used the conference_slots option to control the assignment of topics to reviewers.
* Code Changes :

****

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as an instructor
** Go to an Assignment
** Go to topics section, add few topics for the assignment. Later check the enable bidding checkbox button.
** Go to add participants page, add few student to the assignment.
** Now save the assignment.
** Login as student (who is a participant of the assignment)
** Go to the assignment page, then go to sign up sheet and bid for topics.
** Do the above the step for the participants.
** Login back to an instructor, go to manage assignments page, click on the run intelligent assignment button.
** Go to the topics tab in the assignment, here you will be able to view all the team's assignments to their topics.

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T08:03:17Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T08:02:27Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T08:01:32Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

==== Forming teams even for conference reviewers ====
* We decided to make Part A run even for conference reviewers. Thus, even reviewers can form teams.
* Files Changed: Lottery_controller.rb
* Code:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:59:17Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:58:48Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

==== New Assignemnt Type ====
* We realize that we must create the conference "assignment" to be compatible with the whole of Expertiza, and it must be different from a course assignment. To do this, we create a new column "is_conference?" in the assignment table
* This variable is boolean, and if true, then it means the assignment is a conference assignment
* We created a migration for the same
* Code:

class AddIsConferenceToAssginment < ActiveRecord::Migration
def change
add_column :assignments, :is_conference?, :boolean , default: false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:55:31Z

Bsingh8: /* New Bid Object Model */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** When all the reviewers submit their preferences.
** When there are no preferences from any user.
** When few have their preferences and few don't.

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:55:07Z

Bsingh8: /* New Bid Object Model */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming
attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end
def persisted?
false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:54:50Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming

attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end

def persisted?
false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:54:04Z

Bsingh8: /* Log: Key Changes */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== New Bid Object Model ====
Following the design methodology, we defined a new model to store the "scorelist" of bids along with the topics and the team ids.

* Files created: bid_score.rb
* Code:

class BidScore
include ActiveModel::Validations
include ActiveModel::Conversion
extend ActiveModel::Naming

attr_accessor :score, :team_id, :topic_id
validates_presence_of :name, :team_id, :topic_id
# Since the toics assignment is done by sorting of scores, this object keeps track of every bid: score, topic and team
def initialize(score,teamId,topicId)
@score = score
@team_id = teamId
@topic_id = topicId
end

def persisted?
false
end
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:50:22Z

Bsingh8: /* Method Refactoring */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== Balanced Assignemnt of Topics ====
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:49:55Z

Bsingh8: /* Changing Existing Files */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Log: Key Changes ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined. We discuss these changes in the following sub sections:

==== Method Refactoring ====
* Files Changed: lottery_controller.rb
* Description of Change: Earlier the lottery_controller.rb consisted of one method of 50 lines, which is not readible. We divided it into several methods, as shown below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
if !assignment.is_conference?
begin
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid assignment
end
redirect_to controller: 'tree_display', action: 'list'
end

* As can be seen, not all methods are shown. However, the new methods size is more or less the same size.

==== Balanced Assignemnt of Topics ====
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T07:41:55Z

Bsingh8: /* Key Changes in Part B of the Design */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
n4 = R
where the meanings of R and P are as above

* Then calcucate m1 = floor (n1 * n3 / n2)
* Calcuate m2 = floor (n2 * n4 / n1)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , m1) topics, provided that the score of the bid is lesser then the base score
* The topics are assigned on the basis of this order of scores, with the restriction that one reviwer should not be assigned more then P topics, and each topic should not get more then the minimum of (R , m2) reviewers, provided that the score of the bid is lesser then the base score
* When the bid score is greater then the base score, then the normal test conditions apply
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T06:27:53Z

Bsingh8: /* Changing Existing Files */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
where the meanings of R and P are as above

* Then calcucate n = floor (n1 * n3 / n2)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , n) topics.
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
We extended the code for bidding in the lottery controller, made a separate type of assignment in the database using a new column "is_conference?". In addition, a new bid object model was also defined.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T01:14:50Z

Bsingh8: /* Key Changes in Part B of the Design */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
where the meanings of R and P are as above

* Then calcucate n = floor (n1 * n3 / n2)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , n) topics.
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

== Design Methodology ==
The design methodologies that we will use while writing the code is discussed here.

* We will implement this new algorithm such that the code is DRY
* Use of naming conventions: To define the new variable to differentiate between project and conference topics as is_conference?
* Refactoring the methods: The method that we are working on was very long. We will refactor it into several smaller methods of around 20 lines (altoghether, these new smaller methods should be visible on a screen as a whole)
* No hard coding of the data assignments
* Use of new database objects if required
* To understand and to use existing database column values, even in the case they are not used in the situation as of now. We have to implement this with the entry conference_slots

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T01:05:33Z

Bsingh8: /* Design */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

== Key Changes in Part B of the Design ==
In the 2 part design proposed in the above section, there were major changes in part B. These are discussed below:

====Proposed Score Calculation Algorithm====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
where the meanings of R and P are as above

* Then calcucate n = floor (n1 * n3 / n2)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , n) topics.
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T01:03:38Z

Bsingh8: /* Assignment of Topics in Part B */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores of Part B====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics in Part B====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
where the meanings of R and P are as above

* Then calcucate n = floor (n1 * n3 / n2)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , n) topics.
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T01:02:51Z

Bsingh8: /* Meaning of the Scores of Part B */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores of Part B====
* As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.

==== Assignment of Topics in Part B====
* All scores are arranged in ascending order.
* For the following values:

n1 = total number of teams
n2 = total number of topics
n3 = P
* Then calucate n = floor (n1 * n3 / n2)
* The topics are assigned on the basis of this order of scores, with the restriction that one topic should not be assigned to more then R reviewers, and each reviewer should not get more then the minimum of (P , n) topics.
* This last modification is done for the balancing of assignment of topics in the case no team has submitted a preference list. In this case, this change tries to ensure that every topic is assigned in such a way that there are more or less equal number of reviewers reviewing each topic.

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T00:55:06Z

Bsingh8: /* Design */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Meaning of the Scores of Part B====
As we can see, each user gets a separate score for each topic there is to bid for. The scores are assigned on the basis of the preference list submitted.
* Two users can get identical scores only when their preference list is identical. Even if user 2 has all topics of user 1 except the last one in the same order as user 1, still all topics of both users get different scores.
* All scores are arranged in ascending order. The topics are assinged on the basis of this order of scores, with the restriction that one topic should not be assigned to more then P reviewers, and each reviewer should not get more then T topics.

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T00:48:13Z

Bsingh8: /* Design */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M (M is a positive number)
* Based on preference lists submitted by people/teams having a minimum of 0 items and a maximum of L items, to allot items to teams such that each team gets at most P items and each item (to be bid on) is assigned to at most R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M >= 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = 1 / (num/den + B)
if(user has preferred no topic)
score[topic][user] = 1 / B
if(user has not preferred this topic)
num = B * N
score[topic][user] = 1/ (B - num/den)

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T00:44:00Z

Bsingh8: /* Existing Algorithm */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 0 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T00:43:16Z

Bsingh8: /* Project Requirements */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer
* To ensure that the topic assignment algorithm assigns topics in a balanced manner in the case no reviewer has bid for any topic
* To develop a variable name in the database so as to distinguish between a project topic and a reviewer topic

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-25T00:39:03Z

Bsingh8:

This page describes the project E17A7 which is one of the several projects to allow Expertiza support a conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the functionality for bidding for topics is present though only for specific situations. The bidding ability is only for bidding for a topic for (teams of) students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is responsible for the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers can be individual or team based

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-13T16:26:28Z

Bsingh8: /* Project Requirements */

This page describes the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifically, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is a software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then post a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in a team) with close preferences are assigned to a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams, the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section, we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment) and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also, the second requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are allocated to teams such that each team gets a unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user, not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X-axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully, more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving topic and participant information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics and bid====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding Algorithm====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Display assigned paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes in bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline, all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algorithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process until the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Changing Existing Files ==
This subsection details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for the bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward, it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as shown below:
The update involves the use of the existing assignment bidding tables since the UI and table entries specific to conference paper bidding has not been created and is out of scope for this project.

Proposal: Allow the instructor to choose whether is it a topic assignment bid or conference paper bid via an update of a flag variable from the UI. For example, the is_assignment variable is used to indicate if it is an assignment bidding. If not the flag is set to false and assumes it is a conference bid. This enables the reuse of existing precondition variables and table entries. The bidding is run by calculating the weighted scores for each team's bidding priority of all the topics. Score calculation is done as explained in the Design section.

Upon score calculation, the topics are used to populate the singed_up table for the corresponding teams and topic id. The resulting assigned topics can be viewed through the sing up sheet view by each participant. Each topic can be assigned to multiple teams as denoted by the max_per_topic variable. A hash map is used to keep track of how many times the same topic has been assigned to a team. Hence we are able to restrict the number of times the same paper/topic can be reviewed.

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
incomplete_teams = Hash.new(0)
incomplete_topics = Hash.new(0)
all_topics=[]
all_topics = assignment.sign_up_topics
score_list=Array.new(teams.length*all_topics.length){Array.new(3)}
sorted_list=Array.new(teams.length*all_topics.length){Array.new(3)}
temp=[]
base = 10
p = 0
#looping through each team to calculate score for each topic in the assignment
teams.each do |t|
incomplete_teams.store(t.id,0)
team_bids = Bid.where(team_id: t.id)
denom = 0
b_length = team_bids.length
(1..b_length).each do |i|
denom = denom+all_topics.length - i
end
#Score calculation based on bid priority
all_topics.each do |j|
if(team_bids.any?{|tb| tb.topic_id == j.id})
bid_priority = Bid.where(team_id:t.id,topic_id:j.id).first.priority
score = base+((all_topics.length+1-bid_priority)*base*all_topics.length) / denom
elsif(team_bids.length!=0)
score = base-(base*all_topics.length)/denom
else
score = base
end
score_list[p][0] = score
score_list[p][1] = t.id
score_list[p][2] = j.id
p+=1
end
end
sorted_list = score_list.sort_by{|e| [e[0],e[1],e[2]]}.each{|line| p line}
sorted_list.reverse!
all_topics.each do |k|
incomplete_topics.store(k.id,0)
end
if(all_topics.length*team_per_topic<teams.length)
flash[:error] = 'There are not enough reviews to be assigned'
end

#Assigning topics to teams based on highest score
sorted_list.each do |s|
if((incomplete_topics[s[2]]<team_per_topic) && (incomplete_teams[s[1]]<topic_per_team))
SignedUpTeam.create(team_id: s[1], topic_id: s[2])
incomplete_teams[s[1]]+=1
incomplete_topics[s[2]]+=1
end
end
assignment.update_attribute(:is_intelligent,false)
flash[:notice] = 'The intelligent assignment was successfully completed for ' + assignment.name + '.'
end

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.
* No UI has been created been created for the conference paper review bidding. Hence assignment bidding controller is used to manually test the code on a local host to determine the feasibility of code.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance, we will do Rspec test in cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is matched first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many teams with same first preference and assigning many teams with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create an object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submit exactly the same list of topics to review.

* Case 3: Number of teams exceed the number of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T21:20:19Z

Bsingh8: /* Data Flow Diagram */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

We explain in the terms of the actors: Reviewer and Administrator
* Before the deadline all the reviewers have to submit preference list of papers. They can save, modify list as many times as they want, but they can submit once
* This information is saved in the database
* When the deadline is passed, stop accepting preference lists
* The administrator runs the paper assignment algroithm
** For every topic, calculate the score for each user
** Make a common scorelist, which keeps score assigned to every topic of every user. There should be a way to find out the topic and user a particular element of the scorelist belongs to
** Now start assigning topics according to highest score
** Keep track of the number of topics assigned to each user and the number of users assigned to each topic
** If the number of topics assigned to one user reaches the maximum value, remove that user from consideration (remove all scores corresponding to that user from scorelist)
** If the number of users assigned to one topic reaches the maximum value, remove that topic from consideration (remove all scores correspoinding to that topic from scorelist)
** Continue the above process till the scorelist is empty
* Save the bidding into the database
* Inform users about the results of the bidding

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:32:11Z

Bsingh8: /* Methodology */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, proper naming conventions are used

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:31:57Z

Bsingh8: /* Design */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

==== Methodology ====
We will implement this new algorithm such that the code is DRY, propere naming conventions are used

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:30:57Z

Bsingh8: /* Test Plan */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

** Log in as a participants/student
** Go to Assignment
** Go to Other's works and check preferred conference paper topics to bid on reviewing
** Click submit and wait for bidding deadline
** After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:29:21Z

Bsingh8: /* Additional Changes */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:28:56Z

Bsingh8: /* Changing Existing Files */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.

=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:27:34Z

Bsingh8: /* Design */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = (N + 1 - r) * B * N
score[topic][user] = num/den + B
if(user has preferred no topic)
score[topic][user] = B
if(user has not preferred this topic)
num = B * N
score[topic][user] = B - num/den

Now find_den = ∑ [ N + 1 ] - ∑ [k] where the iterating variable k varies from 1 to t

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:21:33Z

Bsingh8: /* Design */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

====Proposed Algorithm for Part B====
Let N = Number of topics
B = Base score
t = Total number of topics an user prefers

For every topic i:
den = find_den
for every user j:
if(user has preferred this topic)
r = priority(topic)
num = find_num
score[topic][user] = num/den

Now den =

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:16:40Z

Bsingh8: /* Design */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

====Proposed Design====
Keeping the existing algorithm and the project requirements in mind, we decided to divide the code into 2 parts:
* Part A: Make teams for people not in a team (and if applicable, complete incomplete teams)
* Part B: Assign topics to teams

We note that Part A has been implemented on a web service independent of Expertiza. We also note that the algorithm explained above applies only to Part A, and that it is pretty sophisticated. Hence we choose to keep the existing code as is for this part. We will simply call the webservice, provide the list of people, their bidding preference and also the max_team_size.

Part B is implemented as one line. It is pretty simple. We also note that it simply cannot be used for conference paper review assignemnt. Hence we completely change this section and implement it using a new algorithm.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T18:09:37Z

Bsingh8:

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====The Combined Problem====
The combined problem statement is as follows: Given a list of people (students or reviewers) and also a list of N items to bid on (project topics or conference paper reviews), we require the following:
* To form teams of Maximum size M
* Based on preference lists submitted by people/teams having a minimum of 1 item and a maximum of L topics, to allot topics to teams such that each team gets P items and each item (to be bid on) is assigned to R teams.

==== Points of Discussion====

We note the differences:
* For Bidding for Assignment Topics, M is generally greater then 1, but P = 1 and R = 1. The item to be bid on is the project topics.
* For Bidding for Conference Paper Reviews, M = 1 but P and R are generally greater then 1. The item to be bid on is the conference paper reviews.

We note that a team of 1 person makes little sense, but we still implement it to be so that the code is compatible for both the applications.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:58:19Z

Bsingh8: /* Design */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

== Design ==
In the following subsections, we discuss the problem, proposed design and code. We are trying to combine the code so that it can be used for both the project topic bidding as well as the conference paper review bidding.

====Understanding the Combined Problem====
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:54:37Z

Bsingh8: /* Use Case Diagram */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario
The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:53:41Z

Bsingh8:

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:53:00Z

Bsingh8: /* Implementing the Project */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Existing Algorithm ==

==== Problem ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

==== Functionality ====
The topics are assigned to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

* Assigning topics to teams: This is implemented as a one line, comparing bid preferences to allot topics.

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:11:44Z

Bsingh8: /* Implementing the Project */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Implementing the Project ==

==== Existing Algorithm ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

assigns the topics to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** The basic cost function used in the algorithm is: C(i,j) = 1/N * ∑ [ M -S(i) ] where i varies from 1 to N
** The Algorithm calculates the weights for every user not a team or part of an incomplete team. It then assigns teams using the weights, by plotting graphs for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M students per team;
** The Algorithm is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:09:13Z

Bsingh8: /* Implementing the Project */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Implementing the Project ==

==== Existing Algorithm ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

assigns the topics to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].
** A short summary is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>
**In the algorithm above, C(i,j) = 1/N * ∑ [ M1 -S(i) ] where i varies from 1 to N

**Using the weights obtained above, graphs are plotted for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:08:47Z

Bsingh8: /* Implementing the Project */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Implementing the Project ==

==== Existing Algorithm ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

assigns the topics to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here].

**A short summary is as follows:
<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)
</nowiki>
**In the algorithm above, C(i,j) = 1/N * ∑ [ M1 -S(i) ] where i varies from 1 to N

**Using the weights obtained above, graphs are plotted for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.

CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

2017-12-01T17:07:51Z

Bsingh8: /* Implementing the Project */

This page describes the the project E17A7 which is one of the several projects to allow Expertiza support (the currently unsupported) conference. Specifcally, it involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Navin Venugopal (nvenugo2@ncsu.edu)

Sathwick Goparapu (sgopara@ncsu.edu)

==Introduction==

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

====Background of the project====
In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic for students in a course. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:
* Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
* Each team is assigned only one topic
* Each topic (if assigned) is assigned to a maximum of one team

====Description of project====
This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:
* Before the bidding close deadline, reviewers submit a list of papers they wish to review.
* After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
** Each paper (if assigned) is assigned to a maximum of R reviewers (here R represents some constant)
** Each reviewer is assigned a maximum of P papers to review (here P represents some constant)
** Assignment of papers is individual

==Project Requirements==

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

====Problem Statement====
* To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review.
* A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
* The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use.

In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially. Also the third requirement was modified to first make the bidding for a conference using any algorithm and if time permits, to use a better algorithm like top trading cycles.

====Current Project Aims====
* To develop code such that both applications (bidding for project teams and bidding for conference paper reviews) use the same code
* To improve the algorithm for calculating the score for a particular project topic/conference paper review assigned to a project team/conference reviewer

We want to state here that we are not responsible for developing any conference features. Specifically, this means that we are not changing any UI features. We will be primarily relying on UI changes done by E17A5

== Implementing the Project ==

==== Existing Algorithm ====
The existing algorithm solves the following problem:

To form teams of students of Maximum size M and then assign topics (there are N topics) based on preference lists submitted either by individual students or teams (complete or otherwise). Preference Lists can have minimum 1 topic and a maximum of L topics.
Then the topics are alloted to teams such that each team gets an unique topic and each topic is assigned only to one team.

assigns the topics to project teams in 2 steps:

* Making teams: This is done using a k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. You can read about it in detail [https://docs.google.com/document/d/1hyJjqfdSPi0NjzLFygUodVLVqABIE2Lu-RBDMkUDvUs/edit?usp=sharing| here]. **A short summary is as follows:

<nowiki>
For every topic j :
For every position i that topic j can be placed in (any) preference list:
Let S(i) = number of students who selected topic j at position i of their (individual) preference list
B(i) = 1/S(i)
Calculate C(i,j)
Weight(i,j) = B(i) / C(i,j)

</nowiki>

**In the algorithm above, C(i,j) = 1/N * ∑ [ M1 -S(i) ] where i varies from 1 to N

**Using the weights obtained above, graphs are plotted for each topic. It is as follows:
*** Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights.
*** Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;

===Understanding the Requirements===
* Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
** Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
**This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
* The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

===Subsequent Discussion===
Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:
* Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
* The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
* In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

==Use Case Diagram==

Actors:
* Conference Reviewer: Submits a preference list of papers and reviews assigned papers
* Conference Administrator: Responsible for the assignment of topics to the reviewers

Scenario

The case of reviewers submitting a list of preferred topics and the administrator running the assignment process. For our project, the main modification would be concentrating on Use Case 4 and 5.

[[File:UML_01.png]]

====Choose and submit preference to be reviewed====
* Use Case Id: 1
* Use Case Description: Participants choose the preference for conference review topics and submit it
* Actors: Participants
* Pre Conditions: Conference papers are presented and submitted, and the participants are eligible for reviewing
* Post Conditions: Conference committee members can view participants preference and run bidding algorithm on it

====Saving bidder information into database====
* Use Case Id: 2
* Use Case Description: bidding preferences and related participants information are processed and saved to database
* Actors: None
* Triggered by: Use Case 1
* Pre Conditions: participants preferences are submitted
* Post Conditions: information can be retrieved and used by bidding algorithm

====View list of topics====
* Use Case Id: 3
* Use Case Description: participants can view list of topic available for conference paper topic
* Actors: Participants

====Run bidding process====
* Use Case Id: 4
* Use Case Description: Committee members can run bidding algorithm on application to help assigning the conference paper topics to participants
* Actors: Conference Committee
* Pre Conditions: preferences must be submitted by participants
* Post Conditions: the bidding result can be used for paper assignment

====Choosing and assigning paper to bidders====
* Use Case Id: 5
* Use Case Description: System assigns participants to conference paper topics according to bidding result
* Actors: None
* Triggered by: Use Case 4
* Pre Conditions: bidding algorithm has run and result has been returned
* Post Conditions: Participants can view topics been assigned to them

====Change assignment of review using CRUD actions====
* Use Case Id: 6
* Use Case Description: Conference committee members can change assignment result manually
* Actors: Conference Committee
* Pre Conditions: topic assignment has been done
* Post Conditions: changes on bidding result is visible to participants and other committee members

== Data Flow Diagram ==
Below is the Data Flow Diagram for process flows of the project. The diagram shows the process of bidding algorithm that we proposed to use for conference paper review assignment.
[[File:Webp.net-resizeimage (1).png]]

= Design =
The changes in the first sub section below were prior to the mentor meeting, where we had a major change in our problem statement.
Earlier, we assumed that we only had to change the existing code, which is reflected above.

However, with the new requirements, in addition we need to add completely new code.

== Changing Existing Files ==
This sub section details what is to be changed in the existing code.
=== Files to be changed ===
*lottery_controller.rb

=== Current Code ===
This is current code for bidding assignment. The code will basically get target assignment and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximum team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
redirect_to controller: 'tree_display', action: 'list'
end

=== Proposed Changes ===

To accommodate this code to fit our needs, we have proposed to add 3 new variable

bidding_for_reviews?
topic_per_team
team_per_topic

new_intelligent_property will check if an assignment is using conference paper bidding or team assignment bidding, it will be true if bidding is on conference paper review and false if bidding is on team assignment; topic_per_team will record required number of topics that can be chosen per a team; and team_per_topic will record required number of teams that can be assigned to a topic.

=== Proposed Code ===

Then we will change the run_intelligent_assignment function to implementation as below:

def run_intelligent_assignment
priority_info = []
assignment = Assignment.find_by(id: params[:id])
topics = assignment.sign_up_topics
teams = assignment.teams
teams.each do |team|
# grab student id and list of bids
bids = []
topics.each do |topic|
bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
end
if new_intelligent_property
team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
else
team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
end
end
if !new_intelligent_property
data = { users: priority_info, max_team_size: assignment.max_team_size }
url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
begin
response = RestClient.post url, data.to_json, content_type: :json, accept: :json
# store each summary in a hashmap and use the question as the key
teams = JSON.parse(response)["teams"]
create_new_teams_for_bidding_response(teams, assignment)
run_intelligent_bid(assignment)
rescue => err
flash[:error] = err.message
end
else
run_conference_bid(teams, assignment, topic_per_team, team_per_topic)
end
redirect_to controller: 'tree_display', action: 'list'
end

== Additional Changes ==

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

def run_conference_bid teams, assignment, topic_per_team, team_per_topic
set topic_set to false;
for each team in teams
append the team in team_list
retrieve bidding ranks of each team
sum all rank values in list
for each topic priority/rank value in ranks list
if topic_set is false
append topic_id to topic_list
end if
calculate percentage of the rank value in comparison to sum of rank values
if sum is 0
change percentage to 1/length_of_rank_list
end if
score rank value by doing (percent_rank_values *(number_of_topics^2) + (number_of_topics^2) / length_of_rank_list)
append [team_id, score] to team_scores dictionary with team_id as key
end for
set topic_set to true
end for
sort team_scores dictionary by values/scores in descending order
for score in team_scores
if team not in team_list or topic not in topic_list
remove score
continue to next iteration
end if
append topic_id to topic_assignment list
if number of occurrences of current topic in topic_assignment is same as topic_per_team
remove this topic_id from topic_list
end if
append team_id to team_assignment list
if number of occurrences of current team in team_assignment is same as team_per_topic
remove this team_id from team_list
end if
append topic_id and team_id pair to team_bids
end for
save sign up information stored in team_bids to database
end def

=Test Plan=

===Manual Testing===
* UI testing of the implemented functionality to be done.

1. Log in as a participants/student

2. Go to Assignment

3. Go to Other's works and check preferred conference paper topics to bid on reviewing

4. Click submit and wait for bidding deadline

5. After deadline reaches, go back to assignment and click on Other's work to view topics assigned for reviewing

===Automated Test Cases===
* TDD and Feature Test cases to be written. For instance we would do Rspec test on cases below:
** Test on score calculation, see if the calculation gives right scoring distribution when no preference is given, when one preference is given, and when multiple preferences are given;
** Test on preference levels matching, check if the algorithm is match first, second, third, and so on preferences and evaluating them correctly;
** Test if teams per topic or topics per team can exceed required number by assigning many team with same first preference and assigning many team with multiple preferences (preference number per team > number of topics/2);
* For the Rspec tests, we will have to create object of team, assignment, and ranks. A sample team object in JSON format would be:
{"users":[{"ranks":[1,0,2,3], "pid":1023}],"max_team_size":1}
* The objects would be generated inside the spec file lottery_controller_spec.rb using model object initializer;

===Edge cases===

* Case 1: No reviewer submits a list of preferred topics to review

* Case 2: All reviewers submits exactly the same list of topics to review.