CSC/ECE 517 Fall 2017/E17A7 Allow Reviewers to bid on what to review

This project describes the work done for the project E17A7, which involves adding the ability of conference paper reviewers to bid for what they want to review. The members of this project are:

Leiyang Guo (lguo7@ncsu.edu)

Bikram Singh (bsingh8@ncsu.edu)

Introduction

Expertiza is an open source project created using Ruby on Rails. This project is an software primarily to create reusable learning objects through peer review and also supports team projects. Expertiza allows the creation of instructors and student accounts. This allows the instructors to post projects (student learning objects) which can be viewed and worked upon by students. These can also be peer reviewed by students later.

Background of the project

As explained above, Expertiza is currently able to support a university course, hosting students and instructors and able to assign and grade projects by the appropriate users.

Expertiza is currently not able to support a conference. The difference between a conference and an university course are several. However the focus of our project is the bidding ability.

In the existing Expertiza functionality, the bidding ability is only for bidding for a project topic. This involves the instructor posting a list of project topics, and each student (or all students together as a team, if some students have already formed a team) then posts a preference list, listing topics (s)he wants to work on. Then the bidding algorithm assigns the topics, particularly with the following features:

• Students (not in at team) with close preferences are assigned into a project team, and the common preference is the project topic assigned to the team. Alternatively, for existing teams the project topic is assigned as per the common project topic preference list
• Each team is assigned only one topic
• Each topic (if assigned) is assigned to a maximum of one team

Description of project

As stated above, Expertiza currently is not able to support a conference. This project is not responsible for adding code so as to support a conference. Rather, we are interested in the bidding algorithm used in the conference, which is significantly different from the project bidding algorithm as explained above.

For the purposes of the project, we assume that there are several reviewers in a conference who review papers which are proposed to be presented in the conference. Also, the entire list of papers proposed to be presented in the conference is also available.

Then the basic working of the project assumes:

• Before the bidding close deadline, reviewers submit a list of papers they wish to review.
• After the bidding deadline, the algorithm assigns papers to reviewers to review, such that:
  • Each paper (if assigned) is assigned to a maximum of R reviewers
  • Each reviewer is assigned a maximum of P papers to review
  • Assignment of papers is individual, that is no paper is to be reviewed by a "team" of reviewers

Project Requirements

In this section we discuss the problem statement, then discuss the existing code and possible changes required.

Problem Statement

• To take the existing bidding code in Expertiza (which is meant for students bidding for project topics) and make it callable either for bidding for topics or bidding for submissions to review. In the subsequent discussion with the mentor, it was concluded that the two bidding situations are very different hence it was decided to keep the two separate, at least initially.
• A possible extension is to add other methods, like a "late reviewer registration" method, in which each new reviewer is assigned one of the least-reviewed submissions so far. Methods like these require an extension of the basic working of the project.
• The matching algorithm is currently not very sophisticated. Top trading cycles is implemented in the web service (though it is currently not used by bidding assignment), and could be adapted to this use. However, in the subsequent discussion, as noted above, because the two bidding situations were found different the requirement is to first make the bidding for a conference (using any algorithm) and if time permits, to use a better algorithm like top trading cycles.

Understanding the Requirements

• Objects (being bid on) are often assigned by having the bidders providing a preference list of bids.
  • Expertiza provides a way for students to bid on different assignment topics. This allows the fair topic assignment distribution among students, particularly when multiple students or groups (of students) are bidding on same topic or paper.
  • This is very similar to the kind of bidding that a conference implements for reviewers (who bid for proposed papers)
• The other requirements are extending the above idea to get new features or an improved algorithm. We discuss these in more detail below.

Existing Algorithm

A course consists of following users: Students, Teacher

Course Project: Students must form teams of a Maximum size M1 and complete the Course Project(s) during the course, guided by the teacher

Selecting topics:

• Teacher puts out a list of available project topics (let number of topics be N)
• Making a Preference List with minimum 1 topic, maximum M2 topics:
  • Individual Students (who have not formed teams) make a list of topics they (individually) prefer.
  • Or Every paired teams makes a (combined common) list.
• Bidding Priority Assignment is as follows:

for each topic j':

  for each position i that topic j has been placed on (in any preference list
      

largest ← L[0]

  for each item in L, do
    if item > largest, then
      largest ← item
  return largest
   

For every topic j :

      For every position i that topic j can be placed in (any) preference list:
             Let S(i) = number of students who selected topic j at position i of their (individual) preference list
             B(i) = 1/S(i)
             C(i,j) = 1/N * :<math>\sum_{i \mathop =1}^{N}(M1 -S(i) ).</math>           
             *i=1N
             Weight(i,j) = B(i) / C(i,j)

• Graph drawing is as follows: Draw a graph for every topic j with X axis as the priority position i and the Y axis as the Weights
• Selecting the topic for the team: A topic can at maximum be assigned only to one team. Some topics may be unassigned
  • Use Hierarchical K means to select teams such that all students in a team are close to each other in the graph above, hopefully more towards the left of the graph, and also such that there are a minimum of 1 and a maximum of M1 students per team;
  • In case of several students (greater then M1 in number) are together in one cluster, select those students into a team which have collaborated before.
  • In each graph, only one team will be selected as per the above parameters

Subsequent Discussion

Because the two situations are similar, it was hoped that the conference bidding would call the code of the (existing) assignment bidding. It was also hoped that the conference bidding would be modified to make it more general.

However, in a subsequent meeting with the mentor, it was found that the two bidding algorithms are incompatible. The existing algorithm will not work for conference bidding are:

• Team based versus Individual based: The existing algoritm is designed to assign topics to teams, whereas we have to assign topics to individuals
• The current bidding implementation only returns one bidding result (per graph) each time, whereas expected bidding implementation must be able to return multiple bidding results every time.
• In the case of a priority tie, the current bidding algorithm assigns topic to bidding team that has most members whom have worked together in the past. This will not work for a conference bidding since now there is no collaboration in conference paper review.

In addition since the existing algorithm is partly running on a web-service we cannot directly modify, it will be difficult to modify the code for the conference requirements and later integrate the two.

The changes related to this project are required for reviewing conference papers and journals. The users are able to submit their works as either individuals or teams to be scrutinized. The other participants are allowed to review the papers of their interest from the list of submitted papers. The participants are allowed to bid on the papers of their preference in a preference list with the top most entry taking the highest priority. The bidding algorithm already exists as called by a web service in Peerlogic. The goal of this project is to generalize the algorithm for both review and topic based bidding.

===Test The existing bidding algorithm used by the application is only available for team assignment bidding. The algorithm itself groups all team members preferring specific topic together, then evaluates how frequently these people have worked together in the past through concept of top-trading cycle and obtains the k-mean distribution of resulted evaluation for assigning topics.

Bidding Algorithm Overview

Currently, the Bidding algorithm implementation is based on k-means clustering and a weighting formula that favors increasing overall student satisfaction and adding members until the maximum allowable team size is reached. The approach to meeting these criteria is addressed by mining student preferences for topics with a clustering approach, and then matching them in groups to topics that suit their shared interests. You can read about it in detail here.

Our goal here is as follows:

• To implement a bidding algorithm for conference paper reviewer assignment functionality.
• To attempt to merge the existing and the new algorithm as much as possible

Possible Extension

A possible extension is to combine reviewer assignment by bidding with one of the other assignment methods--instructor assignment, or “automated” assignment, where each new reviewer is assigned one of the least-reviewed submissions so far. In this case, submissions that are not bid upon would be assigned to reviewers who “show up later,” as would reviews that are not completed by the reviewers who bid on them.

Use Case Diagram

Files to be Modified

• lottery_controller.rb

This is current code for bidding assignment. The code will basically get target assignment (conference paper assignment in our case) and then get all topics and teams for the assignment. Then it will assign priority of each team-topic pair according to user preference for each user, and append it to priority information list. Afterward it sends priority information along with maximem team size information to web-server and rearranging the priority order according to top-trading cycle algorithm, and pass returned result to create_new_teams_for_bidding_response function for bidding handling on topics with preferences and run_intelligent_bid function for bidding handling for leftover assignment as well as cleaning up.

 def run_intelligent_assignment
   priority_info = []
   assignment = Assignment.find_by(id: params[:id])
   topics = assignment.sign_up_topics
   teams = assignment.teams
   teams.each do |team|
     # grab student id and list of bids
     bids = []
     topics.each do |topic|
       bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
       bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
     end
     team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
   end
   data = { users: priority_info, max_team_size: assignment.max_team_size }
   url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
   begin
     response = RestClient.post url, data.to_json, content_type: :json, accept: :json
     # store each summary in a hashmap and use the question as the key
     teams = JSON.parse(response)["teams"]
     create_new_teams_for_bidding_response(teams, assignment)
     run_intelligent_bid(assignment)
   rescue => err
     flash[:error] = err.message
   end
   redirect_to controller: 'tree_display', action: 'list'
 end

To accommodate this code to fit our needs, we have proposed to add a new variable

 anti_intelligent_property

which will be true if bidding is on conference paper review and false if bidding is on team assignment. Then we will change the run_intelligent_assignment function to implementation as below:

 def run_intelligent_assignment
   priority_info = []
   assignment = Assignment.find_by(id: params[:id])
   topics = assignment.sign_up_topics
   teams = assignment.teams
   teams.each do |team|
     # grab student id and list of bids
     bids = []
     topics.each do |topic|
       bid_record = Bid.find_by(team_id: team.id, topic_id: topic.id)
       bids << (bid_record.nil? ? 0 : bid_record.priority ||= 0)
     end
     if anti_intelligent_property
       team.users.each { |user| priority_info << { pid: user.id, ranks: bids }
     else
       team.users.each { |user| priority_info << { pid: user.id, ranks: bids } if bids.uniq != [0] }
     end
   end
   if !anti_intelligent_property
     data = { users: priority_info, max_team_size: assignment.max_team_size }
     url = WEBSERVICE_CONFIG["topic_bidding_webservice_url"]
     begin
       response = RestClient.post url, data.to_json, content_type: :json, accept: :json
       # store each summary in a hashmap and use the question as the key
       teams = JSON.parse(response)["teams"]
       create_new_teams_for_bidding_response(teams, assignment)
       run_intelligent_bid(assignment)
     rescue => err
       flash[:error] = err.message
     end
   else
     run_conference_bid(teams, assignment)
   end
   redirect_to controller: 'tree_display', action: 'list'
 end

With this change, we will have to add a new function: run_conference_bid(teams, assignment), with pseudo-code as shown below:

 def run_conference_bid teams, assignment function
   
 end def

Test Plan

Manual Testing

• UI testing of the implemented functionality to be done.

Automated Test Cases

• TDD and Feature Test cases to be written.

Edge cases

• Case 1: Users signing up after bidding is done need to be assigned the papers/journals with no or least reviews done.

• Case 2: Extending edge case 1, if somebody doesn't provide a list of topics s(he) wants to review, the topic with the least number of bids is assigned.

• Case 3: If a participant is assigned to review a paper in the (topics) preference list but decides to drop the review after the bidding process is done.