CSC/ECE 517 Fall 2019 - E1986. Allow reviewers to bid on what to review

From Expertiza_Wiki
Jump to navigation Jump to search

Why Should a Student bid for Reviewing Submissions?

If the choice of which submission to review is left to student itself, he/she would choose to review those topics which are related to their project or whatever they are highly interested in. This would benefit both the reviewer and reviewee because

From a Student's Perspective:

  • Because he/she does not have to spend a lot of time understanding what is going on.
  • The Student would be more focused on giving the review and hence judgment made tend to be more objective than subjective.
  • This would improve the quality of reviews and will also be helpful for the reviewer.
  • By implementing the color-coding feature, the student at any point knows which topic/s are more likely to get and he/she optimizes between "what they want" and "what they can get"
  • Also, Bidding in Fun !!!!!!!


From the Instructor's Standpoint:

  • Because opinions expressed in the reviews are a result of informed decisions, the inference made from the Machine Learning models trained on these data sets would make more sense.

Problem Statement

Currently Deployed Implementation

When an assignment participant wants to review others’ work, they will be asked to choose an assignment topic. The participant will be allowed to choose from among those assignment topics that have not been already assigned to 10 other participants. If they choose the ‘I do not care about the topic I review’ option, a random topic will be chosen from all the available topics and will be assigned to them. That is, the reviews are being allocated on a First Come First Serve basis.

Previous Work on this issue

E1856

E1928

What’s Deeply wrong in those implementations?

To understand what's wrong, first consider the differences between Teams Bidding for Assignments and Students Bidding for Reviews. Both of these are Matching or resource Allocation Problems.

The former can be modelled as a one-to-one matching problem. (i.e team and assignment has one-to-one correspondence).

However, Students and Reviews have many-to-many relationship (a student can choose multiple submissions for review and a submission can be given to multiple students for review)

The Mathematical formulation is itself wrong in E1856 and E1928 and they have used the below shown diagram to represent the relationship.

Why does the difference is the representation matter?

Since they have modeled the problem on the same lines, they have used the same version of the Gale-Shapley or Top Trading Cycles Algorithm used for one-to-many or one-to-one approach. Famous Problems Dealt on these lines.

Stable Marriage Problem (One-to-One)

School Choice Problem (One-to-Many)

Other Implementation pitfalls observed

E1928

  • Once the bidding for review topics is done, the selections need to be saved to the database which is not happening and when the page is refreshed, the UI does not retain the bids.
  • The button responsible for running the algorithm cannot be checked and it is a hunk-like icon.
  • The button appears multiple times on the page.
  • The algorithm needs to be implemented in the web service which should be ideally be used from the lottery controller which is not they have tried to implement. The entire code is written in Ruby.
  • The algorithm needs to skip the review topic that the user has worked on. This Code already exists for topic and again written in the new implementation that needs to be integrated into the existing one and hence needs refactoring.
  • Proper tests need to be written.

Approach we have adopted for matching students with topics

Taken in reference from Many-to-Many-Matching-Problem-with-Quotas-by-Freer-and-Titova (Basically a Gale-Shapely version for many-to-many matching situations)

We have a set of students S = {S1, ..., Sn}, and a set of topics T = {T1, ..., Tm}. Each student must get exactly qS topics (The threshold in our case is qS = 4, it’s up to the student to decide how many of these they want to actually review.), while each topic must be assigned to at least qT reviewers. Students will submit a list of preferences over the set of topics in linear order. If they submit the preferences for only a certain topics, the rest of the topics will be appended to their preference list in a random order. The topics shall have (possibly different) linear order preference over the set of students according to the timestamps of their time of bid and the total number of topics they have bid for. Students with lower timestamps and who bid for lower number of topics will be given higher preference.

To begin, we will try to obtain a matching with an approximately equal number of students reviewing each topic, which we call uniform distribution. If no uniform stable matching exists, our mechanism will nonetheless return a stable matching, provided one exists for a given profile of preferences.


The average number of reviewers assigned to each course will be

where n is the number of students, and m is the number of courses.

Clearly, this number is not necessarily an integer. Let us assume that

where


Matching 𝝁 is a mapping which assigns exactly qS different topics to each student in Sand at least qT different students to each topic in T. We denote the set of topics assigned by matching 𝜇 to the student si and the set of students who were assigned the topic tj as 𝜇(si) and 𝜇(tj),respectively.

Blocking Pair (Defined according to our problem): A pair (s, t) is called a blocking pair, if the student is associated with the topic (If he/she is in the group which works on topic t). A Matching Set M is considered Stable if it does not have any blocking pairs.

The matching is (pairwise) stable if there are no blocking pairs.

Mechanism:

First we calculate k, p|, and p|. The goal is for each topic to be assigned to either p| and p| students, and to assign exactly qS courses.

The algorithm:

  • Step 1:

Each topic proposes to accept the first pI students in its preference list. Each student accepts no more than qS proposals according to his/her preferences, rejecting the rest.

  • Step k:

Each course that has z < pI students proposes to accept pI - z students it has not yet proposed to. Each student accepts no more than qS proposals according to his/her preferences, rejecting the others.

The algorithm stops when every topic that has not reached the maximum quota pI has proposed acceptance to every student.

High Level Overview of the functionality implemented

Implementation Details

Changes Made in Expertiza Source Code

Modified Files

  • routes.rb
  • app/views/student_task/view.html.erb
  • app/views/assignments/edit.html.erb
  • app/controllers/student_task_controller

New Files

  • app/controllers/review_bidding.rb
  • app/models/review_bidding_controller.rb
  • app/views/sign_up_sheet/review_bid.html.erb
  • Source Code for the Web Service [1]

Facilitating bidding functionality for the participants

  • A new link is added in the student_task/view.html.erb file, so that the user can be redirected to the review bidding view to bid for a review topic.
  • The UI looks similar to topic bidding but the catch here is that any participant should not see his own topic either inside the selection or topics table, which is handled in the review_bidding_controller methods of set_priority and review_bid.
  def review_bid
    @participant = AssignmentParticipant.find(params[:id].to_i)
    @assignment = @participant.assignment
    @sign_up_topics = SignUpTopic.where(assignment_id: @assignment.id, private_to: nil)
    team_id = @participant.team.try(:id)
    my_topic = SignedUpTeam.where(team_id: team_id).pluck(:topic_id).first
    @sign_up_topics -= SignUpTopic.where(assignment_id: @assignment.id, id: my_topic)
    @max_team_size = @assignment.max_team_size
    @selected_topics =nil
    @bids = team_id.nil? ? [] : ReviewBid.where(participant_id:@participant,assignment_id:@assignment.id).order(:priority)
    signed_up_topics = []
      @bids.each do |bid|
        sign_up_topic = SignUpTopic.find_by(id: bid.sign_up_topic_id)
        signed_up_topics << sign_up_topic if sign_up_topic
      end
      signed_up_topics &= @sign_up_topics
      @sign_up_topics -= signed_up_topics
      @bids = signed_up_topics
      @num_of_topics = @sign_up_topics.size
      render 'sign_up_sheet/review_bid'
  end

  def set_priority
    if params[:topic].nil?
      ReviewBid.where(participant_id: params[:participant_id]).destroy_all
    else
      participant = AssignmentParticipant.find_by(id: params[:participant_id])
      assignment_id = SignUpTopic.find(params[:topic].first).assignment.id
      team_id = participant.team.try(:id)
      @bids = ReviewBid.where(participant_id: params[:participant_id])
      signed_up_topics = ReviewBid.where(participant_id: params[:participant_id]).map(&:sign_up_topic_id)
      signed_up_topics -= params[:topic].map(&:to_i)
      signed_up_topics.each do |topic|
        ReviewBid.where(sign_up_topic_id: topic, participant_id: params[:participant_id]).destroy_all
      end
      params[:topic].each_with_index do |topic_id, index|
        bid_existence = ReviewBid.where(sign_up_topic_id: topic_id, participant_id: params[:participant_id])
        if bid_existence.empty?
          ReviewBid.create(priority: index + 1,sign_up_topic_id: topic_id, participant_id: params[:participant_id],assignment_id: assignment_id)
        else
          ReviewBid.where(sign_up_topic_id: topic_id, participant_id: params[:participant_id]).update_all(priority: index + 1)
        end
      end
    end
  end
  • We are assigning color based on number of people who have chosen for that particular topic, this is handled in Review_bidding_controller.get_quartiles(topic_id) method
  • In the equation for the average number of reviews assigned to each course, we can compute n and m which are essentially #number of participants and #num of topics, the value is qs is taken from @num_reviews_allowed parameter which is associated with every assignment. if the number of students who have chosen that topic is less than k/2 then the color is shown to be green if it's greater than k/2 and less than k, then the color is yellow if the value is greater than red, then red is shown.
  def get_quartiles(topic_id)
    assignment_id = SignUpTopic.where(id: topic_id).pluck(:assignment_id).first
    num_reviews_allowed = Assignment.where(id: assignment_id).pluck(:num_reviews_allowed).first
    num_participants_in_assignment = AssignmentParticipant.where(parent_id: assignment_id).length
    num_topics_in_assignment = SignUpTopic.where(assignment_id: assignment_id).length
    num_choosers_this_topic = ReviewBid.where(sign_up_topic_id: topic_id).length
    avg_reviews_per_topic = (num_participants_in_assignment*num_reviews_allowed)/num_topics_in_assignment

    if num_choosers_this_topic < avg_reviews_per_topic/2
      return 'rgb(124,252,0)'
    elsif num_choosers_this_topic > avg_reviews_per_topic/2 and num_choosers_this_topic < avg_reviews_per_topic
      return 'rgb(255,255,0)'
    else
      return 'rgb(255,99,71)'
    end
  end
  helper_method :get_quartiles
  • These Bids are stored in new ReviewBid Model created.
  create_table "review_bids", force: :cascade do |t|
    t.integer  "priority",         limit: 4
    t.integer  "sign_up_topic_id", limit: 4
    t.integer  "participant_id",   limit: 4
    t.datetime "created_at"
    t.datetime "updated_at"
    t.integer  "user_id",          limit: 4
    t.integer  "assignment_id",    limit: 4
  end

  add_index "review_bids", ["assignment_id"], name: "fk_rails_549e23ae08", using: :btree
  add_index "review_bids", ["participant_id"], name: "fk_rails_ab93feeb35", using: :btree
  add_index "review_bids", ["sign_up_topic_id"], name: "fk_rails_e88fa4058f", using: :btree
  add_index "review_bids", ["user_id"], name: "fk_rails_6041e1cdb9", using: :btree

Invocation of Web Service

  • A new link was added in the app/views/assignments/edit.html.erb which redirects to the Review_Bidding_Controller.assign method.
  def assign
    assignment_id = params[:id]
    reviewers = AssignmentParticipant.where(parent_id: assignment_id).ids
    topics = SignUpTopic.where(assignment_id: assignment_id).ids
    bidding_data = assignment_bidding_data(assignment_id,reviewers)
    matched_topics = reviewer_topic_matching(bidding_data,topics)
    assign_matched_topics(assignment_id,reviewers,matched_topics)
  end
  • This method inturn serializes the data stored in ReviewBid Table and sends that to the service. The service responds back with the participant-review mappings.
  • It is guarenteed that every participant would have recieved a certain number of reviews based on the hyperparameter @num_reviews_allowed defined for every assignment in the Assignment
  • Inputs for the service is as follows:
Participant:
        Topic ID's
        Time Stamps indicating how early they have made the bids
        Priority order for the bids
        Participant's own topic
q_S: #Num reviews allowed for that assignment.
  • Output is a map of arrays where every user has a list of topic id's that are to be done.

Storing the mappings received from the service

  • The reviews are stored in the table associated with the ResponseMap Model. Reviewed_object_id is the assignment_id.
  • This is handled in the assign_matched_topics method.
def assign_matched_topics(assignment_id,reviewers,matched_topics)
    for reviewer in reviewers do
      reviewer_matched_topics = matched_topics[reviewer.to_s]
      for topic in reviewer_matched_topics do
        # puts(topic)
        # puts(topic.class)
        matched_reviewee = SignedUpTeam.where(topic_id: topic).pluck(:team_id).first
        if(matched_reviewee != nil)
          ReviewResponseMap.create({:reviewed_object_id => assignment_id, :reviewer_id => reviewer, :reviewee_id => matched_reviewee, :type => "ReviewResponseMap"})
        end
      end
    end
  end

Tests

A REST client was used to test the web service.

Three kinds of inputs were given:

1) The ideal case where most of the reviewers have different bids with different timestamps and where no reviewer has included their own project id in their preferences.

2) The case where all the reviewers select only one project and all of them select the same project, with the same timestamps.

3) None of the reviewers have participated in the bidding.

The web service does not produce any error in any of the above-mentioned cases.

Project Mentor

Ramya Vijayakumar (rvijaya4@ncsu.edu)

Team Members

Yaswanth Soodini (ysoodin@ncsu.edu)

Sai Shruthi Madhuri Kara (skara2@ncsu.edu)

Ayushi Rungta (arungta@ncsu.edu)

Vivek Reddy Karri (vkarri@ncsu.edu)

Deployment guide for the web service

0) We were not able to deploy the service on any web server, which could expose the app to the internet. Although, the application was perfectly running on the flask default server.

1) Our tries failed both for gunicorn and Apache Web Server. We are writing this guide just so that it'll be helpful for someone who is deploying this application.

2) Flask Default Server is not advisable to be used in the production environment.

3) Guide using gunicorn server: 

  (Install os-specific gunicorn package and gunicorn python package)
  
  Run the below-specified command in the root folder of the web service, where the main.py package is located. 
  
  gunicorn -b <Ip-addr>:8080 -w <num_of_workers> main:app (main is the python file, and app is the name of the flask app.

  Remarks Observed: The app which was perfectly running on the default flask server, running into weird issues sometimes while operating behind the gunicorn server.

4) Guide for using Apache Web Server 

  (Install os-specific apache2 & libapache2-mod-wsgi package)
  
  Two files are important in this process. i) app.wsgi and ii) app.conf (in the source code repo)
  
  app.conf is a configuration file for the Apache. Edit WSGIScriptAlias(pointing to the app.wsgi file) and Directory(flask app root folder) paths.
 
  Copy this file into the /etc/apache2/sites-available/ directory (This Path might differ depending on the OS)
  
  Now edit the path of the preferred interpreter on the first line in API.wsgi and then edit the paths and environment variables accordingly (It is suggested use a virtual environment)
  
  app.wsgi is the entry point into the flask app imported as application.

  a2dissite 000-default.conf (Remove the symbolic link of the default configuration file pointing into the sites-enabled folder)

  a2ensite app.conf (This will create a symbolic link of this file inside the sites-enabled folder (kind of staging area) which is originally in the sites-available folder)

  Error While running the service: 
  SyntaxError: Non-ASCII character '\\xc5' in file /home/ubuntu/expertiza-topic-bidding/venv/lib/python3.6/functools.py on line 8, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

 This error has nothing to do with the application and is raised in the line from flask import Flask and fyi we've added the #-*- coding: utf-8 -*- line on top of the file but with no use.

Important references

InteligentAssignment

Many-to-Many-Matching-Problem-with-Quotas-by-Freer-and-Titova

Project_E1928._Allow_reviewers_to_bid_on_what_to_review

E1856_Allow_reviewers_to_bid_on_what_to_review

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2019_-_E1986._Allow_reviewers_to_bid_on_what_to_review&oldid=131331"