CSC/ECE 517 Spring 2017/E1731 Improve Score Calculation

Introduction

Expertiza is an opensource web based platform developed to assist students in educational institutions to perform peer reviews and group projects. The reviewing is done through a mechanism where students will be sent request to review work and the system will analyze and publish a summarized report. The software is designed in MVC (Model-View-Controller) architecture using Ruby on Rails.

Problem Description

Current Scenario

Today, Expertiza stores the scores based on each response to each criterion, but no holistic scores are stored. The "answers" table only contains the scores that user A has given user B on each criterion. This means that if we need to know what score A has given B based on 100, we have to rely on the code to calculate it every time the score is requested. This design slows down Expertiza as every time an instructor clicks on "View Scores", all the calculations are done to display the score. The situation is even worse if we want to weight the scores by the reputation of each reviewer, which is a measure of how closely that reviewer's scores match other reviewers. In this case, the system potentially has to do thousands of arithmetic calculations to display the score for a particular assignment.

Proposed Solution

We propose that we have two mechanisms to handle the holistic scores, depending on the current state of the assignment: This new approach will take the responses and the database will act as a storage environment.

• OnTheFlyCalc:
  • Calculates holistic scores over a set of data: A set of records in answers table, namely the responses from user A to user B on each criterion.
  • When an assignment is still ongoing, the reputations for each reviewer will be calculated and handled by OnTheFlyCalc
  • This is similar to the current scenario
• LocalDBCalc:
  • Calculates the holistic score over a single db query: "get the score that user A gives user B on assignment 999 on round 2"
  • When an asignment has finished, all the reputations will be calculated by LocalDBCalc and stored, since they are finalized. So, when the scores are requested, they will be fetched using a single db query and displayed
  • This will be a new addition to the code as currently there is no class.

In this project, our goal is to make the OnTheFlyCalc and LocalDBCalc work only for peer-review scores. As future projects will introduce other types of scores, the solution must be extensible.

Design Plan

• A new database table "local_db_scores" with following columns will be created to store the holistic scores:
  • id: int
  • score_type: string
  • round: int
  • score: int
  • response_map_id: int (this will be reference to the response_map table)
• A new column "local_scores_calculated" with boolean type will be added to "assignments" table
  • This has the default value false
  • The value will change to true when the holistic scores for that assignment are stored in local_db_scores table
• A new icon will be added to the list of Assignments on the Manage Assignments page for instructor
  • When this icon is clicked, the holistic scores will be calculated using store_total_score method in LocalDBCalc class and then inserted in this new table "local_db_score"
    • If the record to be inserted is a finalized peer-review score, the type stored in the database will be "ReviewLocalDBScore" and the reference_id will be the response_map id
    • The peer-review scores will be calculated and stored for each of the review rounds because it is possible that user A only reviews user B in one out of several rounds
• When a request is made to view the total scores, depending on whether local_scores_calculated column of the assignment is false or true, either OnTheFlyCalc or LocalDBCalc will be called to calculate the total score

To achieve this functionality, following classes will be created:

• OnTheFlyCalc:
  • This class will contain a method "compute_total_scores" which will compute the total score for an assignment by summing the scores given on all questionnaires and return this total score when an instructor tries to view scores for an ongoing assignment
  • It will also contain methods to calculate the average score and score range (min, max) for each reviewee(team) for peer-review
  • Currently OnTheFlyCalc already exists as a module. As it is only used by assignments, we plan to change it into a class and change all its methods into static methods
• LocalDBCalc:
  • This class will also contain a method "compute_total_scores". But this method will compute the total score by querying and summing the scores saved in local_db_scores table instead of summing the scores given on all questionnaires
  • It will also contain a method "store_total_scores" which will be called when an instructor clicks on "Save Scores to db" icon for an assignment
    • This method will compute and save the total scores for all the response maps (reviewer -> reviewee) in the assignment for each round in local_db_scores table.

As seen in the above image, when a user A reviews another user B (or team), a response map is created containing individual scores and responses to questionnaires. When an assignment is finished, LocalDBCalc class calculates the holistic scores from the individual scores and saves it in local_db_scores table, so that next time someone tries to view the scores, they can be immediately queried and returned without any calculations.

The above image shows the mechanism which will be followed. When an instructor tries to view scores for an assignment, 1st it will be checked if the assignment is ongoing or completed. If it is an ongoing assignment, the holistic scores are calculated and returned. However, if it is a finished assignment, no calculations are done. The holistic scores are directly returned from the local_db_scores table where they are stored.

Changes

Code snippets including:

Icon added for storing into db:

Storing into db:

Use Cases

The project can be broken down into two primary use cases.

• The user wants to view the scores before the assignment is finished. Scores are calculated "on the fly" and presented to the user.
• The user wants to view the scores after the assignment is finished. Scores are calculated on assignment completion and stored in the database for quick retrieval.

Test Plan

The above two use cases can then be translated into three major scenarios we need to test. The use case where the assignment is completed and scores are stored into the database is broken down into two scenarios to test the data input and data retrieval parts separately.

The following table contains the main scenarios we will cover when testing our project.

Scenario Number	Description	Expected Result for Pass
1	An assignment is finished and a score is calculated correctly and placed into the local_db_score table using the scheduled task feature. First, set up an assignment with some score information. Create a scheduled task to calculate the score and insert the score into the database upon assignment completion.	Query the database and verify score is correct
2	Assignment is NOT finished and user wants to view scores. First, set up an assignment with a deadline in the future. Put in some score information. The score should be calculated correctly on the fly using OnTheFlyCalc.	After the assignment is created, this test will run on the UI. The user will click the view scores link and the page should display the expected score.
3	Assignment is finished and the user wants to view scores. This test will mainly just be a check if clicking view scores on the UI will result in the expected value showing up on the UI. The interaction in getting the information in the database is already tested in Scenario 1.	User clicks on view scores on a past assignment and sees the expected score on the UI

The following diagrams show how the chosen scenarios above map to functionality.