CSC/ECE 517 Spring 2017/E1731 Improve Score Calculation

Introduction

Expertiza is an opensource web based platform developed to assist students in educational institutions to perform peer reviews and group projects. The reviewing is done through a mechanism where students will be sent request to review work and the system will analyze and publish a summarized report. The software is designed in MVC (Model-View-Controller) architecture using Ruby on Rails.

Problem Description

Current Scenario

Today, Expertiza stores the scores based on each response to each criterion, but no holistic scores are stored. The "answers" table only contains the scores that user A has given user B on each criterion. This means that if we need to know what score A has given B based on 100, we have to rely on the code to calculate it every time the score is requested. This design slows down Expertiza as every time an instructor clicks on "View Scores", all the calculations are done to display the score. The situation is even worse if we want to weight the scores by the reputation of each reviewer, which is a measure of how closely that reviewer's scores match other reviewers. In this case, the system potentially has to do thousands of arithmetic calculations to display the score for a particular assignment.

Proposed Solution

We propose that we have two mechanisms to handle the holistic scores, depending on the current state of the assignment: This new approach will take the responses and the database will act as a storage environment.

• OnTheFlyCalc:
  • Calculates holistic scores over a set of data: A set of records in answers table, namely the responses from user A to user B on each criterion.
  • When an assignment is still ongoing, the reputations for each reviewer will be calculated and handled by OnTheFlyCalc
  • This is similar to the current scenario
• LocalDBCalc:
  • Calculates the holistic score over a single db query: "get the score that user A gives user B on assignment 999 on round 2"
  • When an asignment has finished, all the reputations will be calculated by LocalDBCalc and stored, since they are finalized. So, when the scores are requested, they will be fetched using a single db query and displayed
  • This will be a new addition to the code as currently there is no class.

In this project, our goal is to make the OnTheFlyCalc and LocalDBCalc work only for peer-review scores. As future projects will introduce other types of scores, the solution must be extensible.

Design Plan

• A new database table "local_db_scores" with following columns will be created to store the holistic scores:
  • id: int
  • score_type: string
  • round: int
  • score: int
  • response_map_id: int (this will be reference to the response_map table)
• A new column "local_scores_calculated" with boolean type will be added to "assignments" table
  • This has the default value false
  • The value will change to true when the holistic scores for that assignment are stored in local_db_scores table
• A new icon will be added to the list of Assignments on the Manage Assignments page for instructor
  • When this icon is clicked, the holistic scores will be calculated using store_total_score method in LocalDBCalc class and then inserted in this new table "local_db_score"
    • If the record to be inserted is a finalized peer-review score, the type stored in the database will be "ReviewLocalDBScore" and the reference_id will be the response_map id
    • The peer-review scores will be calculated and stored for each of the review rounds because it is possible that user A only reviews user B in one out of several rounds
• When a request is made to view the total scores, depending on whether local_scores_calculated column of the assignment is false or true, either OnTheFlyCalc or LocalDBCalc will be called to calculate the total score

To achieve this functionality, following classes will be created:

• OnTheFlyCalc:
  • This class will contain a method "compute_total_scores" which will compute the total score for an assignment by summing the scores given on all questionnaires and return this total score when an instructor tries to view scores for an ongoing assignment
  • It will also contain methods to calculate the average score and score range (min, max) for each reviewee(team) for peer-review
  • Currently OnTheFlyCalc already exists as a module. As it is only used by assignments, we plan to change it into a class and change all its methods into static methods
• LocalDBCalc:
  • This class will also contain a method "compute_total_scores". But this method will compute the total score by querying and summing the scores saved in local_db_scores table instead of summing the scores given on all questionnaires
  • It will also contain a method "store_total_scores" which will be called when an instructor clicks on "Save Scores to db" icon for an assignment
    • This method will compute and save the total scores for all the response maps (reviewer -> reviewee) in the assignment for each round in local_db_scores table.

As seen in the above image, when a user A reviews another user B (or team), a response map is created containing individual scores and responses to questionnaires. When an instructor clicks on "Save Scores to DB" icon for an assignment, "store_total_scores" method is called in LocalDBCalc class which calculates the holistic scores for each response map from the individual scores and saves it in local_db_scores table. This also sets the "local_scores_calculated" attribute to true for the assignment. Now, the next time someone tries to view these scores, they are computed using the scores saved in local_db_scores.

The above image shows the mechanism which will be followed. When an instructor tries to view scores for an assignment, 1st it will be checked if the local_scores_calculated attribute is true or false for the assignment. If it is false, compute_total_scores from OnTheFlyCalc class is called which will add the scores from the questionnaires. However, if it is true, compute_total_scores from LocalDbCalc class is called which will add the corresponding scores from local_db_scores table.

Changes Done

• Created a new table local_db_scores with columns id, score_type, round, score, response_map_id (foreign key to response_map table)

• Added a new boolean type column named "local_scores_calculated" with default value false to assignments table

• OnTheFlyCalc class:
  • Changed "OnTheFlyCalc" module from on_the_fly_calc.rb file into a class
  • Changed all its methods into static methods and passed assignment as a parameter for all the methods
  • Removed the line "include OnTheFlyCalc" from assignment.rb file
  • Made changes in all the places (eg. review_mapping_controller.rb) where any method of OnTheFlyCalc was called, so that the corresponding static method gets called now with assignment as a parameter

• LocalDbCalc class:
  • Created a new file local_db_calc.rb with a new class "LocalDbCalc"
  • Created a static method "compute_total_score" with assignment as parameter
    • This method calculates the total score by adding all the scores of the given assignment stored in local_db_scores table
  • Created a static method "store_total_scores" with assignment as parameter
    • This method saves the total score of each response map of the given assignment for each round in local_db_scores table

• Condition for choosing LocalDbCalc/OnTheFlyCalc
  • compute_total_scores is called from 3 files namely "assignment_team.rb", "assignment_participant.rb" and "participant.rb"
  • Added a new condition in the scores method in these files which checks if the "local_scores_calculated" attribute is set (true) for the given assignment
    • If it is set (true), compute_total_scores from LocalDbCalc class is called
    • If it is not set (false), compute_total_scores from OnTheFlyCalc class is called

• Manage Assignments UI
  • Created a new icon for each assignment. Clicking this icon stores the total scores for each response map of the assignment for each round in local_db_scores table. It then sets the local_scores_calculated attribute of the assignment to true
  • The code for this is in assets/javascripts/tree_display.jsx
  • Created a new method in assignments_controller.rb named "store_scores_to_db" which gets called when the above mentioned icon is clicked
  • This method calls the "store_total_scores" method in LocalDbCalc class
  • A route is also created for this newly created store_scores_to_db method
  • Note: Clicking on the icon twice for the same assignment saves the latest scores again, and while viewing the scores, only the latest entries from the local_db_scores table are considered

Some of these code snippets are attached below

The newly created LocalDbCalc class in models/local_db_calc.rb file with 2 static methods "compute_total_score" and "store_total_scores"

Some of the changes to models/on_the_fly_calc.rb. Changing model to class and methods to static methods by passing assignment parameter.

The addition of the new icon on Manage Assignments page to save the scores to database. Filename: assets/javascripts/tree_display.jsx

The new method in assignment_controller.rb which gets called when the newly created icon is clicked.

The condition used for calling either OnTheFlyCalc or LocalDbCalc. This is similar in assignment_team.rb, assignment_participant.rb and participant.rb

Use Cases

The project can be broken down into below use cases:

• Instructor wants to store the total scores for each response map, round for an assignment in the database. Instructor clicks on "save scores to db" icon and the scores are stored
• Instructor/User wants to view the scores for an assignment before storing the local scores. Total scores are calculated "on the fly" by adding score given for each question by each reviewer
• Instructor/user wants to view the scores for an assignment after storing the local scores. Scores are calculated by using the values stored in local_db_calc database

Test Plan

The above use cases can then be translated into three major scenarios that we need to test. The following table contains the main scenarios we will cover when testing our project.

Scenario Number	Description	Expected Result for Pass
1	Instructor clicks on "save scores to db" for an assignment. The scores are calculated correctly and placed into the local_db_score table. First, set up an assignment with some review questions. Let a student answer the review questions. Call the store_total_scores method in LocalDbCalc and check if a new record has been added to local_db_scores table.	Query the database and verify that a new record has been added to local_db_scores table
2	Local scores are not stored and user wants to view scores. For an assignment for which the scores are not calculated, click on View Scores as an instructor, or Your Scores as a student. The score displayed should be calculated correctly on the fly using OnTheFlyCalc.	Click on view scores for an assignment whose local scores are not stored and verify that the page displays the expected scores.
3	Local scores are stored and user wants to view scores. For an assignment, first click on save scores to db icon to store the total scores. Then click on View Scores as an instructor, or Your Scores as a student. The score displayed should be calculated correctly on the fly using LocalDbCalc. This test will mainly be a check if clicking view scores on the UI will result in the expected value showing up on the UI. The interaction in getting the information in the database is already tested in Scenario 1.	Click on view scores for an assignment whose local scores are stored and verify that the page displays the expected scores.

Future Work

• Currently OnTheFlyCalc and LocalDbCalc are used to compute the total score for an assignment. More functionalities as below can be added to calculate/display the total scores for a team in an assignment or the total score given by each reviewer. The below mentioned additions/changes will heavily improve the response time.
  • LocalDbCalc stores the total score for each response map for each round
    • These stored values in local_db_scores table can be used directly every time a student wants to view the total score they received from a particular reviewer in a particular round or when an instructor wants to see the total scores by each reviewer given to each team.
    • These stored values in local_db_scores table can also be used to calculate the total score received by each team, every time a team wants to view their total score or an instructor wants to view the total scores received by each team.
• Adding conditions to show the "save scores to db" icon only when an assignment has finished, and stop showing it again once the scores are stored.
  • Currently this icon is present all the time and can be clicked on multiple times, even when an assignment is ongoing. So clicking on this adds the same/updated entries to the database. While viewing the scores, only the latest scores saved in database are considered.
  • So, as a future work, this can be modified such that the icon is seen only when an assignment has finished and the scores are not yet stored. This will prevent the scores being stored multiple times or before an assignment has finished.
• Adding the functionality to remove the scores stored in the database. This will especially be useful after implementing the above mentioned future work because after the above future work functionality, the icon to store the scores will not be shown and if scores are updated somehow, they can't be stored again.
  • In our current functionality, if the scores are updated, the latest scores can be stored again because the icon is always present.
• Add functionality to call OnTheFlyCalc to compute the scores of an assignment whose scores are already stored (if required).
  • Currently, once the scores are stored in database for an assignment, every time LocalDbCalc will be used to view the scores of that assignment.