CSC/ECE 517 Fall 2016 E1682: Improve score calculation
E1682. Improve score calculation
Introduction
Purpose
The methodology behind score calculation in expertiza can be significantly improved. The design behind calculating and storing scores can be tweaked to offer significant performance improvements. To achieve this, it is ideal to store overall holistic stores in addition to individual criterion scores.
Problem Statement
The current drawback of the expertiza system lies in how the scores are stored: Expertiza stores the scores based on each response to each criterion, but no holistic scores are stored. As a result of this mechanism, the process slows down. This means that if we need to know what score user A gives to user B based on 100, we have to rely on the code to calculate it (since in answers table we only have the score that A gives B on each criterion).
To overcome this bottleneck, two additional mechanisms can be included for handling the holistic scores.
Proposed changes
The two suggested mechanisms for handling the storage and calculation of holistic scores for peer-reviews, as specified in the requirements document are :
- on_the_fly_calc
- localDB_calc
on_the_fly_calc
Calculate holistic scores over a set of data (in this case, a set of record in answers table, namely the responses from user A to user B on each criterion) and store it in local_db_scores. This stored score is retrieved and printed when the user A clicks on the "Your scores" link.
In the current implementation, when a user A wants to view the score given by user B, user A's score(i.e. out of 100) will be calculated everytime using the scores given by user B for each criterion (i.e.. score out of 5).
Handles the calculation of reputations for each reviewer in the case of an ongoing assignment.
localDB_calc
Calculate the holistic score over a single db query (in this case, a query would be: “get the score that user A gives user B on assignment 999 on round 2”). The query can also ask to print the average of all the scores in round 1 and round2, this average will be retrieved from local_db_scores and printed.
Handles the calculation of reputations for each reviewer in the case of a finished assignment. (all the reputations are calculated and stored since they are finalized).
Observer Pattern
Observer pattern will be used where the dependent:local_db_score will be updated when the deadline_type is changed. If the dead_line_type is anything other than "review of review", the weighted scores for each review will be calculated and scored in local_db_scores. If the deadline_type is "review of review", then the average of the weighted score for each round across users will be calculated and stored in local_db_score.
Implementation
Parent Class:ScoreCalc
ScoreCalc stores the score calculated by its sub classes: on_the_fly_calc and LocalDBcalc to the database. In the database, a new table local_db_scoresn is created with the schema as explained in section 2.2. Depending upon the deadline_type, object of either on_the_fly_calc or LocalDBcalc is created and the appropriate functions within the classes are called.
Database design
A table local_db_scores will be created as:
create_table "local_db_scores",force::cascade do|t|
"local_db_scores" will have the following scores
| t.integer "id", limit: 24 | id | Primary Key, Auto generated | |
| t.string "Type", limit: 255 | Type | Type of score: either final or intermediate | |
| t.integer "Score", limit: 24 | Score | Total score calculated based on each response to each criterion | |
| t.integer "Round", limit:24 | Round | The completed round number | |
| t.integer "Reference_id" , limit:45 | Reference_id | Acts as the foreign Key |
Sub class 1: on_the_fly_calc
An object of on_the_fly_calc is called when the deadline_type is not "review of review" which means that the assignment is ongoing and has not been completed and stores the scores till the current stage that will change once the assignment passes the deadline
The function compute_total_scores(scores) is used to calculate the total score based on each response to each criterion. The function is as below:
def compute_total_scores(scores)
total=0
self.questionnaires.each{|questionnaire| total+=questionnaire.get_weighted_score(self,scores)}
total
end
The value in variable "total" needs to be stored in local_db_scores under "Score" and will be retrieved by using another function retrieve_total_score to display the total score instead of calculating it every time a user wants to view the scores.
When the user accesses the scores for a review, the current implementation does not store a holistic score but rather calculates the holistic score from the score for every individual criterion. To calculate the total holistic score for a specific round, the individual scores are weighted and added.
This functionality is achieved by on_the_fly_calc.
The current system calculates the holistic score every single time from the individual criterion. A snapshot of the system before implementing the changes is shown below.
The proposed implementation involves computing the holistic score once and storing it in the local_db_scores database for a particular user and for a specified round. The stored value is accessed and obtained instead of computing the holistic score every single time.
Sub class 2: LocalDB_calc
An object of LocalDB_calc is called when the deadline_type is "review of review" which means that the assignment is completed and stores the final score in the database local_db_scores after computing it rather than computing it every time the score is required.
Function compute_total_scores(scores) is added to the class which calculates the total score based on each response to each criterion. The function(polymorphism) is as below:
def compute_total_scores(scores)
total=0
self.questionnaires.each{|questionnaire| total+=questionnaire.get_weighted_score(self,scores)}
total
end
The value in variable "total" needs to be stored in local_db_scores under "Score" and will be retrieved by using another function retrieve_total_score to display the total score instead of calculating it every time a user wants to view the scores.
The snapshots prior to and post implementation are shown below :
The proposed implementation retrieves the round wise scores from the local_db_scores database and displays the average score once the deadline has passed rather than computing the average of the weighted scores for every access of the page as shown below.
Testing
The testing for model: on_the_fly_calc is carried out by questionnaire_spec.rb in the spec folder. The minimum and maximum score that an user gives for each criterion is checked.
Two more test cases to check whether the average score of each review falls within a given range:(0,100) will be added.
For LocalDB_calc, the average calculated across round 1 and round 2 will be checked to fall within 0 to 100. These test cases will be added to questionnaire_spec.rb.