CSC/ECE 517 Fall 2016 E1682: Improve score calculation
E1682. Improve score calculation
Introduction
Expertiza
Expertiza is an open source project for school assignment management for instructors and students based on the Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and word documents. Expertiza provides a dashboard for all the assignments corresponding to a course and provides absolute control to the Instructors and Teaching Assistants. In addition to assignments, it encompasses peer reviews wherein participants are allowed to provide feedback anonymously about each other's work thereby providing scope for the better outcome. The due_date.rb file is responsible for informing the users about the deadline for submission of the each assignment. Due dates in Expertiza have their association with many other components like assignments, reviews etc.
Purpose
The purpose of the project is to improve the functionality of score calculation. The scores in expertiza were calculated by taking the sum of weighted scores to calculate the total every time it is called. This takes considerable amount of time as there is a calculation involved each time. The methodology behind score calculation in expertiza can be significantly improved. The design behind calculating and storing scores can be tweaked to offer significant performance improvements. To achieve this, it is ideal to store overall holistic stores in addition to individual criterion scores.
Problem Statement
The current drawback of the expertiza system lies in how the scores are stored: Expertiza stores the scores based on each response to each criterion, but no holistic scores are stored. As a result of this mechanism, the process slows down. This means that if we need to know what score user A gives to user B based on 100, we have to rely on the code to calculate it (since in answers table we only have the score that A gives B on each criterion).
To overcome this bottleneck, two additional mechanisms can be included for handling the holistic scores.
Overall Description
Expertiza currently calculates scores dynamically when a page is called by taking the weighted scores of each question into consideration. This calculation is a time consuming process with a considerable overhead that significantly affects the performance. To solve the issue we propose a solution to store the holistic scores in a database, retrieving it every time it is required. This reduces the calculation that is required every time the score is called. When the user wants to check the scores of a completed assignment, a method localDB_calc is called and expertiza retrieves the scores of the student from the database rather than calculating the total from weighted scores. But, when the scores of an ongoing assignment is required, a method on_the_fly_calc is called that calculates the current score of the user. To determine which method will be invoked(on_the_fly_calc and localDB_calc), the current system time will be compared with last_review_deadline
- If system time is greater than last_review_deadline, indicating that the deadline has passed, the method localDbcal will retrieve the holistic score from the database
- If system time is lesser than last_review_deadline, indicating that the deadline has not passed, the method on_the_fly_cal will calculate the holistic score and display it.
*FLOW CHART
Tasks Identified
The two suggested mechanisms for handling the storage and calculation of holistic scores for peer-reviews, as specified in the requirements document are :
- on_the_fly_calc
- localDB_calc
on_the_fly_calc
When the assignment is still ongoing , this method is called and it calculates holistic scores over a set of data (in this case, a set of record in answers table, namely the responses from user A to user B on each criterion) and store it in localdbscores. This stored score is retrieved and printed when the user A clicks on the "Your scores" link.
In the current implementation, when a user A wants to view the score given by user B, user A's score(i.e. out of 100) will be calculated everytime using the scores given by user B for each criterion (i.e.. score out of 5).
Handles the calculation of reputations for each reviewer in the case of an ongoing assignment.
localDB_calc
Once the deadline for an assignment is completed, the method localDB_calc calculates the holistic score over a single db query (in this case, a query would be: “get the score that user A gives user B on assignment 999 on round 2”). The query can also ask to print the average of all the scores in round 1 and round2, this average will be retrieved from localdbscores and printed.
This method handles the calculation of reputations for each reviewer in the case of a finished assignment. (all the reputations are calculated and stored since they are finalized).
Database
To store the holistic scores when an assignment is completed, a database localdbscores needs to be created. The value from this db is retrieved and displayed when the scores of a completed assignment needs to be viewed. The database Design is shown below.
Database Design
A table localdbscores will be created as:
class CreateLocalDB<ActiveRecord::Migration
def self.up
create_table :localdb_scores do |t|
t.column :id, :integer
t.column :score, :integer
t.column :round, :integer
t.column :type, :string
t.column :reference_id, :integer
end
with id, score, round, type, reference_id as its attributes as shown in the table below
"localdbscores" will have the following scores
| t.integer "id", limit: 24 | id | Primary Key, Auto generated | |
| t.string "Type", limit: 255 | Type | Type of score: either final or intermediate | |
| t.integer "Score", limit: 24 | Score | Total score calculated based on each response to each criterion | |
| t.integer "Round", limit:24 | Round | The completed round number | |
| t.integer "Reference_id" , limit:45 | Reference_id | Acts as the foreign Key |
Implementation
Assignment.rb
A new function get_last_review_date is added which takes assignid as a parameter and returns the last review deadline of the assignment. The deadline obtained from AssignmentDueDate for the corresponding assignid is stored in last_review_deadline and is returned.
Parent Class:ScoreCal
ScoreCal stores the score calculated by its sub classes: on_the_fly_cal and LocalDbcal to the database. In the database, a new table localdbscore is created with the schema as explained in section 2.2. Depending upon the deadline_type, object of either on_the_fly_cal or LocalDbcal is created and the appropriate functions within the classes are called.
Sub class 1: on_the_fly_cal
This is a subclass of class Scorecal contains a method calc_score whose functionality is to compute the holistic score. An object of on_the_fly_cal is called when the current time has not yet surpassed the time stored in last_review_deadline variable which contains the review duedate indicating that the assignment is still ongoing and has not been completed. The individual scores are retrieved from the database ScoreView and is stored in QuestionnaireData from which the holistic score of a student is calculated .
When the user accesses the scores for a review, the current implementation does not store a holistic score but rather calculates the holistic score from the score for every individual criterion. To calculate the total holistic score for a specific round, the individual scores are weighted and added.
This functionality is achieved by on_the_fly_calc.
A snapshot of the system before implementing the changes is shown below.
Sub class 2: LocalDbcal
This is a subclass of Scorecal and contains a function calc_score in class localDbcal. An object of LocalDbcal is called when the current time has surpassed the time stored in last_review_deadline variable which contains the review due date indicating that the assignment has been completed. This subclass retrieves the score from the database localDbscores for a corresponding student. If the score is not found in the database localDbscores, the function calc_score of on_the_fly_cal class is called to calculate the total from the weighted scores of each question. This calculated score is then inserted into the database.
This score is now returned to the view from where it was called.
_review_table.html.erb
The current implementation of this was to calculate the score and display it which as we know is a time consuming process. It has been changed to check if the current time has crossed the last_review_deadline obtained in assignments.rb and call the corresponding class as required.
If the current date has surpassed the last_review_deadline, it is an indication that the due date has passed and the class localDbcal is called to retrieve a value from the database as mentioned above. On the other hand, on_the_fly_cal is called when the current date has not surpassed the last_review_deadline, indicating that the assignment is ongoing and the holistic score is calculated and passed to the view.
Design Pattern
The ScoreCal superclass has 2 subclasses: OnTheFlyCal and LocalDbCal. ScoreCal acts as an interface to obtain the scores, either from OnTheFlyCal or LocalDbCal depending on the last_review_deadline obtained from the get_last_review_date() in the assignment model. This pattern is similar to strategy pattern where the client is the _review_tble.html.erb, interface is the superclass and the subclasses are called accordingly.
Implementation
Parent Class:ScoreCalc
ScoreCalc stores the score calculated by its sub classes: on_the_fly_calc and LocalDBcalc to the database. In the database, a new table local_db_scoresn is created with the schema as explained in section 2.2. Depending upon the deadline_type, object of either on_the_fly_calc or LocalDBcalc is created and the appropriate functions within the classes are called.
Database design
A table local_db_scores will be created as:
create_table "local_db_scores",force::cascade do|t|
"local_db_scores" will have the following scores
| t.integer "id", limit: 24 | id | Primary Key, Auto generated | |
| t.string "Type", limit: 255 | Type | Type of score: either final or intermediate | |
| t.integer "Score", limit: 24 | Score | Total score calculated based on each response to each criterion | |
| t.integer "Round", limit:24 | Round | The completed round number | |
| t.integer "Reference_id" , limit:45 | Reference_id | Acts as the foreign Key |
Sub class 1: on_the_fly_calc
An object of on_the_fly_calc is called when the deadline_type is not "review of review" which means that the assignment is ongoing and has not been completed and stores the scores till the current stage that will change once the assignment passes the deadline
The function compute_total_scores(scores) is used to calculate the total score based on each response to each criterion. The function is as below:
def compute_total_scores(scores)
total=0
self.questionnaires.each{|questionnaire| total+=questionnaire.get_weighted_score(self,scores)}
total
end
The value in variable "total" needs to be stored in local_db_scores under "Score" and will be retrieved by using another function retrieve_total_score to display the total score instead of calculating it every time a user wants to view the scores.
When the user accesses the scores for a review, the current implementation does not store a holistic score but rather calculates the holistic score from the score for every individual criterion. To calculate the total holistic score for a specific round, the individual scores are weighted and added.
This functionality is achieved by on_the_fly_calc.
The current system calculates the holistic score every single time from the individual criterion. A snapshot of the system before implementing the changes is shown below.
The proposed implementation involves computing the holistic score once and storing it in the local_db_scores database for a particular user and for a specified round. The stored value is accessed and obtained instead of computing the holistic score every single time.
Sub class 2: LocalDB_calc
An object of LocalDB_calc is called when the deadline_type is "review of review" which means that the assignment is completed and stores the final score in the database local_db_scores after computing it rather than computing it every time the score is required.
Function compute_total_scores(scores) is added to the class which calculates the total score based on each response to each criterion. The function(polymorphism) is as below:
def compute_total_scores(scores)
total=0
self.questionnaires.each{|questionnaire| total+=questionnaire.get_weighted_score(self,scores)}
total
end
The value in variable "total" needs to be stored in local_db_scores under "Score" and will be retrieved by using another function retrieve_total_score to display the total score instead of calculating it every time a user wants to view the scores.
The snapshots prior to and post implementation are shown below :
The proposed implementation retrieves the round wise scores from the local_db_scores database and displays the average score once the deadline has passed rather than computing the average of the weighted scores for every access of the page as shown below.
The Dataflow diagram (DFD) for the proposed system is shown below.
Testing
The testing for model: on_the_fly_calc is carried out by questionnaire_spec.rb in the spec folder. The minimum and maximum score that an user gives for each criterion is checked.
Two more test cases to check whether the average score of each review falls within a given range:(0,100) will be added.
For LocalDB_calc, the average calculated across round 1 and round 2 will be checked to fall within 0 to 100. These test cases will be added to questionnaire_spec.rb.