CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation

From Expertiza_Wiki
Jump to navigation Jump to search

This wiki page has been created to document the changes that will be made under E1998 - Weights in grade calculation as a part of the final project for Fall 2019, CSC/ECE 517.

About Expertiza

Expertiza is an open source project which can be used by instructors for a multitude of use cases such as creating assignments of various types, creating rubrics, questionnaires, allowing teams to be created (for group assignments) etc. It also allows grading of the assignments created, and provides the ability to let students peer review other students' work. Additionally, it also allows an instructor to export data for various objects created into multiple formats for further analysis.

Project Overview

Background and Problem Statement

The questionnaire/rubric creation feature of Expertiza also allows weights to be added to the questions being created, thereby allowing an instructor to create a questionnaire with relative ordering among the questions from a grading standpoint i.e., correctly answering a question with a higher weight will fetch more marks as compared to correctly answering a question with a lower weight.

However, the same functionality is not present for quiz questionnaires. Quiz questionnaires are essentially a way to ensure that students review others' work genuinely. When an assignment has quizzes enabled, each student gets an option to create a quiz for each of his/her reviewers after the initial submission is done and a review round has occurred. While creating this quiz, a student is supposed to ask questions related to his/her work which ideally a reviewer must be able to answer. Once it is created, reviewers are able to answer these quizzes and the corresponding scores are evaluated for each quiz separately. Ideally, each question should have a weight associated with it which a student creating the quiz should be able to specify, which in turn should be used while calculating scores for those taking these quizzes. However, currently, there is no way to specify weights for individual questions while creating quizzes and hence scores are being calculated by giving each question equal weightage.

Goal of the Project

The goal of this project is to understand the cause of the aforementioned issue and to ensure that the weights associated with questions are taken into consideration when grading all types of questionnaires involving weights. Our main focus will be to implement weights for quiz questionnaires, To achieve this, first we will have to introduce a way for a student creating a quiz to be able to enter weights for individual questions. Once weights for each question are being successfully stored, we can proceed to change how scores for each reviewer(for a particular quiz) are being calculated such that these newly added weights are also being taken into consideration.

Current Implementation

Expertiza allows multiple questionnaires to be associated with an assignment. For instance, an assignment may have 3 questionnaires - one each for reviewing teammates, one for reviewing others' work, and one for giving feedback to the reviews received. There are various questions that can be created for each type of questionnaire but they can be broadly classified into two types - weighted and non-weighted.

The user-submitted responses to these questions can be accessed via a view called score_view which extracts results from three different tables - questions, questionnaires, and answers. This view has a column named question_weight which is populated by the user entered weights for weighted questions and is set to null for non-weighted questions. This is the view which the code uses for calculating grades for a particular assignment.

Currently, for questionnaires other than quiz questionnaires, an instructor creates a rubric for the questionnaire where he/she can specify weights for each weighted question that exists for that questionnaire. However, the way it is implemented is that the instructor first has to create a question without specifying any weight. The weight for this newly created question is set to 1 by default. After this is done, the instructor has to edit that question to trigger a way to enter a weight for that question. This method for specifying weights is non-intuitive and cumbersome.

For quiz questionnaires on the other hand, there is no way to enter weights for newly added questions at all. The weights are set to 1 by default for all the weighted questions and are not modified henceforth. As a result, scores are calculated for each quiz for a particular reviewer by giving equal weightage for each question. This calls for a complete overhaul of how scores for quiz questionnaires are being calculated to include weights which will need to be accepted from the student creating the quiz.

Changes made by the team

We followed a two-pronged approach for this project.


Questionnaires except quiz questionnaire
  • The first step involved handling all questionnaires except the quiz questionnaire, ensuring that all of them store weights associated with any weighted questions they might have. If so, during score calculation, these weights need to be taken into consideration and the appropriate score displayed.
  • We found that method get_total_score was responsible for the score calculation and that it took weights into consideration according to this equation:

def get_total_score screenshot

  • Our next step was to address the issue of how weight entry while creating a rubric was being handled. As mentioned in the previous section, the user had to first create a question and then edit it to specify a weight. What we did was add a text box during the creation of a question itself where the user can enter the appropriate weight. One important point we needed to address was to only allow weight entry for weighted questions(Criterion and Scale).

text box for add question in questionnaire screenshot(weighted and unweighted)

  • The last stage in this step was to ensure that appropriate changes are made to tests written for questionnaires controller if they fail.






Our main plan is to ensure that all the code that is responsible for calculating scores is taking weights into account if the final scores depend on doing so. Since the questions can be weighted or unweighted, while calculating the final scores, the score assigned by a user to individual questions will be handled (aggregated) differently depending on the type of question. Our job will be to verify whether weights are being multiplied with the individual scores for weighted questions and modify the code if weights are not considered.

To achieve the aforementioned objective, the following steps will need to be followed:

  • We need to first identify the files where the code for score calculation exists in the current implementation
  • Then we will need to peruse the code responsible for creating rubrics/questionnaires and editing or manipulating them in any way
  • In particular, a module named on_the_fly_calc.rb is responsible for score calculation, which will be our main area of focus

Another important task for our team will be to ensure our changes do not break existing functionality by writing relevant tests for the different types of questionnaires(review, teammate review, author feedback, meta-review, quiz, etc).

Please find below a high level overview of the approach that we'll be following to tackle this problem:

Files which require modification (Tentative)

The following methods in the files might require modifications:

  • answer.rb
compute_scores
This method receives data, validates it and called get_total_score which does has the logic for taking weights into consideration for weighted questions.
get_total_score
This is the method which multiples the weights for questions with the score received for the corresponding questions. Currently, this is being done directly via a query followed by logic to aggregate the score received for multiple questions. We will have to investigate the logic and ensure that the code is working as expected.
  • grades_controller.rb
view_team
This (one of the) controller method(s) which receives the request for score calculation, which in turn invokes the aforementioned methods during the course of its execution.
  • assignment_participant.rb
scores
This is one of the intermediate methods called during the calculation of score which prepare the input to be used by some of the aforementioned methods.
compute_assignment_score
Again, an intermediate method which prepare input data so that it can be used by other methods which do grade calculations.
We may have to modify these method if required.
  • questionnaires_controller.rb
add_new_questions
This is the controller method which gets called when a question is created for the first time and sets the questions weight to 1 by default for weighted questions.
save_all_questions
This is another controller methods which get called when modifying any question attributes including weights after the questions have been created.
  • questionnaire.rb
compute_weighted_score
Again, another intermediate question during the course of execution to calculate grades. We may need to modify this if required.
  • on_the_fly_calc.rb
calc_review_score
This method aggregates the scores for all the questions in a questions and return the total score.

Test Plan

  • All of the above-mentioned files and the corresponding methods already have rspec files defined to test their functionalities, so our first task after making our changes will be to ensure they do not break for the new changes.
  • The next course of action will be to write new tests to cover all the code changes we will be making.
  • Since this project involves locating, replicating and then fixing a bug, we are yet to identify precisely which methods will be changed.
  • That being said, we can say with reasonable certainty that we will require additional tests for questionnaires_controller.rb to ensure that while adding new questions they are assigned weights if they are weighted questions. In case they are, appropriate weights must be assigned.
  • Another test we will have to either modify or write will be for compute_weighted_score method of questionnaire.rb model to ensure that while calculating the final score weights are being considered. Similar tests will have to be written if applicable for calc_review_score and compute_avg_and_ranges_hash for the on_the_fly_calc.rb file.
  • Finally, we will attempt to create tests which would include creating questionnaires of every relevant type with weighted questions, and ensuring that the weights are included correctly in the grade calculations.

Quick Links

Team

Our team:

  • Sanket Pai (sgpai2@ncsu.edu)
  • Sanveg Rane (ssrane2@ncsu.edu)
  • Saurabh Mhatre (smhatre@ncsu.edu)
  • Saurabh Shingte (svshingt@ncsu.edu)

Mentor: Carmen Bentley (cnaiken@ncsu.edu)

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2019_-_E1998._Weights_in_grade_calculation&oldid=129682"