CSC/ECE 517 Fall 2019 - E1998. Weights in grade calculation

This wiki page has been created to document the changes that will be made under E1998 - Weights in grade calculation as a part of the final project for Fall 2019, CSC/ECE 517.

About Expertiza

Expertiza is an open source project which can be used by instructors for a multitude of use cases such as creating assignments of various types, creating rubrics, questionnaires, allowing teams to be created (for group assignments) etc. It also allows grading of the assignments created, and provides the ability to let students peer review other students' work.

Project Overview

Background and Problem Statement

The questionnaire/rubric creation feature of Expertiza also allows weights to be added to the questions being created, thereby allowing an instructor to create a questionnaire with relative ordering among the questions from a grading standpoint i.e., correctly answering a question with a higher weight will fetch more marks as compared to correctly answering a question with a lower weight.

However, the same functionality is not present for quiz questionnaires. Quiz questionnaires are essentially a way to ensure that students review others' work genuinely. When an assignment has quizzes enabled, each student gets an option to create a quiz for each of his/her reviewers after the initial submission is done and a review round has occurred.

The flow which is followed from quiz creation to score evaluation is as follows:

While creating this quiz, a student is supposed to ask questions related to his/her work which ideally a reviewer must be able to answer. Once it is created, reviewers are able to answer these quizzes and the corresponding scores are evaluated for each quiz separately. Ideally, each question should have a weight associated with it which a student creating the quiz should be able to specify, which in turn should be used while calculating scores for those taking these quizzes. However, currently, there is no way to specify weights for individual questions while creating quizzes and hence scores are being calculated by giving each question equal weightage.

Goal of the Project

The goal of this project is to understand the cause of the aforementioned issue and to ensure that the weights associated with questions are taken into consideration when grading all types of questionnaires involving weights. Our main focus will be to implement weights for quiz questionnaires, To achieve this, first we will have to introduce a way for a student creating a quiz to be able to enter weights for individual questions. Once weights for each question are being successfully stored, we can proceed to change how scores for each reviewer(for a particular quiz) are being calculated such that these newly added weights are also being taken into consideration.

Current Implementation

Expertiza allows multiple questionnaires to be associated with an assignment. For instance, an assignment may have 3 questionnaires - one each for reviewing teammates, one for reviewing others' work, and one for giving feedback to the reviews received. There are various questions that can be created for each type of questionnaire but they can be broadly classified into two types - weighted and non-weighted.

The user-submitted responses to these questions can be accessed via a view called score_view which extracts results from three different tables - questions, questionnaires, and answers. This view has a column named question_weight which is populated by the user entered weights for weighted questions and is set to null for non-weighted questions. This is the view which the code uses for calculating grades for a particular assignment.

Currently, for questionnaires other than quiz questionnaires, an instructor creates a rubric for the questionnaire where he/she can specify weights for each weighted question that exists for that questionnaire. However, the way it is implemented is that the instructor first has to create a question without specifying any weight. The weight for this newly created question is set to 1 by default. After this is done, the instructor has to edit that question to trigger a way to enter a weight for that question. This method for specifying weights is non-intuitive and cumbersome.

For quiz questionnaires on the other hand, there is no way to enter weights for newly added questions at all. The weights are set to 1 by default for all the weighted questions and are not modified henceforth. As a result, scores are calculated for each quiz for a particular reviewer by giving equal weightage for each question. This calls for a complete overhaul of how scores for quiz questionnaires are being calculated to include weights which will need to be accepted from the student creating the quiz.

Changes made by the team

We followed a two-step approach for this project.

Questionnaires except quiz questionnaire

• The first step involved handling all questionnaires except the quiz questionnaire, ensuring that all of them store weights associated with any weighted questions they might have. If so, during score calculation, these weights need to be taken into consideration and the appropriate score displayed.

• We found that method get_total_score was responsible for the score calculation and that it took weights into consideration. A code snippet for the same:

  def self.get_total_score(params)
    @response = params[:response].last
    if @response
      @questions = params[:questions]

      weighted_score = 0
      sum_of_weights = 0
      max_question_score = 0

      @questionnaire = Questionnaire.find(@questions[0].questionnaire_id)

      questionnaireData = ScoreView.find_by_sql ["SELECT q1_max_question_score ,SUM(question_weight) as sum_of_weights,SUM(question_weight * s_score) as 
      weighted_score FROM score_views WHERE type in('Criterion', 'Scale') AND q1_id = ? AND s_response_id = ?", @questions[0].questionnaire_id, @response.id]
      # zhewei: we should check whether weighted_score is nil,
      # which means student did not assign any score before save the peer review.
      # If we do not check here, to_f method will convert nil to 0, at that time, we cannot figure out the reason behind 0 point,
      # whether is reviewer assign all questions 0 or reviewer did not finish any thing and save directly.
      weighted_score = (questionnaireData[0].weighted_score.to_f unless questionnaireData[0].weighted_score.nil?)
      sum_of_weights = questionnaireData[0].sum_of_weights.to_f
      # Zhewei: we need add questions' weights only their answers are not nil in DB.
      all_answers_for_curr_response = Answer.where(response_id: @response.id)
      all_answers_for_curr_response.each do |answer|
        question = Question.find(answer.question_id)
        # if a questions is a scored question (criterion or scale), the weight cannot be null.
        # Answer.answer is nil indicates that this scored questions is not filled. Therefore the score of this question is ignored and not counted
        # towards the score for this response.
        if answer.answer.nil? && question.is_a?(ScoredQuestion)
          question_weight = Question.find(answer.question_id).weight
          sum_of_weights -= question_weight
        end
      end
      max_question_score = questionnaireData[0].q1_max_question_score.to_f
      if sum_of_weights > 0 && max_question_score && !weighted_score.nil?
        return (weighted_score / (sum_of_weights * max_question_score)) * 100
      else
        return -1.0 # indicating no score
      end
    end
  end

• Our next step was to address the issue of how weight entry while creating a rubric was being handled. As mentioned in the previous section, the user had to first create a question and then edit it to specify a weight. What we did was add a text box during the creation of a question itself where the user can enter the appropriate weight. One important point we needed to address was to only allow weight entry for weighted questions (Criterion and Scale). This was taken care by JavaScript and the corresponding changes can be found in _question_weight.html.erb. Finally, we changed the assignment of question.weight variable in questionnaires_controller.rb to ensure that the user entered weights are being taken into consideration.

Code screenshot:

Corresponding UI changes: (We added a Weight field to allow its entry as shown)

• The last stage in this step was to ensure that appropriate changes are made to tests written for questionnaires controller if they fail.

Quiz questionnaire

• For the quiz questionnaire, the first step was to implement a UI change to allow the student creating a quiz to be able to enter the weight for each question. We've used a number field to input the weights, and ensured that users cannot enter negative weights for a question. In case the user enters a negative weights, an appropriate error message is displayed when the user tries to save the quiz. Further, we've also made changes to the edit view for the quiz which allows students to edit the question weights in addition to other question attributes for the quiz.

A screenshot of this change is provided below:

The changes in the following screenshot ensure that the user entered weights are being stored corresponding to the questions as compared to the previously hard coded value of 1:

• The next course of action was to actually implement a way to use these stored weights while calculating the score for a quiz. This involved making changes in the app/models/quiz_response_map.rb file and replacing the existing logic to calculate the score.

New logic for quiz score calculation:

• The "score_views" view is used to calculate the score of the quiz by considering weights set by the quiz creator on each question.
• The score for each answer is stored as 1 if the quiz taker selected the correct option and 0 otherwise.
• The weight of each question is multiplied with the score to compute the score received the by quiz taker for the corresponding answer.
• The total score for all questions is divided by the total quiz score (sums of question weights) and converted into a percentage to get the final score.
• No partial credits are considered for multiple-choice questions, all the options must be correct to gain marks.

Code changes for the above:

• The last stage was to write automated tests for the new functionality we implemented such that it tested as many kinds of quizzes as possible and ensured that the scoring of questions took into consideration the weights being set for individual questions.

The final flow of how quiz questionnaires are handled after our changes can be summarized as follows:

Design Pattern

We have utilied the Facade Design Pattern to hide the complexity of quiz score calculation and provided a single method quiz_score to calculate and return the score of the quiz considering weights on each question. The method makes the query call which gets the question, the weight, the corresponding answer score obtained by the student. Then it converts the values into a total and calculates the final percent score obtained by the student in the quiz. The method encapsulates the entire logic and returns the total weighted score for the corresponding quiz instance.

Files modified

The following files were modified to accomplish the aforementioned changes:

• questionnaires_controller.rb

add_new_questions

The weights were being hard coded earlier, which have now been replaced by the user specified weights via the UI. This method creates new questions when creating questions in questionnaires of types such as review questionnaire, author feedback questionnaire, teammate review questionnaire, etc.

save_new_questions

This method saves questions when a student creates a quiz questionnaire. There was no way to associate weights with questions in quiz questionnaire earlier, we've made changes to allow adding weights to questions in a quiz questionnaire. The corresponding changes are present in this method.

• _new_question_template.html.erb

The input field for adding question weights at the time of creation of questions in a quiz has been added to this file. This functionality was not present earlier, all questions in a quiz were associated with a default weight of 1. The weights entered in this input field are used by the save_new_questions method in questionnaires_controller.rb

• _questionnaire.html.erb

The input field for adding question weights at the time of creation of scored questions for questionnaires of all types except quiz questionnaire has been added in this file. This functionality was not present earlier, as questions had to be created first and then edited to change the default weight (1) associated with it. The weights entered in this input field are used by the add_new_questions method in questionnaires_controller.rb

• _question_weight.html.erb

This file contains the JavaScript code which handles the conditional display of the field to input weights depending on the type of question the user wants to create in the above view. This code ensures that the field is displayed only if the user is trying to created a question with which weights can be associated - questions of type Criterion and Scale.

• edit.html.erb

This files was had to be modified to include the above partial which contains the JavaScript for conditional display of the input field for entering question weight.

• quiz_response_map.rb

quiz_score

This method has been modified to include the logic for calculation of quiz scores. The added logic incorporates the weights which the user can now input corresponding to questions when creating quizzes.

• score_view.rb

readonly

This method has been modified to allow factories of this class to be created for testing.

The following file was modified to included automated tests for creation and grading of a quiz questionnaire:

• quiz_spec.rb

This file contains the necessary changes for testing the entire flow, right from creating a quiz to submitting it, and finally scoring it. The test has been created to check all the modifications we've made to the above files. Also, some of the existing tests had to be modified to incorporate the changes in the way the score is calculated using the new logic.

• quiz_factory.rb

This file contains the factory for score_views which is used by quiz_score to compute the grade by taking weights into consideration.

Test Plan

As a part of creation of RSpec tests, we created an example in quiz_spec.rb file which sets up the necessary actions before a student A can create a quiz questionnaire. We then proceed with quiz questionnaire creation and associate weights with questions created. We create all three types of questions that can be created in a quiz questionnaire - True/False questions, Multiple Choice Radio questions and Multiple Choice Checkbox questions. Further, we login as student B who has reviewed student A's work and take the quiz. Finally, we submit the quiz and ensure the that the quiz is scored after taking the weights associated with the questions into consideration.

Please follow the following steps to test our changes and/or reproduce the functionality implemented by us:

1. Find an assignment and find some participants for the assignment using add participant option corresponding to the assignment in the assignment page.
2. Enable the has quiz option for the corresponding assignment if it is not already enabled.
3. An input field will appear at the right of the has quiz checkbox, please enter the number of questions in the quiz in that checkbox.
4. Now, modify the submission deadline of assignment to some date in the near future.
5. Also, select yes for the stage in which you wish to allow students to take quizzes in the due dates tab.
6. Login as one of the participants (or impersonate one) say A, submit something in the your work section for the assignment.
7. This will enable the create quiz link in the your works section, please go ahead and create a quiz.
8. Now, modify the submission deadline of assignment to some date in the past, so that the assignment is now in review stage.
9. Now login as another assignment participant say B, review A's work.
10. Click on the take quizzes link, a radio button with the quiz created by participant A should appear, go ahead and take the quiz.
11. The quiz will be graded taking into consideration the weights you used for creating the questions in the quiz as participant A.

Quick Links

• Video Walk-through Link: https://drive.google.com/drive/u/1/folders/1GcdSuqH4jcM4Ax-rQ-4FDPYe82NG3cfJ
• Forked Repository Link: https://github.com/theSaurabhMhatre/expertiza
• Pull Request Link: https://github.com/expertiza/expertiza/pull/1633
• Demo Link: http://152.7.99.125:8080/

Team

Our team:

• Sanket Pai (sgpai2@ncsu.edu)
• Sanveg Rane (ssrane2@ncsu.edu)
• Saurabh Mhatre (smhatre@ncsu.edu)
• Saurabh Shingte (svshingt@ncsu.edu)

Mentor: Carmen Bentley (cnaiken@ncsu.edu)