CSC/ECE 517 Fall 2020 - E2081. Add a "cake" item type to rubrics

Problem Definition

Expertiza rubrics are utilized to build questionnaires and these rubrics incorporate several kinds of items, including Criterion (dropdown + comment), Checkbox, MultipleChoice, and Scale. When we use these questionnaires for reviews, for example the teammate review assessment we encounter a few problems. One “problem” with all of these types is that there is nothing to stop a reviewer (say some student) from assigning the maximum score to all the reviewees (student's teammates). This is indeed a problem for teammate assessment, when the faculty asks for what fraction of the work each teammate did.So an alternative is needed, let’s call it a “Cake” item type, that allows a reviewer to divide a “cake” in any way between the reviewees, but does not allow him/her to divvy up more than 100% of the cake.

1. When the reviewer submits a score that would bring the total assigned for this item to > 100%, the system needs to warn.
2. The system must a reviewer to give him/herself a score
3. The proposed design needs to be compatible with existing self reviews code and teammates reviews
4. Design should be extensible to other kind of reviews apart from teammate reviews as well

For instance, the above figure consists of a team of 4 members, with self included as a team-member when reviewing every member's contribution.
If A has reviewed his other 3 teammates B, C, D with contributions of 15%, 10%, and 45% respectively, he should only be allowed to review self with a contribution of 30% or lesser.

Design

Proposed Solution

The issue asks us to have a Cake type for the question taking in a participant’s contribution, whenever s/he is reviewing the other teammates. We add in a new Question type ‘Cake’, which will be extended from the Scored Question model [cake < scored question < choice question < question].

Potential ways of displaying the Cake question on the UI:

• Stars: Existing design for teammate reviews uses stars to symbolize the contribution provided by each student. We can implement the cake type using the same.

Cons: Stars are not very versatile when there are a greater number of students per team and if the student wants to equally rate their contribution.

• Drop Down: In order to give the student more flexibility, another way a student can pick the contribution of each team member is using a dropdown of the % values.

Cons: Drop down values need to be restricted to intervals of 5 or more, as the drop down becomes too long to display all values from 0-100.

• Text box with up-down arrows: Provides utmost flexibility and precision to the student while adding contribution of his team members, we can provide a text box with necessary validations which lets the student provide the contribution % for his teammates as any integral number within the limits. There will also be text to warn the user about how much contribution is remaining based on what is typed into the box and any previously completed teammate reviews.

We plan to implement the text box as follows:

Addressing self-reviews for Cake Type:

The amount of credit assigned to self will be the remaining amount of percentage not given to the other teammates for the reviews. This will eliminate the need to do a self-review to specifically assign the leftover percentage to oneself. The self percentages for the cake questions will only be assigned to a student once they have completed the reviews for all other teammates. This will ensure that the students are motivated to complete the teammate reviews and the self-review calculation is executed once (instead of needing to re-calculate after each review).

In the current system workflow, we found that the teammate reviews are taken as instances of TeammateReviewResponses. The questionnaire for reviewing teammates includes the question asking the cake (contribution) factor. Ideally, the cake should include the reviewer as well.

Currently, whenever a participant is reviewing their teammates they are shown a view where the names of their teammates are displayed with a 'Review' link next to them. The last line has the user name displayed with no link but reminder text to complete their reviews. We plan on updating this view once the reviews are complete for all teammates with a 'View' link in the user's line item that will show all of the self cake contributions.

Steps to reproduce the proposed workflow:

1. Log in to expertiza to view the home page
2. Login as an Instructor, and then impersonate a student or login as a Student
3. Go to Assignments -> Your team
4. You will see a list of your teammates with a link: ‘Review’
5. You will see yourself with no present link, this will become visible once you complete all reviews as: 'View'
6. You can see the questions for asking the contribution as a cake type
7. There will be a text description next to it denoting what part of the cake is taken (what contribution factor of the work is used)
8. For yourself the view will automatically be filled out with the leftover percentages for each cake contribution

Why we chose this approach

We found that it was efficient to calculate a self contribution as the remaining percentage not assigned to all other teammates. This eliminates the need to create a specialized self-review to assign a contribution to oneself. We decided that it would be best to allow access to the self-review once all other teammate reviews are complete because it encourages users to do their reviews and the final calculation for self contribution needs to only be done once.

Implementation

A new class Cake has been newly introduced as a subclass of ScoredQuestion. The Cake class is a type of question that can be offered as part of any questionnaire, which keeps an account of all the answer values recorded for one specific question. The maximum value that can be given to a question of type cake is 100, and any value entered above 100 is automatically void, setting the answer entered to zero by default. The user is informed of the same and also can keep track of how much of the “cake” has already been taken, which helps him determine the value that he can enter. Upon entering a value greater than 100, a warning is displayed, informing the user that the value has exceeded 100, and setting the value back to 0. A textbox input with up-down arrows is being used, to help the user increment/decrement values as he pleases.

Files involved:

1. app/controllers/questionnaires_controller.rb

Added cake type in editing the questionnaire.

2. app/controllers/response_controller.rb

A new method calculate_total_score was added. And it is called in methods new and edit to calculate and update the total contribution for a cake question when a student wants to create or edit a review.

3. *app/models/cake.rb

A new model cake.rb was added.

class Cake < ScoredQuestion
    include ActionView::Helpers
    validates :size, presence: true
    #method is called during creation of questionnaire --> when cake type is added to the questionnaire.
    def edit(_count)
      html = '<td align="center"><a rel="nofollow" data-method="delete" href="/questions/' + self.id.to_s + '">Remove</a></td>'
      html += '<td><input size="6" value="' + self.seq.to_s + '" name="question[' + self.id.to_s + '][seq]"'
      html += ' id="question_' + self.id.to_s + '_seq" type="text"></td>'
      html += '<td><textarea cols="50" rows="1" name="question[' + self.id.to_s + '][txt]"'
      html += ' id="question_' + self.id.to_s + '_txt" placeholder="Edit question content here">' + self.txt + '</textarea></td>'
      html += '<td><input size="10" disabled="disabled" value="' + self.type + '" name="question[' + self.id.to_s + '][type]"'
      html += ' id="question_' + self.id.to_s + '_type" type="text"></td>'
      html += '<td><input size="2" value="' + self.weight.to_s
      html += '" name="question[' + self.id.to_s + '][weight]" id="question_' + self.id.to_s + '_weight" type="text"></td>'
      html += '<td>text area size <input size="3" value="' + self.size.to_s
      html += '" name="question[' + self.id.to_s + '][size]" id="question_' + self.id.to_s + '_size" type="text"></td>'
      safe_join(["<tr>".html_safe, "</tr>".html_safe], html.html_safe)
    end

    # Method called after clicking on View Questionnaire option
    def view_question_text
      html = '<TD align="left"> ' + self.txt + ' </TD>'
      html += '<TD align="left">' + self.type + '</TD>'
      html += '<td align="center">' + self.weight.to_s + '</TD>'
      questionnaire = self.questionnaire
      html += '<TD align="center">' + questionnaire.min_question_score.to_s + ' to ' + questionnaire.max_question_score.to_s + '</TD>'
      safe_join(["<TR>".html_safe, "</TR>".html_safe], html.html_safe)
    end

    def complete(count, answer = nil, total_score, view_type)
      if self.size.nil?
        cols = '70'
        rows = '1'
      else
        cols = self.size.split(',')[0]
        rows = self.size.split(',')[1]
      end
      html = '<table> <tbody> <tr><td>'
      html += '<label for="responses_' + count.to_s + '"">' + self.txt + '  </label>'
      html += '<input class="form-control" id="responses_' + count.to_s + '" min="0" name="responses[' + count.to_s + '][score]"'
      html += 'value="' + answer.answer.to_s + '"' unless answer.nil?
      html += 'type="number" size = 5 onchange="validateScore(this.value,' + total_score + ',this.id,\''+view_type.to_s+'\')"> '
      html += '</td></tr></tbody></table>'
      html += '<td width="10%"></td></tr></table>'
      html += '<p>Total contribution so far (excluding current review): ' + total_score + '% </p>' #display total
      html += '<textarea cols=' + cols + ' rows=' + rows + ' id="responses_' + count.to_s + '_comments"' \
          ' name="responses[' + count.to_s + '][comment]" class="tinymce">'
      html += answer.comments unless answer.nil?
      html += '</textarea>'
      html += '<script> function validateScore(val, total_score,id, view_type) {
                var int_val = parseInt(val);
                var int_total_score = parseInt(total_score);
                if(view_type == "teammate")
                {
                  if (int_val+int_total_score>100)
                    {
                      alert("Total contribution cannot exceed 100, current total: " + (int_val+int_total_score));
                      document.getElementById(id).value = 0
                    }
                }
                else
                 {
                    if (int_val > 100)
                    {
                      alert("Total contribution cannot exceed 100, current total: " + (int_val));
                      document.getElementById(id).value = 0
                    }
                  }
              }</script>'
      safe_join(["".html_safe, "".html_safe], html.html_safe)
    end

    # This method returns what to display if a student is viewing a filled-out questionnaire
    def view_completed_question(count, answer)
      score = answer && !answer.answer.nil? ? answer.answer.to_s : "-"
      html = '<b>' + count.to_s + ". " + self.txt + "</b>"
      html += '<div class="c5" style="width:30px; height:30px;' \
        ' border-radius:50%; font-size:15px; color:black; line-height:30px; text-align:center;">'
      html += score
      html += '</div>'
      html += '<b>Comments:</b>' + answer.comments.to_s
      safe_join(["".html_safe, "".html_safe], html.html_safe)
    end

    # Finds all teammates and calculates the total contribution of all members for the question
    def get_total_score_for_question(review_type, question_id, participant_id, assignment_id, reviewee_id)
      # get the reviewer's team id for the currently answered question
      team_id = Team.joins([:teams_users, teams_users: [{user: :participants}]]).where("participants.id = ? and teams.parent_id in (?)", participant_id, assignment_id).first
      if team_id
        team_id = team_id.id
      end
      if review_type == 'TeammateReviewResponseMap'
        answers_for_team_members =  get_answers_for_teammatereview(team_id, question_id, participant_id, assignment_id, reviewee_id)
      # else
      #   answers_for_team_members = get_answers_for_review(question_id, participant_id, assignment_id)
      end
      calculate_total_score(answers_for_team_members)
    end

    # Finds the scores for all teammates for this question
    def get_answers_for_teammatereview(team_id, question_id, participant_id, assignment_id, reviewee_id)
      # get the reviewer's team members for the currently answered question
      team_members = Participant.joins(user: :teams_users).where("teams_users.team_id in (?) and participants.parent_id in (?)", team_id, assignment_id).ids
      # get the reviewer's ratings for his team members
      Answer.joins([{response: :response_map}, :question]).where("response_maps.reviewee_id in (?) and response_maps.reviewed_object_id = (?)
        and answer is not null and response_maps.reviewer_id in (?) and answers.question_id in (?) and response_maps.reviewee_id not in (?)", team_members, assignment_id, participant_id, question_id, reviewee_id).to_a
    end

    # def get_answers_for_review(question_id, participant_id, assignment_id)
    #   Answer.joins([{response: :response_map}, :question]).where("response_maps.reviewed_object_id = (?)
    #     and answer is not null and response_maps.reviewer_id in (?) and answers.question_id in (?)", assignment_id, participant_id, question_id).to_a
    # end

    # Sums up the scores given by all teammates that should be less than or equal to 100
    def calculate_total_score(question_answers)
      question_score = 0.0
      question_answers.each do |ans|
        # calculate score per question
        unless ans.answer.nil?
          question_score += ans.answer
        end
      end
      question_score
    end

  end

4. app/views/questionnaires/_questionnaire.html.erb

Added cake to the dropdown list. Now the instructor can choose to add a cake type teammate review question to the questionnaire.

<%= select "question", "type",
              {"Criterion" =>"Criterion",
               "Scale"=>"Scale",
               "Dropdown"=>"Dropdown",
               "Checkbox"=>"Checkbox",
               "TextArea"=>"TextArea",
               "TextField"=>"TextField",
               "UploadFile"=>"UploadFile",
               "SectionHeader"=>"SectionHeader",
               "TableHeader"=>"TableHeader",
               "ColumnHeader"=>"ColumnHeader",
               "Cake" => "Cake"
               }, {}, {class: "form-control", onchange: "checkQuestionType()"} %> question(s)

5. app/views/response/response.html.erb

Added response showing how much of the total contribution has been assigned for cake-type questions.

<% elsif question.instance_of? Cake %>
              <%= question.complete(i, answer, @total_score[question.id].to_s, @return) %>  

6. app/views/student_teams/view.html.erb

Added a new table intended to display the remaining score in each question the reviewer currently has.

</tr> <!--self contribution for Cake questions-->
    <% if @teammate_review_allowed %>
      <tr style="border: 1px outset #000000; padding: 10px 20px" >
        <th class="head">Teammate review question</th>
        <th class="head">Self contribution percentage</th>
      </tr>
    <% end %> <!--end self contribution for Cake questions-->

Test plan

There are several things we plan on testing in this project as follows:

1. Test the view for 'Your team' to verify the reminder text for completing reviews is there
2. Test the view for 'Your team' to verify that when all reviews are complete there is a link to the auto-generated self-review cake contributions
3. Test the string reminder text for the cake questions that indicate the amount of contribution left to assign
4. Test the cake contribution text boxes for valid responses (no negative numbers, over-allocation, or non-numeric characters)
5. Test the auto-generated review to verify that the remaining contribution fractions are assigned properly to each cake question

Team Members

• Jordan, Dylan Tyler dtjordan@ncsu.edu

• Ruoyun Ma rma9@ncsu.edu

• Han, Lige lhan6@ncsu.edu

• Kollipara, Siddhartha skollip@ncsu.edu

• Mentor: Ed Gehringer (efg@ncsu.edu)

References

• CSC/ECE 517 Fall 2019 - E1992. Add cake type to rubrics design
• Expertiza on GitHub
• The live Expertiza website
• Expertiza project Details
• Expertiza YouTube Channel