CSC/ECE 517 Fall 2021 - E2156. Issues related to meta-reviewing

From Expertiza_Wiki
Jump to navigation Jump to search

Introduction

Expertiza Meta-Reviewing

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

Previous Work

This assignment is a reimplementation of projects E2025 and E1997.

Previous implementations can be found at E2025 and E1997

Meta-Review Issues & Solutions

Issue 1 - No "Begin" Link

Description: When a user requests a new meta-review, (s)he is unable to get 'Begin' link to start a new meta-review.


Solution:

During testing we found that the Begin link was working as expected. It should not link to the questionnaire if work has not been submitted and that work has not been reviewed as shown in the image in Issue 1. We believe this was an issue either resolved previously, or someone tried to test the function without creating and completing a review.


Issue 2 - Incorrect value for "Number of meta-reviews left" / Disappearing button to request new meta-review

Description: When a user requests a new meta-review, the 'number of meta-reviews left' does not change and the button to request a new meta-review disappears.

Refer to image above


Solution:

The current code is: 

 <h4> Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %></h4>

We believe this line should read the number of reviews you're allowed to request. This is an easy fix: 

 <h4> Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %></h4>


Issue 3 - "Has Meta-Review Limit?" and "Set Allowed/Required number of meta-reviews per Reviewer" shown at incorrect times

Description: 'Has Meta-Review Limit?' and 'Set Allowed/Required number of meta-reviews per Reviewer' should only be shown if the instructor has checked the 'Use meta-review deadline' in the Due Dates tab of edit assignment.


Solution:

In order to synchronize Has Meta-Review Limit? and Set Allowed/Required number of meta-reviews per Reviewer, we first added a variable that will track if Meta-Reviews are being used. The new code in assignment edit.html.erb takes care of this when the page is originally pulled up.

In app/views/assignments/edit.html.erb 

  <% if @assignment.num_metareviews_allowed == 0 %>
    <% @metareview_true = false %>
  <% else %>
    <% @metareview_true = true %>
  <%end%>

While the user is still on the assignment edit page, the Meta-Review usage still needs to be tracked and synced across tabs. We do this in _review_strategy.html.erb using the same variable as before. Every time the Use meta review? is either checked or unchecked the function hasMetaReviewChanged() runs. This function will evaluate whether the box is checked or unchecked and fill the num_metareviews_required and num_metareviews_allowed text boxes accordingly. It also will either set or clear the metareview_true variable. This is done so the due dates tab can fill in the due dates table according to whether or not there needs to be a meta-review deadline. As you will see in the function, hasMetaReviewChanged() also calls removeOrAddMetareview which will update the due date table.

The function hasMetaReviewChanged() in _review_strategy.html.erb 

 function hasMetaReviewChanged() {
        removeOrAddMetareview();
        var checkbox = jQuery('#meta_review');
        var meta_reviews_allowed_field = jQuery('#meta_reviews_allowed');
        if (checkbox.is(':checked')) {
            meta_reviews_allowed_field.removeAttr('hidden');
            if(<%= @assignment_form.assignment.num_metareviews_allowed %>){
                jQuery('#assignment_form_assignment_num_metareviews_allowed').val(<%=@assignment_form.assignment.num_metareviews_allowed%>);
                jQuery('#assignment_form_assignment_num_metareviews_required').val(<%=@assignment_form.assignment.num_metareviews_required%>);
            } else{
                jQuery('#assignment_form_assignment_num_metareviews_allowed').val(<%=@assignment_form.assignment.num_metareviews_allowed = 3%>);
                jQuery('#assignment_form_assignment_num_metareviews_required').val(<%=@assignment_form.assignment.num_metareviews_required = 3%>);
            }

            jQuery('#assignment_form_assignment_metareview_true').val('false');
        } else {
            meta_reviews_allowed_field.attr('hidden', true);
            jQuery('#assignment_form_assignment_num_metareviews_required').val('nil');
            jQuery('#assignment_form_assignment_num_metareviews_allowed').val('nil');
            jQuery('#assignment_form_assignment_metareview_true').val('false');
        }
    }


Issue 4 - "Has meta-review Limit" should fill 3 and 3 for the required and allowed number

Description: In the database (assignment table), the default number of meta-reviews allowed and required should be null. When an instructor checks 'Has meta-review Limit' box in the review strategy tab, the UI should fill in 3 and 3 as the required and allowed number.

Solution:

In _review_strategy.html.erb the function hasMetaReviewChanged() would take care of filling in the meta-review limits. This was not the issue, that function was working as intended. For some reason, both of the text boxes under Has meta-review Limit were set to change num_metareviews_allowed. This is shown here: 

  <tr id="assignment_review_topic_threshold_row">
    <td>
        <span id="meta_reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_metareviews_allowed.nil?%> class="form-inline">

        <%= label_tag('assignment_form[assignment][num_metareviews_allowed]', ' Set allowed number of meta-reviews per reviewer') %>
          <%if @assignment_form.assignment.num_metareviews_allowed && @assignment_form.assignment.num_metareviews_allowed >= 0 %>
                <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]', @assignment_form.assignment.num_metareviews_allowed ||= 3, size: 1, class: "form-control width-70", :style => 'margin-left :100px') %>
           <%else%>
          <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]',  3, size: 1, class: "form-control width-70", :style => 'margin-left :100px') %>
            <%end%>
          </br>
          <%= label_tag('assignment_form[assignment][num_metareviews_allowed]', 'Set required number of meta-reviews per reviewer') %>
          <%if @assignment_form.assignment.num_metareviews_allowed && @assignment_form.assignment.num_metareviews_allowed >= 0 %>
              <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]', @assignment_form.assignment.num_metareviews_allowed ||= 3, size: 1, class: "form-control width-70", :style => 'margin-left :90px') %>
        <%else%>
              <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]',  3, size: 1, class: "form-control width-70", :style => 'margin-left :90px') %>
        <%end%>
        </span>
    </td>
  </tr>

This was causing issues with the user's ability to save custom review limits. In order to remedy the issue, we simply changed the second text box under Has Meta-Review Limit to link to num_metareviews_required. This quickly resolved the issue and allowed users to now save their review limits. With the function (mentioned previously) working properly, the text boxes now display the correct value every time as long as the checkbox is checked. The fixed code is shown below: 

 <tr id="assignment_review_topic_threshold_row">
    <td>

        <span id="meta_reviews_allowed" <%= 'hidden' if not @metareview_true  %> class="form-inline">

        <%= label_tag('assignment_form[assignment][num_metareviews_allowed]', ' Set allowed number of meta-reviews per reviewer') %>
          <%if @assignment_form.assignment.num_metareviews_allowed && @assignment_form.assignment.num_metareviews_allowed >= 0 %>
                <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]', @assignment_form.assignment.num_metareviews_allowed ||= 3, size: 1, class: "form-control width-70", :style => 'margin-left :100px') %>
           <%else%>
          <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]',  3, size: 1, class: "form-control width-70", :style => 'margin-left :100px') %>
            <%end%>
          </br>
          <%= label_tag('assignment_form[assignment][num_metareviews_required]', 'Set required number of meta-reviews per reviewer') %>
          <%if @assignment_form.assignment.num_metareviews_required && @assignment_form.assignment.num_metareviews_required >= 0 %>
              <%= text_field_tag('assignment_form[assignment][num_metareviews_required]', @assignment_form.assignment.num_metareviews_required ||= 3, size: 1, class: "form-control width-70", :style => 'margin-left :90px') %>
        <%else%>
              <%= text_field_tag('assignment_form[assignment][num_metareviews_required]',  3, size: 1, class: "form-control width-70", :style => 'margin-left :90px') %>
        <%end%>
        </span>
    </td>
  </tr>

Testing

Usernames and Password for Expertiza

  1. username: instructor6 , password: password
  2. username: student10, password: password
  3. username: student18, password: password
  4. username: student20, password: password

Perform a Meta-Review as Student (Manual)

  1. Login to Expertiza as student
  2. Select an assignment
  3. Select "Others work"
  4. Click on the button "Request a new meta review" and a new review will be dynamically assigned for metareview.
  5. Complete review and click save metareview upon completion

Add Meta-Review option as Instructor (Manual)

  1. Login to Expertiza as instructor
  2. From Manage>Assignments, click on the Edit Assignment button for the appropriate assignment.
  3. On the Due Date tab, the instructor can select if they want to have meta-reviews done for the assignment, by clicking on the “Use meta-review deadline” checkbox and setting a meat-review deadline below.
  4. Under the “Review Strategy” tab of edit assignment, instructor can set Allowed and Required number of meta-reviews per reviewer by checking the "Has meta-review Limit?" checkbox

Testing Issue 1 - "No Begin Link"

  1. Login as instructor
  2. Go to Manage --> Assignments
  3. Create a new assignment (Blue plus symbol).
  4. Edit new Assignment
    1. Enable meta-review
    2. Set due dates for assignment submission (before current time), review (before current time), and meta-review (after current time).
    3. Put rubrics as needed
  5. Add students to assignment (ex. student10, student18, student20)
  6. Logout of instructor and login as students
  7. Go to assignment and try to make a meta-review. (You should find that you are unable to do a metareview)
  8. Login as instructor and change due dates of assignment submission (after current time) and review (after current time).
  9. Submit an assignment for all students and do reviews as each student.
  10. Do a meta-review (You should see the "Begin" link as expected).

Testing Issue 2 - "Incorrect value for "Number of meta-reviews left" / Disappearing button to request new meta-review"

  1. Login as instructor
  2. Go to Manage --> Assignments
  3. Create a new assignment (Blue plus symbol).
  4. Edit new Assignment
    1. Enable meta-review
    2. Set due dates for assignment submission (before current time), review (before current time), and meta-review (after current time).
    3. Put rubrics as needed
  5. Add students to assignment (ex. student10, student18, student20)
  6. Logout of instructor and login as students
  7. Go to assignment and try to make a meta-review. (You should find that you are unable to do a metareview)
  8. Login as instructor and change due dates of assignment submission (after current time) and review (after current time).
  9. Submit an assignment for all students and do reviews as each student.
  10. Do a meta-review (You should see the value of "Number of Meta-Reviews Left" to decrement).

Testing Issue 3 - "Has Meta-Review Limit?" and "Set Allowed/Required number of meta-reviews per Reviewer" shown at incorrect times

  1. Login as instructor
  2. Go to Manage --> Assignments
  3. Find "SLOAN Demo" assignment and click "Edit" icon
  4. Go to Due Dates tab (You will notice that "meta-review due dates" checkbox is no longer there
  5. Go to Review Strategy Tab
  6. Enable/Disable "Has meta-review" checkbox (You will see that the allowed/required values will only be visible if the checkbox is clicked)

Testing Issue 4 - "Has meta-review Limit" should fill 3 and 3 for the required and allowed number

  1. Login as instructor
  2. Go to Manage --> Assignments
  3. Find "SLOAN Demo" assignment and click "Edit" icon
  4. Go to review strategy tab
  5. Enable "Has meta-review" checkbox (if not already checked) and set allowed/required values to 5/5.
  6. Click "Save"
  7. Disable "Has meta-review" checkbox
  8. Click "Save"
  9. Enable "Has meta-review" checkbox (Values for allowed/required should be 3/3)

Automated Tests

  • Due to scope of this project and the need to have a automated test for a functionality that has many changes of breakage, we will be borrowing testing code done by E2132 - Add test cases to review_mapping_helper.rb (this code is also being used by E2168. Testing - Reputations).
  • To use the automated tests for Meta-Reviewing, please follow the documentation provided by E2132 regarding Assignments.
Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2021_-_E2156._Issues_related_to_meta-reviewing&oldid=142720"