CSC/ECE 517 Fall 2017/E17A4 Allow calibration to be part of an assignment

This wiki page describes the changes made according to the specification of E17A4 - final project for CSC/ECE (517)Fall 2017.

Introduction

Calibration is the term used for rectifying, or checking or determining something. In the context of this project, a 'Calibrated review' is the one which is performed by student and instructor as well, so that student can check on his work by comparing with instructors' response for the same assignment. If the student’s review “resembles” the staff-member’s review, then the student is presumed to be a competent reviewer. Since calibrated reviews are just for checking on our progress, they are not graded. They are just meant for practice, so that peer review (which are performed later) can be performed well which is graded.

Problem Statement

In order to gain practice for giving a reviews , the Student should be able to give calibration reviews before providing peer reviews. Thus, the Instructor should be able to enable calibration for every new assignment. By this way, the student can improve peer reviewing by comparing his/her results with expert reviews for the same assignment.

Limitation

At present, the instructor is able to calibrate the students’ peer review abilities on Expertiza by creating only a separate assignment for this purpose. In this separate assignment that the instructor creates, the student and the instructor will simultaneously do peer review for the project. After submission of the reviews, the student's and instructor's reviews are compared and a calibration report is generated for the student to view and improve their peer reviewing skills.

Unfortunately, for students to practice peer reviewing, a separate assignment has to be created every time. In order to facilitate sufficient practice for peer review, this is not a convenient method.

Proposed System

The objective of this project is to enable calibration to be one part of the normal assignment. The instructor should be able to turn on this feature if necessary and provide the corresponding calibration deadline. The student should be pre-assigned with sample assignments. Once the student submits the calibration reviews, the calibration results comparing the student and expert review are provided to the student. When the actual review period starts, the student will review other assignments, and submit peer reviews. So the total number of reviews calculated should not count calibration reviews.

Tasks are listed as below:

1. To integrate calibration feature into a normal assignment, a new due date, that is calibration due date needs to be added in deadline_types table. So a migration needs to be written for implementing this.

2. When instructor checks “Calibration for training?” and saves the assignment, in “Due dates” tab, a row should appear to deal with calibration due date. The other content of this row is just the same as other due dates.

3. In student_task/list page, current stage column should display calibration when the assignment is in calibration period.

4. In student_review/list page, in Other's work, calibration reviews need to be excluded from the peer reviews because calibration reviews should not count towards the total number of peer reviews done or towards the peer review grading. For this purpose, they need to be separated.

To achieve this goal, filter the reviews with timestamp earlier than submission due date, which means reviews are done in the calibration period. The wording and numbering on this view should be changed to: Calibrated review 1,2,3 to represent review done in calibration period; Review 1, 2, 3 to represent reviews done in the normal review period.

System Overview

The current system has the following actors:

a) Instructors:Users who have administrative privileges to create/edit assignments and impersonate students

b) Students: Users who submit assignments and peer reviews

In the current system, the calibration functionality works correctly. In the assignments' edit page, once the instructor enables “Calibration for training?” and saves the assignment, an extra tab "Calibration" appears. After this, the instructor needs to add several sample assignments (right now instructor needs to impersonate several students to submit sample assignments) and do expert review beforehand. This is the existing functionality.

In the proposed system, the entire flow of the assignment submission done by the student, changes. First is the calibration review of sample assignment, then normal assignment submission followed by rounds of peer reviews for the submitted assignments.

Flow:

1. As an instructor, a new Assignment "Project 4" was created.

2. In assignment edit mode, when calibration is not enabled, the "Due date" tab appears as shown below:

3. Once the “Calibration for training?” (see the below figure) is enabled, in addition to the “Calibration” tab, a new row for calibration deadline entry appears under the “Due dates” tab (see the figure below) before submission row. The due date for the calibration review is entered and the assignment is saved.

4. 2 students are assigned as participants (student1, student2) and 1 student (student3) as reviewer for the assignment "Program 4". Then, the Instructor impersonated one of the participant students, say student1, and submitted a sample assignment that is to be used for calibration review.

5. The review strategy was set to "Instructor-selected" for calibration review. Then, the instructor assigned student3 as a reviewer for this sample assignment submitted by the expert and also, submitted an expert review for this sample submission.

6. Now student3 is impersonated. The "current stage" of the assignment "Program 4" shows as "calibration" on the student_task page as shown below:

7. As student3, on getting into "Program 4", the "Other's work" link is enabled (as shown below). This is enabled even before submission period as we should allow students to enter calibration review beforehand. If calibration is not enabled for the assignment, the others' work link will be enabled only after submission period.

8. In "Other's work", one calibration review for the sample assignment is displayed for which the reviewer can submit the calibration review.

9. Once the calibration review is submitted, the "show calibration results" link is displayed which shows the scores against the expert review. Show calibration results gets enabled when the expert and the student submits the review.

10. Once the calibration due date elapses, the assignment enters the "submission" period. During this period, student submissions are made. The review strategy is set as "auto-selected" and number of reviews allowed are set to 2.

11. When impersonated as student3, in other's work, both the calibration and assigned peer reviews are displayed as shown below

The reviewer can now request new submission for peer review.

Also, the number of reviews required and allowed considers only the peer reviews and not the calibration reviews. Since only one peer review is present, the "request new submission for review" button is enabled. Once a new review is requested since there are 2 peer reviews, the "request new submission for review" button is not displayed. This shows that only the peer reviews are considered for the total number of reviews allowed and required.

After completion of the calibration reviews (before the due date), the student should submit the assignment on/before the submission deadline, which is the existing functionality. Since the students have had enough experience during the calibration period on peer reviewing, the next task will be carried over effectively.

Peer reviewing is the concept of review your peer’s work. This concept enables the students to understand the solution to a problem from multiple perspectives. This will also embed new ideas and the skill of constructive criticism in the student. Hence, the student will provide peer reviews after the submission of their work.

System Use Cases

Use Case Diagram :

1. Add/Edit Assignment: Enable Calibration

Use Case Id: 1

Use Case Description: Instructor can select the option - "Calibration for training?"

Actors: Instructor

Pre Conditions: The assignment is present in the assignments list

Post Conditions: Instructor can navigate to the edit assignment page where the enable calibration option is present

2. Edit Assignment: Set Due Dates for Calibration and Peer Reviews

Use Case Id: 2

Use Case Description: Instructor can set due date for calibration under due dates tab

Actors: Instructor

Pre Conditions: Once calibration option is enabled, a separate row for calibration should be created

Post Conditions: The instructor should be able to see the new date set for calibration.

Special cases: Calibration deadline should come before the submission deadline.

3. View the current stage in assignments as "Calibration" during the calibration period

Use Case Id: 3

Use Case Description: In assignments home page, the assignment should now be in the calibration period.

Actors: Student

4. View the pre-assigned sample assignments in "other's work" tab which can be opened even before submitting the assignment

Use Case Id: 4

Use Case Description: Student should be able to see the list of assignments in others' work tab for reviewing

Actors: Student

Pre Conditions: Sample assignments are provided by the instructor for the student to review and the tab "other's work" is enabled.

5. Submit 'n' pre-assigned Calibration Reviews

Use Case Id: 5

Use Case Description: Student can now review those assignments and submit them

Actors: Student

Pre Conditions: Sample assignments are provided by the instructor for the student to review

Post Conditions: Student can now see and edit the newly submitted review and also see the instructor's review for the assignment

6. View the 'Calibration Result' after submitting

Use Case Id: 6

Use Case Description: Student can see the comparison between student's and instructor's reviews for the pre-assignment

Actors: Student

Pre Conditions: Student had already submitted a calibration review

Post Conditions: Student can now view the comparison results

7. Submit the assignment before the due date

Use Case Id: 7

Use Case Description: Student should submit the assignment before the submission deadline

Actors: Student

Pre Conditions: An assignment exists and the calibration review is submitted

Post Conditions: Student can successfully submit the assignment before the deadline

8. Request for a Peer Review

Use Case Id: 8

Use Case Description: After the student submits the assignment, the student can request to do peer review of other student's submission

Actors: Student

Pre Conditions: peer review period has started and other students' submissions are available for reviewing

Post Conditions: Student should be assigned with a project for reviewing.

9. Submit the Peer Review and request for next one after minimum number of peer reviews are completed

Use Case Id: 9

Use Case Description: After the student requests for a peer review, he can review the assignment and submit it, After minimum number of reviews are commpleted, he/she can request for more assignments to be reviewed

Actors: Student

Pre Conditions: Student received an assignment for peer reviewing

Post Conditions: View and edit options are available for the submitted review until the peer review period deadline

All the actions performed by the student can be done by the instructor by impersonating a student

Design

1. MVC

The project is implemented in Ruby on Rails that uses MVC architecture. It separates an application’s data model, user interface, and control logic into three distinct components (model, view and controller, respectively).

2. Dry Principle

The existing functionalities in Expertiza are reused, thus avoiding code duplication. Whenever possible, code modification based on the existing classes, controllers, or tables will be done instead of creating a new one. Example: Currently, calibrated reviews are handled separately and peer reviews are handled separately. In our enhancement, we need to merge both the reviews. So instead of duplicating the code in calibrated reviews, the same code is used to handle both the reviews.

No design patterns are used because existing methods are modified and no new methods are created. So the framework remains the same.

Files Modified

The following files were modified to implement the feature:

migration file:

 class InsertCalibrationDeadline < ActiveRecord::Migration
  def self.up
    execute "INSERT INTO `deadline_types` VALUES (12,'calibration');"
  end
 end

app/controllers/student_review_controller.rb

   @first_submission_due_date = AssignmentDueDate.where(parent_id: @assignment.id, deadline_type_id: '1').first.due_at
   @peer_reviews = []
   # If assignment is calibrated, all reviews performed before calibration due date are assigned to calibration reviews
   if @assignment.is_calibrated?
     @calibration_reviews = []
     calibration_due_date = AssignmentDueDate.where(parent_id: @assignment.id, deadline_type_id: '12').last.due_at
     if Time.now <= calibration_due_date
       @calibration_reviews = ReviewResponseMap.where(reviewer_id: @participant.id)
     else
       # When current time exceeds calibration due date, the reviews are segregated based on the time of creation
       @review_mappings = ReviewResponseMap.where(reviewer_id: @participant.id)
       @review_mappings.each do |review_map|
         if review_map.created_at <= calibration_due_date
           @calibration_reviews << review_map
         else
           @peer_reviews << review_map
         end
       end
     end
     @calibration_reviews = @calibration_reviews.sort_by {|mapping| mapping.id % 5 }
   else
     # If assignment is not calibrated, all reviews are considered as peer erviews
     @peer_reviews = ReviewResponseMap.where(reviewer_id: @participant.id)
   end

views/student_review/list.html.erb

 <% if @assignment.is_calibrated? && !@calibration_reviews.nil? %>
    <%= render :partial => 'responses', :locals => {:mappings => @calibration_reviews, :title => 'Calibrated Review'} %>
 <% end %>
 <!-- Only when the first submission due date elapses, the peer reviews will start displaying in others work -->
 <% if Time.now >= @first_submission_due_date && (check_reviewable_topics(@assignment) or @assignment.get_current_stage(@topic_id) == "Finished") %>
    <% if @assignment.num_reviews_allowed.nil? || @assignment.num_reviews_allowed == -1%>
    <% else %>
        <h4>Number of reviews allowed: <%= @assignment.num_reviews_allowed %></h4>
    <% end %>
    <h4>You are required to do <%= @assignment.num_reviews_required %> reviews</h4>
    <% if !@peer_reviews.nil? %>
    <!-- Render response partial for peer reviews and title of each review as Review -->
        <%= render :partial => 'responses', :locals => {:mappings => @peer_reviews, :title => 'Review'} %>
    <% end %>
    <h4>You are required to do <%= @assignment.num_reviews_required %> reviews</h4>
    <% if !@peer_reviews.nil? %>
    <!-- Render response partial for peer reviews and title of each review as Review -->
        <%= render :partial => 'responses', :locals => {:mappings => @peer_reviews, :title => 'Review'} %>
    <% end %>

views/assignments/edit/_due_dates.html.erb

       <% if @assignment_form.assignment.is_calibrated?  %>
         addDueDateTableElement(<%= @due_date_nameurl_notempty==nil ? false:@due_date_nameurl_notempty %>,'calibration', 0,<%= due_date(@assignment_form.assignment, 'calibration').to_json.html_safe %>);
       <% end %>

app/models/deadline_right.rb

     'calibration' => {
      'submission_allowed' => OK,
      'can_review' => OK,
      'review_of_review_allowed' => NO
     }

app/controllers/assignments_controller.rb

   # Delete due date for Calibration if unchecked
    unless @assignment_form.assignment.is_calibrated?
      DueDate.where(parent_id: params[:id], deadline_type_id: 12).destroy_all
    end

app/views/student_review/_responses.html.erb

                      <!-- Update link is disabled for all calibrated reviews -->    
                      <% elsif title!='Calibrated Review' %> 
                        <td><%= link_to "Update", {:controller => 'response', :action => 'new', :id => map.map_id} %></td>
                      <% end %>

            <!-- Show calibration results if the review has title as calibrated review -->
            <% if @assignment.is_calibrated == true and !calibration_response_map.nil? and !calibration_response_map.response.nil? and !map.response.empty? and map.response[0].is_submitted == true and title=='Calibrated Review'%>
              <td>  </td>
              <td><%= link_to "Show calibration results", {controller: 'response', action: 'show_calibration_results_for_student', calibration_response_map_id: calibration_response_map.id, review_response_map_id: map.id} %></td>
            <% end %>

Test Plan

In GUI

To test this feature in GUI, perform the following steps:

1. Login as instructor in Expertiza.

2. Create a new Assignment.

3. Click edit assignment and in the general tab, select the option - 'Calibration for training?' and click 'save'

4. Once that option is enabled, a new tab opens in the name 'Calibration'. Now, under Due dates tab, a new row would have been created for 'calibration'.

5. Enter the Due Date for the calibration round (Enter a date which is before the submission due date) and save the assignment.

7. Go to the calibration tab and submit one expert review for the sample assignment.

8. Impersonate as student and under assignments home page, the assignment for which calibration was enabled will now have the current stage as 'calibration'

9. Now select that assignment and go to other's work link, which should be enabled.

10. One review will now appear as calibrated review. Begin the review and submit.

11. Now on the same page i.e other's work , you will see a new link: "Show calibration results" for the respective calibration review.

12. Once the calibration period is over and submission period starts, the students can now submit the actual assignment. Post submission due date, the peer review period will start.

13. When the peer review period begins, in others' work, the student will now see two sections: Calibrated review and Review.

14. Here, the total number of reviews count only the peer reviews.

Automated test

Automated tests are written in calibration_spec.rb and staggered_deadline_spec.rb file under specs/features folder.

The RSpec tests cover the following scenarios:

1. If calibration is enabled, check if a new row is created in the deadlines table with the deadline type as Calibration.

2. During the calibration period, check if the current stage shows as calibration

Team Contact

MEMBERS

Harika Malapaka: hsmalapa@ncsu.edu

Vidhyalakshimi Sreenivasan: vsreeni@ncsu.edu

Valli Annamalai: vannama@ncsu.edu

Arjun Sharma: asharm33@ncsu.edu

MENTOR

Zhewei Hu: zhu6@ncsu.edu