CSC/ECE 517 Fall 2018 E1854 Improve self-review, Link peer review & self-review to derive grades

E1854. Improve self-review, Link peer review & self-review to derive grades

This page provides a description of the Expertiza based Semester project.

About Expertiza

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Statement

Expertiza has the ability to require self-reviews on projects, self-review is basically asking each team member in the group to evaluate their project on the same criteria as peer review, these self-review scores do not have much functionality currently. This project seeks to improve upon the self-review system, and extend its functionality. The main points of this project are as follows:

• Create a method to calculate a derived score (composite score) between self and peer-reviews. The closer the self-review is to the peer-review, the greater the score. Display this score on the "View Scores" page.
• Make sure that the peer-reviews should not be visible before self-review completion. Display the correct review scores accordingly.
• Display the self-review scores in the "View Scores" and heat map of peer-reviews, denoting that they are a different type of review.

Application Flow

The instructor has to login and create an assignment by checking allow self-review option. The instructor has to add the topic and assign students to the topic. Now, the student has to login and submit the assignment. The self-review link is visible in the page. The student should give the self-review. After giving the self-review, the student should be able to see the peer review, self-review and the composite score.\

Use case Diagram

Actors:

   Instructor: This actor is responsible for creating assignments and adding students to the assignment. 
   Student: This actor is responsible for submitting, self-reviewing and viewing the scores. 

All the other use cases are implemented except “View Scores with self-review column and composite score”

Use Case: View score with self-review column and composite score

Preconditions:

  1.The Instructor creates Assignments and checks the Allow Self Review check box in the Assignment
  2.The Student should submit the assignment and self-review.
  3.The other students should submit the reviews of the work submitted.

Student Sequence:

  The student should login.
  The student should browse and upload the assignment.
  The student should submit the assignment.
  The student should submit the self-review.
  The student should choose your scores to view the score.
  The student should be able to see the peer-review score with the self-review scores and composite score.

Post Conditions:

  The self-review and composite score will be visible on the page.

Method to calculate Composite Score

The composite score is calculated using both self-review score and peer-review score (both are graded over 5 points for each criteria). The composite score is calculated as follows:

  composite score = [10 - abs(self-review score - peer review score)] * peer review score / 10

By using this formula for calculating composite score we are discouraging students from either exaggerating or underrating their performance. The composite score model assigns more weight to the peer review score since the peer review score being used is the average of all peer review scores assigned and hence tends to be more appropriate than a single self-review.

Also this composite score formula gives the score on a scale which is identical to the scale of the self and peer review score. Thus this formula is not dependent on the scale of the self-review and peer-review score as it will also return the composite score on the same scale.

For example: If someone gets an average peer review score of 3.5/5 and he gives 3.5/5 (self-review score) while reviewing his project, then he will get a composite score of 3.5/5. And instead if the student reviews his project for 2/5 then the student will get a composite score of 2.975/5.

Implementation

As stated above the ability to do self-reviews has already been implemented. This can be seen from the image shown below. The instructor has this ability to enable self-reviews .

In our project, we were successful in displaying both the self-review score and composite review score on the view scores page. Shown below is an example of the output in the student profile. As we can see the self-review icon has a blue background to indicate that it is different from other peer reviews.

The alternate view was also modified to accommodate a pictorial representation of the self-review scores. The modified alternate view scores page is shown below.

The same was achieved in the instructor profile too. In the current implementation of expertiza when self-review was not assigned, the instructor would view the scores of the first student in each team (it wouldn't matter much because without self-review all students in a team would have the same scores). But this would be a problem when self-review was enabled so we decided to display all the self-review scores of all students in the team for instructor view. As it can be seen below self-reviews of all students in the team (distinguished by their usernames) are indicated by blue background to differentiate them from peer review scores.

Test Plan

Test cases are as follows:

• Peer-reviews cannot be viewed before a self-review is completed.

This ensures that the student is not able to see his peer-review score before writing his self-review as the composite score takes into account the difference between the student peer-review and self-review score.

• Self-review scores are displayed with peer-review scores.

This enables the student and instructor to view all his scores in one place.

• The composite score must be properly calculated.

The composite score must be calculated so as to give students more points if their self-review score is closer to the peer review score.

• Composite score must be visible in both view scores page and Alternate view.

The scores must be displayed in both the views.

Files Changed

• app/controllers/grades_controller.rb
• app/models/assignment_participant.rb
• app/models/author_feedback_questionnaire.rb
• app/models/response_map.rb
• app/models/vm_question_response.rb
• app/models/review_questionnaire.rb
• app/models/teammate_review_questionnaire.rb
• app/views/grades/_participant.html.erb
• app/views/grades/_participant_charts.html.erb
• app/views/grades/_participant_title.html.erb
• app/models/vm_question_response_row.rb
• app/views/grades/view_team.html.erb
• spec/controllers/grades_controller_spec.rb
• spec/models/assignment_particpant_spec.rb

References

1. Expertiza on GitHub
2. Pull Request
3. Screencast video