CSC/ECE 517 Fall 2019 - E1984. Improve self-review Link peer review & self-review to derive grades

From Expertiza_Wiki
Jump to navigation Jump to search

E1984. Improve self-review Link peer review & self-review to derive grades

This page provides a description of the Expertiza-based Semester project.

Problem Statement and Overview

In Expertiza the Instructor can create assignments and check the 'Allow self-review' box for Students. When enabled, each member evaluates his/her submission with the same criteria as the peer review. Currently, these self-review scores are not used to compute the final scores for the assignment. The task here is to improve the self-review system pertaining to give it a weightage in the final scores for the submissions.

The objectives we achieved for this project are as follows:

  • Provide a formula that takes both the peer review score and self-review score into account for calculating the composite score.
  • The composite score should get higher as the self-review score gets closer to the peer-review score.
  • Make sure that the peer-review scores are not visible before the self-review submission.
  • Display the composite score on the “View Scores” page.
  • Display the self-review scores in the "View Scores" page and in the "Alternate" (heat-map) view of the peer-reviews, highlighting them as a different type of review.

The need for the self-review section is to know how well the students can judge their work and understand how to evaluate their flaws and rectify them. Also, if they can score themselves close to their peers, then it means that they have good self-assessment skills and they are being honest about what they think of their work. According to self assessment rules: students get more points as their self-review get closer to the scores given by their peer reviewers.

Code Changes

The following flowchart illustrates the proccess of how scores are retreived to implement the self-review score feature.

First, we start in the grades_controllers.rb file which contains all the logic for computing an assignment score.


We access the participant scores by calling the scores method defined in assignment_participant.rb. Here, the scores method returns scores that the participant has been given.


The actual computation for the participant scores is done in compute_assignment_score method located also in assignment_participant.rb.


Inside compute_assignment_score we call another method called get_assessments_for which is located in review_questionnare.rb.


The get_assessments_for method will call the reviews method located also in assignment_participant.rb.


The reviews method will get the scores by simply calling the get_assessments_for method located in response_map.rb.


Finally, all we have left is to apply the derived formula using the scores we just retreived and then show the result in the respective views. All this process is done in grades_controllers.rb, where we initially started.


The self-review scores are obtained by using the true parameter in all the methods calls (as shown in the diagram), whereas the peer-review scores are retreived similarly but omiting this parameter.


List of files changed:

  • app/controllers/grades_controller.rb
  • app/controllers/sign_up_sheet_controller.rb
  • app/models/assignment_participant.rb
  • app/models/author_feedback_questionnaire.rb
  • app/models/response_map.rb
  • app/models/review_questionnaire.rb
  • app/models/teammate_review_questionnaire.rb
  • app/models/vm_question_response.rb
  • app/models/vm_question_response_row.rb
  • app/views/assignments/edit.html.erb
  • app/views/assignments/edit/_general.html.erb
  • app/views/grades/_participant.html.erb
  • app/views/grades/_participant_charts.html.erb
  • app/views/grades/_participant_title.html.erb
  • app/views/grades/view_team.html.erb
  • app/views/reports/_self_review_report.html.erb
  • app/views/student_review/list.html.erb
  • config/routes.rb
  • db/schema.rb
  • spec/controllers/grades_controller_spec.rb
  • spec/controllers/sign_up_sheet_controller_spec.rb
  • spec/features/assignment_creation_spec.rb
  • spec/models/assignment_particpant_spec.rb


Next we show the most relevant improvements performed in this project. Feel free to review all the code changes in Github by following the next link: view file changes in pull request.

  • grades_controller.rb

  • assignment_participant.rb

  • response_map.rb

  • vm_question_response.rb

Use case

The following diagrams illustrate how the Self Assessment feture is meant to be used in Expertiza. Basically, we have two actors: on one side we have the Instructor which will create an assignment, add students to it and enable the self-review option. On the other side, students will be able not only to review their peers' work, but also will be required to review their own work in order to view their final scores, if the self-review option is enabled by the instructor.

  • Actors:
1. Instructor: This actor is responsible for creating assignments and adding students to the assignment.
2. Student: This actor is responsible for submitting, self-reviewing and viewing the scores.
  • Actions
1. Instructor: Create Assignment, Enable Self-Review.
2. Student: Login, Add work and Submit, Submit Self-Review, View Scores.

Current Process

Currently, Expertiza allows students to submit an assignment and provides a link to self-review their work. Once the self-reviewing is done, the self-review score gets stored in the database and it is not further used in calculating the overall score of the assignment. Thus, students can score themselves higher than what they should actually get when considering the peer-review scores. This helps students in no way. So, to make a productive use of the self-review score and to help students learn self-evaluating themselves, we will make changes in the review score calculation, explained in the section below.

New Process

We derived a formula that takes into account the self-review score and calculates a composite score, which will be the student's final score for a given assignment:

More information regarding the derivation of this formula can be obtain from the following document:

Notes for deriving the formula

Demo Video

You are welcomed to watch our 5-min demo video where we briefly explain and test the project functionality.

[Watch the video!]

Test Plan

Manual Testing

If you wish to try the self-review feature in Expertiza as shown in the demo, then follow the next steps:

1. Go to the link of the deployed application: http://152.46.16.39:8080/
2. Log in as instructor. Use the credentials: instructor6, password.
3. Go to Manage->Assigments and clik the '+' icon on the top right corner to create an new assignment.
3.1. In General tab: write the assignment title, select any course and type any submission directory.
3.2. In Rubrics tab: uncheck the tab Review rubric varies by round? and select Design Doc for the Review in the Questionnaire drop-down menu. Select any type of Author Feedback.
3.3. In Review Strategy tab: go to the bottom and check the Allow Self Reviews? box.
3.4. In Due dates tab: for Submission deadline, select a date and time greater than the current date and time when creating this assignment. For Review deadline select a date and time grater than the submission deadline.
3.5. Click on Create to save and create the assignment.
4. Go again to Manage->Assigments, find the assignment you created and click on Add participants (user icon) to add some students.
4.1. Add student3000 and student4000 to the assignment.
5. Sign in as student3000. Log out from the instructor6 account or access to the application from a different browser (incognito window or a different explorer). Use credentials: student3000, password.
6. Go to test assignment and click on work. Submit any link or file.
7. Click on Review my own work button and then the begin link to start the self review. Grade yourself accordingly and make sure to submit the self review when finished.
8. Repeat steps 5 to 7 for student4000. Use credentials student4000, password.
9. Log in as instructor again. Go and edit the assignment you created. In the Due dates tab, choose a date and time in the past. e.g. yesterday. This will enable peer reviews.
10. Sign in again as student4000 Go back to the assignment and click on Others work. Click on Request a new submission to review and review the student3000's work. Don't forget to submit the review when done.
11. Sign in again as student3000. Go to the assignment and click on Your Scores. Here you can see the final average peer review score, which is computed by using the proposed formula in this project. In the scores table, you can see your peer review score and self review score (cyan color) side by side.

Peer-review unlock feature

Note that in step 7 of the Manual Testing section, we are immediatly submitting our self review after our work has been submited. This is the ideal workflow, however imagine the situation where a student forgot to submit his/her self-review questionnarie. Then by requirement of this project, and to make sure that every student submit their self review, the Your scores link will be disabled for the assignment. So students will only be able to review their scores after submitting their self review.

Rspec Testing

The following Test cases were developed to ensure a correct implementation:

  • Peer-reviews cannot be viewed before a self-review is completed.
  • Peer-reviews can be viewed immediately when self-reviews are not required.
  • The composite scores are saved to the database.
  • Self-review scores are displayed with peer-review scores.
  • Student can only see their own self-review.

We overloaded the get_assessments_for method with a second parameter which decides wheather the peer-review scores are retreived (false) or only the student self-review (true). Modifications were made accordingly in the grades_controller_spec.rb and assignment_participant_spec.rb files.

Future Work

The work done in this project can be built upon by future groups. Several areas can be improved upon and new functionality can be added.

  • Refactoring of the ResponseMap features: accessing scores from reviews is difficult, and requires several method calls.
  • Implementing multiple score approaches. A small addition to the instructor UI may allow them to select from different self-review formulas.
  • Add scores from team evaluation (Your team -> Review) to the formula. True peer assesment involves the use of peer-review scores within the team, not only peer-review scores from other teams as currently implemented.

Team and Resources

Team_AABD:

References

Project

Peer Assessment Theory

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Fall_2019_-_E1984._Improve_self-review_Link_peer_review_%26_self-review_to_derive_grades&oldid=131218"