CSC/ECE 517 Fall 2016 E1696 Improve Self-Review

IMPROVE SELF-REVIEW

Introduction

Expertiza is an online system that is used by students to view/submit assignments and review others' work. Expertiza also provides tools to visualize the scores and gauge the improvements made during the course semester. It also facilitates and monitors team projects. It is targeted at educational and non-profit organizations. The project is funded by the National Software Foundation (NSF), NCSU Learning in a Technology-Rich Environment (LITRE) program, the NCSU Faculty Center for Teaching and Learning, the NCSU STEM Initiative, and the Center for Advanced Computing and Communication.

Expertiza is an open-source project with the source code available as a public repository on GitHub. It is developed using Ruby on Rails and is increasingly becoming robust thanks to the innumerable bugs being fixed by the community. The project has a micro-blog on SourceForge where the developer community report bugs and document updates.

Task Description

Display self-review scores juxtaposed with peer-review scores in the regular “View Scores” page and the alternate (heat-map) view.

Combine self-review and peer-review scores to derive a composite score, based on closeness to the peer-review.

Author(s) should be required to submit their self-evaluation(s) before seeing the results of their peer evaluations.

Definitions (in context of the project)

• Peer-review : Review of your work done by others in the class.
• Self-review : Review of your work done by you. This is unique to each individual (and not each team).
• Author : The one that writes the review. Could be a peer or yourself.
• Instructor : The admin of the system, ideally, the professor and the TAs.
• Instructor-score : Score assigned by the instructor or his TAs.

Edge Cases and Constraints

1. One can view the peer-reviews done only upon the completion of his/her self-review.
2. The final score of the self-review depends on the agreement of the self-review scores with peer-review scores.
3. While comparing the self-review scores with peer-review scores, more weight must be given to the instructor-score.
4. While considering the reviews done by others in the class, the average must be used.
5. There is no feedback for self-reviews.
6. Self-reviews are done individually and not as a team.
7. The self-review scores of each member of a team does not affect the self-review scores of the other members in the team.

Approach

Student has the task of performing review of others' work as well as self review. Every team gets their review from peers and at present the composite score is the average of all the review scores that they have received. Added to this score, the self review score is also calculated for each team member, based on its closeness to the peer reviewed score. To implement this, the student needs to review his own work based on the same rubrics used for peer review and only after completion of the self review, their peer review scores will be visible. The composite score is calculated by the sum of the peer review average score and 100 minus the difference between self and peer review score. The self review score is displayed adjacent to the peer review score. As the composite score is based on the individual self reviews, they vary for each member of the team unlike the peer review score which is the same.

MVC Pattern

Expertiza is developed using the nuances of the MVC Design Architecture where Models, Views and Controllers are coupled to a very low level. The model manages all the data and the logic related to the data. The controller accepts data from the Models and feeds the Views with a favorable form of the data. The view presents the data to the user.

Observer Pattern

The advantage of using the Observer Pattern is that the object being observed need not worry about any changes in its state. An Observer Class is added whose objects will look into the changes in the state of the Object being observed.

Here, the observers are the reviewers and the assignment is the object being observed. In case of Self-Review, an Observer can be used to tell the system that the reviewer is now open to reviewing the works of his peers.

Implementation

Current Implementation

Self-Review is partially functional in the current build of Expertiza. The file _self_review.html.erb in submitted_content view provides the links to 'View', 'Update', 'Edit' and 'Begin' self-reviews. Then the control transfers control to view() method in response controller. However, the self-review scores are not put into any use and are not displayed to the user.

Juxtaposing Peer and Self-Review Scores

The system currently uses view_my_score() method in grades_controller to display the scores in peer-review. The same set of statements used in view_my_score() can also be used to display the self-review score.

The purpose behind doing this is to facilitate an easy comparison between the ways in which an assignment is reviewed - by peers and by self. This will also provide a scope for improvement in future assignments when one knows the differences between the expectations of the peers and the deliverables of the project.

Combine Self and Peer-Reviews into a Composite Score

For starters, we are going to use a simple formula to determine the Self-Review score.

Composite Score = 100 - ( | Self-Review Score - Average Peer-Review Score | )

The motive behind doing this is to convert a mostly qualitative resource into a quantitative resource. Though the questions in the review questionnaire contain a score from 1 to 5, they do not make any sense separately. When pitted against other peer-reviews and the instructor-review, they represent an honest metric of how satisfied the members of the project team are.

Complete Self-Review before checking Peer-Review Scores

The inclusion of a check_self_review_status() method in grades controller can be used to determine the presence of an already completed self-review. It returns false in the absence of an already completed self-review, otherwise it returns true.

Testing

1. Juxtaposing Peer and Self-Review Scores - From the Assignment Home Page, click on 'Your Scores' to view both Average Peer and Self-Review Scores.
2. Combine Self and Peer-Reviews into a Composite Score - From the Assignment Home Page, click on 'Your Scores' to view the Composite Score assigned to you based on the closeness of your Self-Review score with the Average Peer-Review Score.
3. Complete Self-Review before checking Peer-Review Scores - From the Assignment Home Page, click on 'Others' Work' to try to submit Peer-Reviews for the projects submitted by others. However, this should be forbidden unless you have submitted a Self-Review of your project.

Scope for Future Work

• The words used in the self-reviews can be used to judge the seriousness of the reviews.
• The changes in the scores of the self-reviews can be used to judge whether teams have worked to their own expectations.
• In order to establish a common standard in the reviewing system, the absolute change in the review scores can be used to judge an individual.