CSC/ECE 517 Spring 2020 - E2015. Conflict notification
Abstract
Expertiza is an open-source web application implemented using Ruby on Rails. It is used for management of courses and the assignments for respective courses, by the faculties and the students. Students can form teams in Expertiza to work on an assignment in a group. A student team can submit their work through multiple means such as file uploads and embedded links. Submissions made on Expertiza can be peer-reviewed by students within the course based on a rubric. Results of the peer review can be accessed by instructors and respective student teams. During the peer evaluation, Expertiza sends appropriate email notifications to the instructor and also provides a complete statistical report of the evaluation.
Problem Statement
In the current implementation, during the peer review phase of an assignment, email notifications are sent out to the instructor whenever a submitted review score differs “significantly” from the average score of other submitted reviews for that submission. The threshold to trigger a notification is specified in the “Notification limit” on the Rubrics tab of assignment creation. The email that gets sent to instructors contains three links: a link the the conflicting review, a link the the summary page of grades for the reviewee, and a link to edit the assignment. Although the links sent in the email are helpful for reference, the process of examining a conflicting review could be made more efficient if we provide an easy to understand report of the conflict. The information sent in the email should be updated to contain a link to a report page which should contain more details about the conflict. In addition to creating a new conflict report page, we also want to change the algorithm to detect and trigger a notification if there is a significant difference between a submitted review and any previous submitted review for that assignment. Rather than compare the review score to the average of all review scores, an email should be sent whenever two review scores differ significantly for the same submission.
Existing Implementation
On submitting a new review for a particular artifact, the submitted score is compared to the average score of the previously submitted reviews. If the difference between the average score and the newly submitted score is more than the notification threshold specified for the assignment an email is sent out to the instructor. The email sent to the instructor contains the names of the reviewer and reviewee and a link to the conflicting review. The flowchart below delineates the entire process:
Problems
The existing implementation has the following flaws:
No dedicated view for analyzing the conflict
The current implementation does not provide any dedicate view or report containing a detailed analysis of the conflict.
Incorrect email contents
The current implementation uses hardcoded URLs mentioned in models/response.rb file. Being hardcoded, these link would not work in on other servers where Expertiza is running. For example, the links won't be valid if the setup was done on the localhost.
The email sent out to the instructor contains the following details:
- Name of both the students - reviewer and reviewee
- Newly submitted conflicting score
- Link to view the conflicting review
- Link to view the summary of all reviews
- Link to change the notification threshold
Design
Proposed Solution
What we need to modify:
- Related to Email:
- App/mailers/mailer.rb in method notify_grade_conflict_message to make the link to the conflict report available in the email.
- App/models/response.rb in method notify_instructor_on_difference to add a link to the conflict report that gets sent to the mailer.
- App/views/mailer/notify_grade_conflict_message.html.erb to add to the message a link to view the conflict report.
- Related to the conflict report view:
- App/views/reports/_searchbox.html.erb to add the option to view a conflict report.
- App/views/reports/response_report.html.haml to render the conflict report partial if the option to view a conflict report was selected.
What we need to create:
- Related to the conflict report view:
- App/views/reports/_conflict_report.html.erb to make the view for a conflict report
The conflict report view should:
- For each team that had a submission with a review that was in conflict:
- List the team name and team members
- Provide a graphical view of the statistics displayed in the report
- For each round where there was a review in conflict:
- List the students who's review caused a conflict and the score they gave
- List the threshold for conflict review score, max review score, average review score, and standard deviation.
Why we chose this approach?
Use Case Diagram
Actors participating:
- Student: In this use case, fills out the reviews for different projects submitted by the peers.
- TA/Instructor: In this use case, goes over the conflicting review submitted by the students for the same project.
Testing
Test Plan
We intend to break down our test plan into 3 phases:
1. Run existing RSpec Tests, make sure those pass.
2. Create new RSpec Test to check if the functionalities still work, and if the tests still pass.
The new RSpec Test will be used to check if the emails are still triggered.
Note:
3. Manually test the application to see if the code breaks anywhere after the new code is written.
Manual Testing - Intended Steps:
Say there is a team of 2 students, Student_1 and Student_2. This team has to review a certain project.
Student_1 logs into Expertiza, visits the project that (s)he is assigned to review. The overall rating given by Student_1 for the project is 100%.
Now, Student_2 logs into Expertiza, visits the same project that Student_1 was assigned to review. The overall rating given by Student_2 for the project is 0%.
The rating difference for the same project by both the students is significantly high, at 100%.
Since there exists a significant contrast in the ratings, an email is triggered to notify Student_2 about the ratings (s)he gave.
Useful Links
Github: https://github.com/sid189/expertiza/tree/beta
Pull Request: https://github.com/expertiza/expertiza/pull/1714
Team Information
Project Mentor:
Pratik Abhyankar
Project Members:
Sahil Papalkar
Sahil Mehta
Siddharth Deshpande
Carl Klier