CSC/ECE 517 Spring 2020 - E2015. Conflict notification

Abstract

Expertiza is an open-source web application implemented using Ruby on Rails. It is used for management of courses and the assignments for respective courses, by the faculties and the students. Students can form teams in Expertiza to work on an assignment in a group. A student team can submit their work through multiple means such as file uploads and embedded links. Submissions made on Expertiza can be peer-reviewed by students within the course based on a rubric. Results of the peer review can be accessed by instructors and respective student teams. During the peer evaluation, Expertiza sends appropriate email notifications to the instructor and also provides a complete statistical report of the evaluation.

Problem Statement

In the current implementation, during the peer review phase of an assignment, email notifications are sent out to the instructor whenever a submitted review score differs “significantly” from the average score of other submitted reviews for that submission. The threshold to trigger a notification is specified in the “Notification limit” on the Rubrics tab of assignment creation. The email that gets sent to instructors contains three links: a link the the conflicting review, a link the the summary page of grades for the reviewee, and a link to edit the assignment. Although the links sent in the email are helpful for reference, the process of examining a conflicting review could be made more efficient if we provide an easy to understand report of the conflict. The information sent in the email should be updated to contain a link to a report page which should contain more details about the newest conflict as well as information on previous review conflicts for the assignment.

Existing Implementation

On submitting a new review for a particular artifact, the submitted score is compared to the average score of the previously submitted reviews. If the difference between the average score and the newly submitted score is more than the notification threshold specified for the assignment an email is sent out to the instructor. The email sent to the instructor contains the names of the reviewer and reviewee and a link to the conflicting review. The flowchart below delineates the entire process:

Problems

The existing implementation has the following flaws:

No dedicated view for analyzing the conflict

The current implementation does not provide any dedicate view or report containing a detailed analysis of the conflict.

Incorrect email URLs

The current implementation uses hardcoded URLs mentioned in the notify_instructor_on_difference method of the models/response.rb file. Being hardcoded, these link would not work in on other servers where Expertiza is running. For example, the links won't be valid if the setup was done on the localhost.

Previous Work

The Previous Work and pull request, although functional, had a few main problems that prevented it from being merged:

1. The UI for the conflict report included a bar graph which showed the grades of reviews that were in conflict. The information in the bar graph was the same information shown as text right beside it, just converted to a percentage. Therefore, the bar graph served no purpose but to clutter the UI. See a screenshot of the previous implementation below:

2. The code for the conflict report view included a lot of logic.
3. There was minimal effort spent on refactoring test cases and the test cases added in the pull request were shallow tests, meaning that they simply asserted that the expected value was not empty rather that the correct way of asserting the expected value matched a specific value.

Design

Proposed Solution

Once the existing code determines that a conflict has occurred we need to create a report view for the conflict. The report view will contain all the relevant statistics(mean, median, standard deviation etc) of all the reviews received for the submission. Next, a unique URL needs to be generated for the report and that URL will be added to the email body. All URLs in the email will be relative URLs and therefore the URLs will be valid on all servers. Following files need to be updated in order to implement these changes:

• Related to Email:
  • app/mailers/mailer.rb
    • Modify method notify_grade_conflict_message to make the link to the conflict report available in the email.
  • app/models/response.rb
    • Modify method notify_instructor_on_difference to add a link to the conflict report that gets sent to the mailer.
  • app/views/mailer/notify_grade_conflict_message.html.erb
    • Modify the html to add to the email message a link to view the conflict report.
  • app/controllers/response_contrller.rb
    • Modify update method to add request.base_url as a param sent to notify_instructor_on_difference method
    • Modify create method to add request.base_url as a param sent to notify_instructor_on_difference method

• Related to the conflict report view:
  • app/views/reports/_searchbox.html.erb
    • Modify the html to add the option to view a conflict report.
  • app/views/reports/response_report.html.haml
    • Modify the html to render the conflict report partial if the option to view a conflict report was selected.

What we need to create:

• Related to the conflict report view:
  • App/views/reports/_conflict_report.html.erb
    • Need to create this entire file to make the view for a conflict report
  • app/helpers/report_formatter_helper.rb
    • Need to add a method conflict_reponse_map to make specific instance variables available to our conflict report view. This method will get called from the ReportsController.
  • app/helpers/summary_helper.rb
    • Need to add a method to get the maximum possible score of an assignment in each round
  • app/models/answer.rb
    • Need to add a method to get the answers from a review for an assignment submitted by a specific reviewee.
  • app/helpers/review_mapping_helper.rb
    • Need to add methods to get the review scores for a team in each round, calculate the average review score for a round, calculate the std. deviation of review scores for a round, and get the team members of each team.

The conflict report view should:

• For each team that had a submission with a review that was in conflict:
  • List the team name and team members
  • Provide a graphical view of the statistics displayed in the report
  • For each round where there was a review in conflict:
    • List the students who's review caused a conflict and the score they gave
    • List the threshold for conflict review score, max review score, average review score, and standard deviation.

Use Case Diagram

Actors participating:

• Student: In this use case, fills out the reviews for different projects submitted by the peers.
• TA/Instructor: In this use case, goes over the conflicting review submitted by the students for the same project.

Test Plan

RSpec Testing

After implementing our changes, we will need to refactor the old rspec tests and make sure they pass. We will also add a few more RSpec and Capybara tests to ensure the new functionalities and the new review conflict report appear and work as expected. We plan to modify/implement changes in the following RSpec files:

• Modify spec/models/answer_spec.rb to add a test for a method we plan on creating that gets the answers to a review for an assignment submitted by a specific reviewee.
• Add new tests to spec/controllers/report_controller_spec.rb to check if the proper view is generated.
• Add new tests to spec/controllers/report_controller_spec.rb to check if a proper data is added to the body of the email if a conflict occurs.

Some edge cases we would like to consider include:

• The score for a review is exactly the threshold to trigger a conflict specified for the assignment, so a conflict email should be sent out and conflict report should show the review score.
• There are no reviews in conflict for a specific team so the conflict report should not show that specific team's information.

Manual Testing

In order to test the functionality manually, we follow the following steps: 1. Create a new assignment with 15% as a conflict threshold limit.
2. Add 4 students to the assignment and divide them into 2 teams of 2 students in each.
3. Add submission for each team.
4. In the review phase, login to the first team student accounts and give scores of 100% and 50% to the second team's assignment.
5. Since the scores differ by more than 15% the email will be sent to the instructor.
6. Access the email and check if the email contains the following details:

• Names of the reviewer and reviewee
• Link to the conflicting review
• Link to the newly created report view corresponding to the conflict
• Link to edit the assignment notification limit

Useful Links

Github: https://github.com/sid189/expertiza/tree/beta
Pull Request: https://github.com/expertiza/expertiza/pull/1714

Team Information

Project Mentor:
Pratik Abhyankar

Project Members:
Sahil Papalkar
Sahil Mehta
Siddharth Deshpande
Carl Klier