CSC/ECE 517 Spring 2020 - E2015. Conflict notification

Abstract

Expertiza is an open-source web application implemented using Ruby on Rails. It is used for management of courses and the assignments for respective courses, by the faculties and the students. Students can form teams in Expertiza to work on an assignment in a group. A student team can submit their work through multiple means such as file uploads and embedded links. Submissions made on Expertiza can be peer-reviewed by students within the course based on a rubric. Results of the peer review can be accessed by instructors and respective student teams. During the peer evaluation, Expertiza sends appropriate email notifications to the instructor and also provides a complete statistical report of the evaluation.

Problem Statement

In the current implementation, during the peer review phase of an assignment, email notifications are sent out to the instructor whenever a submitted review score differs “significantly” from the average score of other submitted reviews for that submission. The threshold to trigger a notification is specified in the “Notification limit” on the Rubrics tab of assignment creation. The email that gets sent to instructors contains three links: a link to the conflicting review, a link to the summary page of grades for the reviewee, and a link to edit the assignment. Although the links sent in the email are helpful for reference, the process of examining a conflicting review could be made more efficient if we provide an easy to understand report of the conflicts. The information sent in the email should be updated to contain a link to a report page which should contain more details about the newest conflict as well as information on previous review conflicts for the assignment.

Existing Implementation

The existing system has the following implementation for conflict notification. On submitting a new review for a particular artifact, the submitted score is compared to the average score of the previously submitted reviews. If the difference between the average score and the newly submitted score is more than the notification threshold specified for the assignment an email is sent out to the instructor. The email sent to the instructor contains the names of the reviewer and reviewee and a link to the conflicting review. The flowchart below delineates the entire process:

Problems

The existing implementation has the following flaws:

No dedicated report for analyzing the conflict

The current implementation does not provide any dedicated view or report containing a detailed analysis of the conflicts that occur during the entire review period.

Incorrect email URLs

The current implementation uses hardcoded URLs mentioned in the notify_instructor_on_difference method of the models/response.rb file. Being hardcoded, these link would not work in on other servers where Expertiza is running. For example, the links won't be valid if the setup was done on the localhost.

Previous Work

The Previous Work resolved the email related issues and also created a simple report for analyzing the conflict. However, the implementation had quite a few issues that prevented it from being merged:

1. The UI for the conflict report included a bar graph that showed the grades of reviews that were in conflict. The information in the bar graph was the same information shown as text right beside it, just converted to a percentage. Therefore, the bar graph served no purpose but to clutter the UI. See a screenshot of the previous implementation below:

2. The code for the conflict report view included a lot of logic.
3. The code logic for fetching the teammates was flawed and therefore the report displayed the teammates incorrectly.
4. There was minimal effort spent on refactoring test cases and the test cases added in the pull request were shallow tests, meaning that they simply asserted that the expected value was not empty rather than the correct way of asserting the expected value matched a specific value.

Design

Proposed Solution

As the mailing functionality was implemented perfectly in the previous implementation we plan to use their implementation as our starting point. Building on top of it we plan to address the highlighted issues about the mailer and completely refactor the conflict report so that it is able to provide better insights into the conflicts. Once the existing code determines that a conflict has occurred will create a report view for the conflict. The report view will contain all the relevant statistics(mean, median, standard deviation etc) of all the reviews received for the submission. Next, a URL for the report page will be added to the email body. All URLs in the email will be relative URLs and therefore the URLs will be valid on all servers. Following files need to be updated in order to implement these changes:

• Related to Email:
  • app/mailers/mailer.rb
    • Modify method notify_grade_conflict_message to add the link to the conflict report in the email body.
  • app/models/response.rb
    • Modify method notify_instructor_on_difference to add a relative links to various pages that get included in the email body.
  • app/views/mailer/notify_grade_conflict_message.html.erb
    • Modify the html to add a link to the email message using which the conflict report viewed.
  • app/controllers/response_contrller.rb
    • Modify update method to add request.base_url as a param sent to notify_instructor_on_difference method
    • Modify create method to add request.base_url as a param sent to notify_instructor_on_difference method

• Related to the conflict report view:
  • app/views/reports/_searchbox.html.erb
    • Modify the html to add the option to view a conflict report.
  • app/views/reports/response_report.html.haml
    • Modify the html to render the conflict report partial if the option to view a conflict report was selected.

What we need to create:

• Related to the conflict report view:
  • App/views/reports/_conflict_report.html.erb
    • Need to create this entire file to make the view for a conflict report
  • app/helpers/report_formatter_helper.rb
    • Need to add a method conflict_reponse_map to make specific instance variables available to our conflict report view. This method will get called from the ReportsController.
  • app/helpers/summary_helper.rb
    • Need to add a method to get the maximum possible score of an assignment in each round
  • app/models/answer.rb
    • Need to add a method to get the answers from a review for an assignment submitted by a specific reviewer.
  • app/helpers/review_mapping_helper.rb
    • Need to add methods to get the review scores for a team in each round, calculate the average review score for a round, calculate the std. deviation of review scores for each round, and get the team members of each team. Apart from these any other method to find and calculate data to be viewed in the report can be added to this helper file.

The conflict report view should:

• For each team that had a submission with a review that was in a conflict:
  • List the team name and team members
  • Provide a graphical view of the statistics displayed in the report
  • For each round where there was a review in conflict:
    • List and highlight the students who's review caused a conflict and the score they gave
    • List the threshold for conflict review score, max review score, average review score, and standard deviation.
      
      

In order to resolve the issues with the previous team's pull request, we will:
1. Not include any unnecessary bar charts with redundant information in our conflict report view.
2. Put the necessary logic into helper methods in helper files to ensure logic stays out of our views.
3. Replace the logic used in the review_mapping_helper.rb file to fetch the team member of a specific team.
4. Refactor the necessary affected test cases and write new RSpec tests in accordance with our proposed test plan to ensure adequate coverage.

UML Diagrams

Use Case Diagram

The usecase diagram highlights the simple workflow of the proposed system when trigger by the user.


Actors participating:

• Student: In this use case, fills out the reviews for different projects submitted by the peers.
• TA/Instructor: In this use case, goes over the conflicting review submitted by the students for the same project.

Sequence Diagram

The sequence diagram models the interactions that will take place between various components of the system for the above use-case.

Implementation

As per the plan stated in the above section, following changes were implemented:

Test Plan

RSpec Testing

After implementing our changes, we will need to refactor the old RSpec tests and make sure they pass. We will also add a few more RSpec and Capybara tests to ensure the new functionalities and the new review conflict report appear and work as expected. We plan to modify/implement changes in the following RSpec files:

• Modify spec/models/answer_spec.rb to add a test for a method we plan on creating that gets the answers to a review for an assignment submitted by a specific reviewee.
• Add new tests to spec/controllers/report_controller_spec.rb to check if the proper view is generated.
• Add new tests to spec/controllers/report_controller_spec.rb to check if a proper data is added to the body of the email if a conflict occurs.

Some edge cases we would like to consider include:

• The score for a review is exactly the threshold to trigger a conflict specified for the assignment, so a conflict email should be sent out and conflict report should show the review score.
• There are no reviews in conflict for a specific team so the conflict report should not show that specific team's information.

Manual Testing

In order to test the functionality manually, we follow the following steps:
1. Create a new assignment with 15% as a conflict threshold limit.
2. Add 4 students to the assignment and divide them into 2 teams of 2 students in each.
3. Add submission for each team.
4. In the review phase, login to the first team student accounts and give scores of 100% and 50% to the second team's assignment.
5. Since the scores differ by more than 15% the email will be sent to the instructor.
6. Access the email and check if the email contains the following details:

• Names of the reviewer and reviewee
• Link to the conflicting review
• Link to the newly created report view corresponding to the conflict
• Link to edit the assignment notification limit

7. View the conflict report and ensure that Team 2 shows up as having a conflicting review submitted by student 2 of Team 1.

Useful Links

Github: https://github.com/sid189/expertiza/tree/beta
Pull Request: https://github.com/expertiza/expertiza/pull/1714

Team Information

Project Mentor:
Pratik Abhyankar

Project Members:
Sahil Papalkar
Sahil Mehta
Siddharth Deshpande
Carl Klier