CSC/ECE 517 Spring 2020 - E2015. Conflict notification

From Expertiza_Wiki
Revision as of 02:53, 14 April 2020 by Caklier (talk | contribs)
Jump to navigation Jump to search

Abstract

Expertiza is an open-source web application implemented using Ruby on Rails. It is used for management of courses and the assignments for respective courses, by the faculties and the students. Students can form teams in Expertiza to work on an assignment in a group. A student team can submit their work through multiple means such as file uploads and embedded links. Submissions made on Expertiza can be peer-reviewed by students within the course based on a rubric. Results of the peer review can be accessed by instructors and respective student teams. During the peer evaluation, Expertiza sends appropriate email notifications to the instructor and also provides a complete statistical report of the evaluation.

Problem Statement

In the current implementation, during the peer review phase of an assignment, email notifications are sent out to the instructor whenever a submitted review score differs “significantly” from the average score of other submitted reviews for that submission. The threshold to trigger a notification is specified in the “Notification limit” on the Rubrics tab of assignment creation. The email that gets sent to instructors contains three links: a link the the conflicting review, a link the the summary page of grades for the reviewee, and a link to edit the assignment. Although the links sent in the email are helpful for reference, the process of examining a conflicting review could be made more efficient if we provide an easy to understand report of the conflict. The information sent in the email should be updated to contain a link to a report page which should contain more details about the newest conflict as well as information on previous review conflicts for the assignment.

Existing Implementation

On submitting a new review for a particular artifact, the submitted score is compared to the average score of the previously submitted reviews. If the difference between the average score and the newly submitted score is more than the notification threshold specified for the assignment an email is sent out to the instructor. The email sent to the instructor contains the names of the reviewer and reviewee and a link to the conflicting review. The flowchart below delineates the entire process:

Problems

The existing implementation has the following flaws:

No dedicated view for analyzing the conflict

The current implementation does not provide any dedicate view or report containing a detailed analysis of the conflict.

Incorrect email URLs

The current implementation uses hardcoded URLs mentioned in models/response.rb file. Being hardcoded, these link would not work in on other servers where Expertiza is running. For example, the links won't be valid if the setup was done on the localhost.

Previous Work

The Previous Work and pull request, although functional, had a few main problems that prevented it from being merged:

1. The UI for the conflict report included a bar graph which showed the grades of reviews that were in conflict. The information in the bar graph was the same information shown as text right beside it. Therefore, the bar graph served no purpose but to clutter the UI. See a screenshot of the previous implementation
2. The code for the conflict report view included a lot of logic.
3. There was minimal effort spent on refactoring test cases and the test cases added in the pull request were shallow tests, meaning that they simply asserted that the expected value was not empty rather that the correct way of asserting the expected value matched a specific value.

Design

Proposed Solution

Once the existing code determines that a conflict has occurred we need to create a report view for the conflict. The report view will contain all the relevant statistics(mean, median, standard deviation etc) of all the reviews received for the submission. Next, a unique URL needs to be generated for the report and that URL will be added to the email body. All URLs in the email will be relative URLs and therefore the URLs will be valid on all servers. Following files need to be updated in order to implement these changes:

  • Related to Email:
    • app/mailers/mailer.rb in method notify_grade_conflict_message to make the link to the conflict report available in the email.
    • app/models/response.rb in method notify_instructor_on_difference to add a link to the conflict report that gets sent to the mailer.
    • app/views/mailer/notify_grade_conflict_message.html.erb to add to the message a link to view the conflict report.
    • app/controllers/response_contrller.rb to add the request.base_url as a param sent to notify_instructor_on_difference in the update and create methods
  • Related to the conflict report view:
    • app/views/reports/_searchbox.html.erb to add the option to view a conflict report.
    • app/views/reports/response_report.html.haml to render the conflict report partial if the option to view a conflict report was selected.
    • app/helpers/report_formatter_helper.rb to add a method like conflict_reponse_map to make specific instance variables available to our conflict report view. This method will get called from the ReportsController.
    • app/helpers/summary_helper.rb to add a method to get the maximum possible score of an assignment in each round
    • app/helpers/review_mapping_helper.rb to add methods to get the review scores for a team in each round, calculate the average review score for a round, calculate the std. deviation of review scores for a round, and get the team members of each team.


What we need to create:

  • Related to the conflict report view:
    • App/views/reports/_conflict_report.html.erb to make the view for a conflict report


The conflict report view should:

  • For each team that had a submission with a review that was in conflict:
    • List the team name and team members
    • Provide a graphical view of the statistics displayed in the report
    • For each round where there was a review in conflict:
      • List the students who's review caused a conflict and the score they gave
      • List the threshold for conflict review score, max review score, average review score, and standard deviation.

Use Case Diagram




Actors participating:

  • Student: In this use case, fills out the reviews for different projects submitted by the peers.
  • TA/Instructor: In this use case, goes over the conflicting review submitted by the students for the same project.

Test Plan

RSpec Testing

Apart from ensuring that the existing RSpec tests pass after implementing the new functionalities we also add a few more RSpec and capybara test to test the new functionalities and the new report views. We plan to modify.implement changes in the following RSpec files:

  • Modify spec/models/answer_spec.rb which contains the existing to test checks for conflict. This file will be changed according to the changing functionality of the corresponding model logic.
  • Add new tests to spec/controllers/report_controller_spec.rb to check if a proper view is generated corresponding to every conflict.
  • Add new tests to spec/controllers/report_controller_spec.rb to check if a proper data is added to the body of the email if a conflict occurs.

Manual Testing

In order to test the functionality manually, we follow the following steps: 1. Create a new assignment with 15% as a conflict threshold limit.
2. Add 4 students to the assignment and divide them into 2 teams of 2 students in each.
3. Add submission for each team.
4. In the review phase, login to the first team student accounts and give scores of 100% and 50% to the second team's assignment.
5. Since the scores differ by more than 15% the email will be sent to the instructor.
6. Access the email and check if the email contains the following details:

  • Names of the reviewer and reviewee
  • Link to the conflicting review
  • Link to the newly created report view corresponding to the conflict
  • Link to edit the assignment notification limit

Useful Links

Github: https://github.com/sid189/expertiza/tree/beta
Pull Request: https://github.com/expertiza/expertiza/pull/1714


Team Information

Project Mentor:
Pratik Abhyankar

Project Members:
Sahil Papalkar
Sahil Mehta
Siddharth Deshpande
Carl Klier

Retrieved from "https://wiki.expertiza.ncsu.edu/index.php?title=CSC/ECE_517_Spring_2020_-_E2015._Conflict_notification&oldid=133574"