This wiki page is for the description of changes made under ECE517 SPRING 2019 PROJECT E1931 CONFLICT NOTIFICATIONS

Problem Statement

Current conflict notification sends an email to the instructor whenever two reviews for the same submission differ “significantly” on their scoring (the threshold is specified in the “Notification limit” on the Rubrics tab of assignment creation). Right now an email is sent at any such time one of these conflicts happen, and it links to the review or submission which initiated the notification. It gives the link in raw HTML, but it should give it as a link associated with text, such as the text “this new review” linking to the page that shows the review.The email send to the instructor should also link to page that shows data on each of the reviews done on this team.

Existing Feature Description

Currently, this feature works as follows:

Whenever a new review is submitted, it is compared with the average of the previously submitted reviews. If the difference between the new submission and existing average is more than the threshold limit set, then a mail is sent to the instructor.With every review submitted for an assignment of a particular student, the average is updated.The mail sent to the instructor contains the links to the conflicted review and a summary of the reviews.

Flaws with the existing Implementation

No view exists to analyze the conflict report

Currently whenever the conflict happens,a summary link is sent to the instructor which contains the score of all the reviews but there is no view where the instructor can see all the conflicts and analyze them.

Incorrect email message links

The existing email body uses hardcoded URLs mentioned in models/response.rb file in the method notify_instructor_on_difference. Being hardcoded, these links wouldn't work on other servers where Expertiza is running. For Example, if the setup is done on localhost, the links will not be functional.

Conflicting Review which triggered the mail

What needs to be done

The scope of this project is to send an email notification to instructor which will contain links to the conflicting review, summary link, and a link to report which can be used for analyzing. The new report will have the information like the Team(having conflicts), the standard deviation for the team review score and pictorial representation of all the review scores.

The email sent to the instructor should link to a page that shows the data on each of the reviews done on this team:

The name of the team.
The members (user-IDs and/or names)
One line for each review received.In this line would be

The name of the person (or team) that submitted the review, which would be hyperlinked to the review itself
The score given for the review
A bar chart showing pictorial view

The average score received from reviewers.
The standard deviation of the score received from reviewers.
Review scores that are outside some “tolerance” limit should be highlighted, e.g., boldfaced or shown in a different color.Probably the tolerance should be, perhaps, one or two standard deviations from the mean.

Implementation

Feature 1: New page to show the conflict report

A new view page of the report which will have the following information:-

The name of the team.
The members of the team.
The name of the reviewer which would be hyperlinked to the review itself.
The score given for the review.
A bar chart showing pictorial view of all the reviews for a particular team.
The average score received from reviewers.
The standard deviation of the score received from reviewers.
Conflicting reviewers are highlighted.These are those which are two standard deviations away from the mean.

This view have a report of all the teams on one page.This report link is added to the mail that is already being sent whenever some conflict happens.This report page will be also accessible from the reports drop-down in the instructor view.

A new partial is created for this view -

<div style="color:green;text-align:adjust">
  Note: <br>
  X- axis represents precentage score of the team<br>
  Y- axis represents the reviewers <br>
  Highlighted reviewers are outside the tolerance limit<br>
</div><br><br>
<table id="myTable" border="1px solid black" margin-top="-9px"  width="100%" cellspacing='0' cellpadding='2' border='0'>
  <thead>
  <tr bgcolor='#CCCCCC'>
    <!-- class value decides whether the column should be sortable or not  -->
    <th width="30%" class="sorter-false" border="1px black solid" style="text-align:center">Team <span></span></th>
    <th width="70%" class="sorter-false" border="1px black solid" style="text-align:center" >Reviewers/Metrics <span></span></th>
  </tr>
  </thead>
  <tbody>
  <% max_scores= max_assignment_review_score_per_round(@id) %>
  <%@reviewers.keys.each do |team_name| %>
    <tr>
      <td style="padding-left: 50px;" >
        <% members = team_members(team_name, @id)%>
        <strong>Team Name: </strong><%= team_name %><br>
        <strong>Team Members: </strong><br>
	<% members.each do |member|%>
          <%=User.find(member.user_id).fullname %><br>
        <% end %>
      </td>
      <td>
      <%= render 'review_conflict_metric' , team_name: team_name , max_scores: max_scores %>
  </td>
  </tr>
  <% end %>
  </tbody>
</table>

Partial for the display for reviewers,scores and metrics

<table width="100%" border="0" >
        <%question_answers = review_score_for_team(@id, team_name) %>
        <% i = 1 %>
        <% question_answers.each do |question_answer| %>
                <% average = average_of_round(question_answer) %>
                <% std = std_of_round(average,question_answer) %>
                <% upper_tolerance_limit = (average+(2*std)).round(2) %>
                <% lower_tolerance_limit = (average-(2*std)).round(2) %>
                <tr>
                <td width="30%">
                <div>
                  <strong>Reviewers with scores</strong><br>
            <ol>
            <% question_answer.each do |reviewer, answer| %>
                <% if answer > upper_tolerance_limit or answer < lower_tolerance_limit  %>
                        <li><% response_id = response_id_review_conflict(reviewer, @teams[team_name], @id) %>
                        <mark style="background-color:#FFFF00">
                        <%= link_to "("+@reviewers[team_name.to_s][reviewer.to_s].to_s+")", :controller => 'popup', :action => 'team_users_popup', :id => @teams[team_name], :id2 => response_id, :assignment_id=>@id %>
                        <%= list_review_submissions(reviewer, @teams[team_name], response_id) %>
                        <%= ": " + answer.to_s %></mark>
                        </li>
                <% else %>
                        <li><% response_id = response_id_review_conflict(reviewer, @teams[team_name], @id) %>
                        <%= link_to "("+@reviewers[team_name.to_s][reviewer.to_s].to_s+")", :controller => 'popup', :action => 'team_users_popup', :id => @teams[team_name], :id2 => response_id, :assignment_id=>@id %>
                        <%= list_review_submissions(reviewer, @teams[team_name], response_id) %>
                        <%= ": " + answer.to_s %>
                        </li>
                <% end %>
            <% end %>
            </ol>
            </div>
          <strong>Max Review Score: </strong><%=max_scores[i-1]%><br>
          <strong>Average Score: </strong><%=  average.to_s %><br>
          <strong>Standard Deviation: </strong><%=  std.to_s %><br><br>
      </td>
      <td>
      <div>
          <strong><u><%= "Round: " + i.to_s %></u></strong>
          <%= generate_score_chart(max_scores[i-1], question_answer) %>
      </div>
      </td>
    </tr>
    <% i+=1 %>
  <% end %>
  </table>

Sample view of the new report -

Feature 2: No hardcoded URLs

Hardcoded URLs are be managed by the config file in order to make these links functional on every server running Expertiza and raw URLs will be given some meaningful names.

Code changed for removing the hardcoded URLs

Test Plan

To test code implementation and correctness of the modified feature, we had done the following tests:-

Run and pass existing RSpec Tests
UI testing for the new functionality
Develop New RSpec Tests for the new code

Rspec Testing

All the existing related rspec tests have tested and they all passed wit 0 failures. Majority of the new functionality concerns with the new view,so we have mainly done the UI testing shown below.Some of the Rspec test added are shown below:

Rspec Tests for Answer Model

let(:response_map) { create(:review_response_map, id: 1, reviewed_object_id: 1, reviewer_id: 1) }
it "returns answers by reviewer for reviewee in round from db which is not empty" do
      expect(Answer.answers_by_round_for_reviewee(@assignment_id, @reviewee_id,@round)).not_to be_empty
end

Rspec Tests for Report Formatter Helper methods

describe 'review_conflict_response_map' do
      context 'when type is ReviewConflictResponseMap' do
        it 'renders response_report page with corresponding data' do
          allow(Team).to receive(:where).with(parent_id: '1').and_return([integer('id')]).ordered
          params = {
              id: 1,
              report: {type: 'ReviewResponseMap'},
              user: 'no one'
          }
          get :response_report, params
          expect(response).to render_template(:response_report)
        end
      end
    end

    describe "reviewers_name_id_by_reviewee_and_assignment" do
      before(:each) do
        @assignment_id = 1
        @reviewee_id = 1
      end
      it "returns reviewers name from Answer by reviewee ans assignment id from db which is not empty" do
        expect(allow(User).to receive(:where).with(@reviewee_id, @assignment_id).and_return([integer('id')])).not_to be_empty
      end
    end

UI Testing

We have crated a video showing the new functionality.

Files changed in this Project

Response Controller
Report Formatter Helper
Review Mapping Helper
Summary Helper
Mailer
Answer model
Response Model
Notify Grade Conflict Message View
Review Conflict Metric View (New Created)
Review Conflict Report View (New Created)
Review Report View
Searchbox View
Response Report View
Reports Controller Spec
Answer model spec

References

Team

Abhishek Arya
Bhargav Deshpande
Ramandeep Kaur
Udita Chattopadhyay

CSC/ECE 517 Spring 2019 - Project E1931. Conflict notification

Contents

Problem Statement

Existing Feature Description

Flaws with the existing Implementation

No view exists to analyze the conflict report

Incorrect email message links

What needs to be done

Implementation

Test Plan

Rspec Testing

UI Testing

Files changed in this Project

References

Team

Navigation menu

CSC/ECE 517 Spring 2019 - Project E1931. Conflict notification

Problem Statement

Existing Feature Description

Flaws with the existing Implementation

No view exists to analyze the conflict report

Incorrect email message links

What needs to be done

Implementation

Test Plan

Rspec Testing

UI Testing

Files changed in this Project

References

Team

Navigation menu

Search