CSC/ECE 517 Fall 2017/E1777 Coherent specification of review requirements.rb

E1777. OSS Project Green: Coherent specification of review requirements

This page provides a description of the Expertiza based OSS project.

About Expertiza

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Statement

The following tasks were accomplished in this project:

• The punctuation, syntax and the capitalization of statements in the review strategy tab requires improvement.
• Information buttons to elaborate difference between the synonymous statements present in review strategy tab need to be included.
• Message to a reviewer regarding how many reviews they are required to and allowed to perform is not clear enough.
• Assignment of work to review for a new reviewer is based on the number of previous reviews assigned for that submission which is incorrect.

Implementation

Drawbacks and Solutions

• Problem 1: To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0).
• Solution: This issue has been fixed previously. The reviewer will get assigned a submission even if it has fulfilled the required number of reviews, to ensure that the reviewer always receives a new submission to review as long as it is not his own.
• Problem 2: The two statements "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." are not well differentiated. It can very confusing for an instructor to give values for them.

• Solution: Information buttons have been provided beside these statements in the review strategy tab to make sure that the instructor knows the difference between the two.
• this modification is made in the app/views/assignments/edit/_review_strategy.html.erb

<!--Adding info button-->
    <img src="/assets/info.png" title='This is the minimum number of reviews a reviewer has to perform'>
  <br/>
    <br/>
    <%= check_box_tag('review', 'true', true, {:onChange => 'hasReviewChanged()'}) %>
    <%= label_tag('review', 'Has max review limit?') %>  
    <span id="reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_reviews_allowed]', ' Set allowed number of reviews per reviewer') %>
      <%if @assignment_form.assignment.num_reviews_allowed && @assignment_form.assignment.num_reviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]', @assignment_form.assignment.num_reviews_allowed, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]',  3, size: 1) %>
      <%end%>
      <!--Adding info button-->
      <img src="/assets/info.png" title='This is the maximum number of reviews a reviewer can perform'<%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    </span>

  </td>

• Problem 3: When the number of allowed or required reviews is not set on the Review Strategy tab, the system does not have a message to display to a reviewer about how many submissions of work they are required to and allowed to review depending on the values set by the instructor in the "allowed number of reviewers per reviewer" field and the "Set Required Number of Reviews per reviewer." field.

• Solution: This view has been implemented in the app/views/student_review/list.html.erb

<h2>Reviews for "<%= @assignment.name %>"</h2>

<!--Changing statements to display on the others work screen-->
<% if @assignment.num_reviews_allowed.nil? || @assignment.num_reviews_allowed == -1%>
    <h4>Your instructor expects you to do <%= @assignment.num_reviews_required %> reviews. You are not allowed to do any extra reviews. </h4>

<% elsif @assignment.num_reviews_allowed == @assignment.num_reviews_required %>
    <h4>You should perform exactly <%= @assignment.num_reviews_allowed %> reviews </h4>
<% else %>

    <h4>You may perform between <%= @assignment.num_reviews_required %> and <%= @assignment.num_reviews_allowed %> reviews</h4>
<% end %>

• Problem 4: The capitalization and punctuations of statements such as "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." in review strategy tab are incorrect.
• Solution: This view has been fixed by making changes in the respective files.
• The changes were made in the file : app/views/assignments/edit/_review_strategy.html.erb

<%= label_tag('assignment_form[assignment][num_reviews_required]', 'Set required number of reviews per reviewer') %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_required]', @assignment_form.assignment.num_reviews_required ||= @assignment_form.assignment.num_reviews, size: 1) %>

    <!--Adding info button-->
    <img src="/assets/info.png" title='This is the minimum number of reviews a reviewer has to perform'>
  <br/>
    <br/>
    <%= check_box_tag('review', 'true', true, {:onChange => 'hasReviewChanged()'}) %>
    <%= label_tag('review', 'Has max review limit?') %>  
    <span id="reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_reviews_allowed]', ' Set allowed number of reviews per reviewer') %>
      <%if @assignment_form.assignment.num_reviews_allowed && @assignment_form.assignment.num_reviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]', @assignment_form.assignment.num_reviews_allowed, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]',  3, size: 1) %>
      <%end%>
      <!--Adding info button-->
      <img src="/assets/info.png" title='This is the maximum number of reviews a reviewer can perform'<%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    </span>

  </td>
    <td width="50%">

    <%= label_tag('assignment_form[assignment][num_metareviews_required]', 'Set required number of meta-reviews per reviewer') %>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_required]', @assignment_form.assignment.num_metareviews_required ||= @assignment_form.assignment.num_metareviews_allowed, size: 1) %>
   <br/>
      <br/>

    <%= check_box_tag('meta_review', 'true', true, {:onChange => 'hasMetaReviewChanged()'}) %>
    <%= label_tag('meta_review', 'Has meta-review limit?') %>  
    <span id="meta_reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_metareviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_metareviews_allowed]', ' Set allowed number of meta-reviews per reviewer') %>
      <%if @assignment_form.assignment.num_metareviews_allowed && @assignment_form.assignment.num_metareviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]', @assignment_form.assignment.num_metareviews_allowed ||= 3, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]',  3, size: 1) %>
      <%end%>
    </span>

  </td>
</tr>

• Problem 5: The assignment of work to review to a reviewer depends on the number of reviews it has previously received, if that work has been assigned to a reviewer previously and its review has not been submitted then the system will still consider it such that it would not assign the same work to another reviewer if it has enough number of reviewers previously assigned to review it. This might lead to a scenario where that work was assigned to enough number of reviewers, but it does not receive enough reviews and cannot be assigned to any new reviewers.
• Solution: This implementation has been fixed. The system now considers only the reviews that were submitted by the reviewers for that work so that every submission gets assigned to reviewers until it receives its allowed number of reviews.

• the modified file is app/models/review_assignment.rb and the modification is done in the method reject_by_max_reviews_per_submission

def reject_by_max_reviews_per_submission(contributor_set)
  contributor_set.reject! {|contributor| contributor.responses.reject {|response|  !response.is_submitted }.count >= max_reviews_per_submission }
  contributor_set
end

Modified Files

• app/views/assignments/edit/_review_strategy.html.erb
• app/models/review_assignment.rb
• app/views/student_review/list.html.erb

References

1. Expertiza on GitHub
2. GitHub Project Repository Fork
3. The live Expertiza website
4. Expertiza project documentation wiki
5. Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin