CSC/ECE 517 Fall 2017/E1777 Coherent specification of review requirements.rb

E1777. OSS Project Green: Coherent specification of review requirements

This page provides a description of the Expertiza based OSS project.

About Expertiza

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Statement

The following tasks were accomplished in this project:

• Improved the punctuation, syntax and the capitalization of statements in the review strategy tab.
• Provided information buttons to elaborate difference between the synonymous statements present in review strategy tab.
• Display of message to a reviewer regarding how many reviews they are required to and allowed to perform.
• Assignment of work to review for a new reviewer is now based on the number of previous reviews submitted for that work and not on the numbers of assigned reviews for that work.

Implementation

Functionality

• In Expertiza, there are two ways of assigning reviews to reviewers: either the instructor decides who reviews whom (“instructor-selected”), or “auto-selected,” in which case reviews are not assigned until a student seeks to choose something to review. To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0).
• The capitalization and punctuation in many statements under the review strategy tab are random.
• On the Review Strategy tab of assignment creation, the "Maximum number of reviews per submission" checks the number of reviews that have been assigned to reviewers instead of checking the number of reviews that are submitted by the reviewers. This means that if required number of reviewers choose submissions and don't submit their reviews, those submissions cannot be reviewed any more.

Drawbacks and Solutions

• Problem 1: To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0).
• Solution: This issue has been fixed previously. The reviewer will get assigned a submission even if it has fulfilled the required number of reviews, to ensure that the reviewer always receives a new submission to review as long as it is not his own.
• Problem 2: The two statements "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." are not well differentiated. It can very confusing for an instructor to give values for them.

• Solution: Information buttons have been provided beside these statements in the review strategy tab to make sure that the instructor knows the difference between the two.
• this modification is made in the app/views/assignments/edit/_review_strategy.html.erb

<!--Adding info button-->
    <img src="/assets/info.png" title='This is the minimum number of reviews a reviewer has to perform'>
  <br/>
    <br/>
    <%= check_box_tag('review', 'true', true, {:onChange => 'hasReviewChanged()'}) %>
    <%= label_tag('review', 'Has max review limit?') %>  
    <span id="reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_reviews_allowed]', ' Set allowed number of reviews per reviewer') %>
      <%if @assignment_form.assignment.num_reviews_allowed && @assignment_form.assignment.num_reviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]', @assignment_form.assignment.num_reviews_allowed, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]',  3, size: 1) %>
      <%end%>
      <!--Adding info button-->
      <img src="/assets/info.png" title='This is the maximum number of reviews a reviewer can perform'<%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    </span>

  </td>

• Problem 3: When the number of allowed or required reviews is not set on the Review Strategy tab, the system does not have a message to display to a reviewer about how many submissions of work they are required to and allowed to review depending on the values set by the instructor in the "allowed number of reviewers per reviewer" field and the "Set Required Number of Reviews per reviewer." field.

• Solution: This view has been implemented in the app/views/student_review/list.html.erb

<h2>Reviews for "<%= @assignment.name %>"</h2>

<!--Changing statements to display on the others work screen-->
<% if @assignment.num_reviews_allowed.nil? || @assignment.num_reviews_allowed == -1%>
    <h4>Your instructor expects you to do <%= @assignment.num_reviews_required %> reviews. You are not allowed to do any extra reviews. </h4>

<% elsif @assignment.num_reviews_allowed == @assignment.num_reviews_required %>
    <h4>You should perform exactly <%= @assignment.num_reviews_allowed %> reviews </h4>
<% else %>

    <h4>You may perform between <%= @assignment.num_reviews_required %> and <%= @assignment.num_reviews_allowed %> reviews</h4>
<% end %>

• Problem 4: The capitalization and punctuations of statements such as "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." in review strategy tab are incorrect.
• Solution: This view has been fixed by making changes in the respective files.
• The changes were made in the file : app/views/assignments/edit/_review_strategy.html.erb

<%= label_tag('assignment_form[assignment][num_reviews_required]', 'Set required number of reviews per reviewer') %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_required]', @assignment_form.assignment.num_reviews_required ||= @assignment_form.assignment.num_reviews, size: 1) %>

    <!--Adding info button-->
    <img src="/assets/info.png" title='This is the minimum number of reviews a reviewer has to perform'>
  <br/>
    <br/>
    <%= check_box_tag('review', 'true', true, {:onChange => 'hasReviewChanged()'}) %>
    <%= label_tag('review', 'Has max review limit?') %>  
    <span id="reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_reviews_allowed]', ' Set allowed number of reviews per reviewer') %>
      <%if @assignment_form.assignment.num_reviews_allowed && @assignment_form.assignment.num_reviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]', @assignment_form.assignment.num_reviews_allowed, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]',  3, size: 1) %>
      <%end%>
      <!--Adding info button-->
      <img src="/assets/info.png" title='This is the maximum number of reviews a reviewer can perform'<%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    </span>

  </td>
    <td width="50%">

    <%= label_tag('assignment_form[assignment][num_metareviews_required]', 'Set required number of meta-reviews per reviewer') %>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_required]', @assignment_form.assignment.num_metareviews_required ||= @assignment_form.assignment.num_metareviews_allowed, size: 1) %>
   <br/>
      <br/>

    <%= check_box_tag('meta_review', 'true', true, {:onChange => 'hasMetaReviewChanged()'}) %>
    <%= label_tag('meta_review', 'Has meta-review limit?') %>  
    <span id="meta_reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_metareviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_metareviews_allowed]', ' Set allowed number of meta-reviews per reviewer') %>
      <%if @assignment_form.assignment.num_metareviews_allowed && @assignment_form.assignment.num_metareviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]', @assignment_form.assignment.num_metareviews_allowed ||= 3, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]',  3, size: 1) %>
      <%end%>
    </span>

  </td>
</tr>

• Problem 5: the assignment of work to review to a reviewer depends on the number of reviews it has previously received, if that work has been assigned to a reviewer previously and its review has not been submitted then the system will still consider it such that it would not assign the same work to another reviewer if it has enough number of reviewers previously assigned to review it. This might lead to a scenario where that work was assigned to enough number of reviewers, but it does not receive enough reviews and cannot be assigned to any new reviewers.
• Solution: This implementation has been fixed. The system now considers only the reviews that were submitted by the reviewers for that work so that every submission gets assigned to reviewers until it receives its allowed number of reviews.

• the modified file is app/models/review_assignment.rb in the method reject_by_max_reviews_per_submission

def reject_by_max_reviews_per_submission(contributor_set)
  contributor_set.reject! {|contributor| contributor.responses.reject {|response|  !response.is_submitted }.count >= max_reviews_per_submission }
  contributor_set
end

Modified File

• app/views/assignments/edit/_review_strategy.html.erb
• app/models/review_assignment.rb
• app/views/student_review/list.html.erb

References

1. Expertiza on GitHub
2. GitHub Project Repository Fork
3. The live Expertiza website
4. Expertiza project documentation wiki
5. Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin