CSC/ECE 517 Fall 2017/E1777 Coherent specification of review requirements.rb

E1777. OSS Project Green: Coherent specification of review requirements

This page provides a description of the Expertiza based OSS project.

About Expertiza

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

Problem Statement

The following tasks were accomplished in this project:

• Information buttons to elaborate difference between the synonymous statements present in review strategy tab need to be included.
• The punctuation, syntax and capitalization of statements in the review strategy tab requires improvement.
• Message to a reviewer regarding how many reviews they are required to and allowed to perform is not clear enough.
• Reviewers are assigned new reviews based on number of previous reviews assigned for that submission, which is incorrect.

Implementation

Drawbacks and Solutions

Issue #402 : To allow reviewers a larger set of topics to choose from, the instructor can set the threshold (on the Review Strategy tab of assignment creation/editing) to some integer k > 0. Then any submission that has within k reviews of the fewest number can be chosen by a new reviewer. Let’s say that all submissions have at least 1 review. If k = 3, then the reviewer can choose any topic where there is a submission that has 4 or fewer reviews so far. Suppose that the minimum number of reviews for any submission is 2, but that I have reviewed all the submissions that have only 2 reviews. Then I’m not allowed to review at all (unless k > 0).

• Solution: This issue has been fixed previously. The reviewer will get assigned a submission even if it has fulfilled the required number of reviews, to ensure that the reviewer always receives a new submission to review as long as it is not his own.

Issue #969:

• a.The two statements "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." are not well differentiated. It can very confusing for an instructor to give values for them.

• Solution: Information buttons have been provided beside these statements in the review strategy tab to make sure that the instructor knows the difference between the two.

This modification is made in the app/views/assignments/edit/_review_strategy.html.erb

<!--Adding info button-->
    <img src="/assets/info.png" title='This is the minimum number of reviews a reviewer has to perform'>
  <br/>
    <br/>
    <%= check_box_tag('review', 'true', true, {:onChange => 'hasReviewChanged()'}) %>
    <%= label_tag('review', 'Has max review limit?') %>  
    <span id="reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_reviews_allowed]', ' Set allowed number of reviews per reviewer') %>
      <%if @assignment_form.assignment.num_reviews_allowed && @assignment_form.assignment.num_reviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]', @assignment_form.assignment.num_reviews_allowed, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]',  3, size: 1) %>
      <%end%>
      <!--Adding info button-->
      <img src="/assets/info.png" title='This is the maximum number of reviews a reviewer can perform'<%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    </span>

  </td>

• b) When the number of allowed or required reviews is not set on the Review Strategy tab, the system does not have a message to display to a reviewer about how many submissions of work they are required to and allowed to review depending on the values set by the instructor in the "allowed number of reviewers per reviewer" field and the "Set Required Number of Reviews per reviewer." field.

• Solution: This view has been implemented in the app/views/student_review/list.html.erb

<h2>Reviews for "<%= @assignment.name %>"</h2>

<!--Changing statements to display on the others work screen-->
<% if @assignment.num_reviews_allowed.nil? || @assignment.num_reviews_allowed == -1%>
    <h4>Your instructor expects you to do <%= @assignment.num_reviews_required %> reviews. You are not allowed to do any extra reviews. </h4>

<% elsif @assignment.num_reviews_allowed == @assignment.num_reviews_required %>
    <h4>You should perform exactly <%= @assignment.num_reviews_allowed %> reviews </h4>
<% else %>

    <h4>You may perform between <%= @assignment.num_reviews_required %> and <%= @assignment.num_reviews_allowed %> reviews</h4>
<% end %>

• c)The capitalization and punctuation of statements such as "Set Allowed Number of Reviews per reviewer" and "Set Required Number of Reviews per reviewer." in review strategy tab are incorrect.
• Solution: This view has been fixed by making changes in the respective files.

The changes were made in the file : app/views/assignments/edit/_review_strategy.html.erb

<%= label_tag('assignment_form[assignment][num_reviews_required]', 'Set required number of reviews per reviewer') %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_required]', @assignment_form.assignment.num_reviews_required ||= @assignment_form.assignment.num_reviews, size: 1) %>

    <!--Adding info button-->
    <img src="/assets/info.png" title='This is the minimum number of reviews a reviewer has to perform'>
  <br/>
    <br/>
    <%= check_box_tag('review', 'true', true, {:onChange => 'hasReviewChanged()'}) %>
    <%= label_tag('review', 'Has max review limit?') %>  
    <span id="reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_reviews_allowed]', ' Set allowed number of reviews per reviewer') %>
      <%if @assignment_form.assignment.num_reviews_allowed && @assignment_form.assignment.num_reviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]', @assignment_form.assignment.num_reviews_allowed, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_reviews_allowed]',  3, size: 1) %>
      <%end%>
      <!--Adding info button-->
      <img src="/assets/info.png" title='This is the maximum number of reviews a reviewer can perform'<%= 'hidden' if @assignment_form.assignment.num_reviews_allowed.nil?%>>
    </span>

  </td>
    <td width="50%">

    <%= label_tag('assignment_form[assignment][num_metareviews_required]', 'Set required number of meta-reviews per reviewer') %>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_required]', @assignment_form.assignment.num_metareviews_required ||= @assignment_form.assignment.num_metareviews_allowed, size: 1) %>
   <br/>
      <br/>

    <%= check_box_tag('meta_review', 'true', true, {:onChange => 'hasMetaReviewChanged()'}) %>
    <%= label_tag('meta_review', 'Has meta-review limit?') %>  
    <span id="meta_reviews_allowed" <%= 'hidden' if @assignment_form.assignment.num_metareviews_allowed.nil?%>>
    <%= label_tag('assignment_form[assignment][num_metareviews_allowed]', ' Set allowed number of meta-reviews per reviewer') %>
      <%if @assignment_form.assignment.num_metareviews_allowed && @assignment_form.assignment.num_metareviews_allowed >= 0 %>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]', @assignment_form.assignment.num_metareviews_allowed ||= 3, size: 1) %>
      <%else%>
    <%= text_field_tag('assignment_form[assignment][num_metareviews_allowed]',  3, size: 1) %>
      <%end%>
    </span>

  </td>
</tr>

• Issue #965: The assignment of work to review to a reviewer depends on the number of reviews it has previously received, if that work has been assigned to a reviewer previously and its review has

not been submitted then the system will still consider it such that it would not assign the same work to another reviewer if it has enough number of reviewers previously assigned to review it. This might lead to a scenario where that work was assigned to enough number of reviewers, but it does not receive enough reviews and cannot be assigned to any new reviewers either.

• Solution: This implementation has been fixed. The system now considers only the reviews that were submitted by the reviewers for that work so that every submission gets assigned to reviewers until

it receives its allowed number of reviews.

The modified file is app/models/review_assignment.rb and the modification is done in the method reject_by_max_reviews_per_submission

def reject_by_max_reviews_per_submission(contributor_set)
  contributor_set.reject! {|contributor| contributor.responses.reject {|response|  !response.is_submitted }.count >= max_reviews_per_submission }
  contributor_set
end

Test Plans

Issue #402 :This is shown graphically with dummy topics and assignment. We create a dummy assignment with two different topics oss1 and oss2. oss1 has 2 slots and oss2 has 1 slot. So when a student who has submitted oss2 tries to review others work he will not be able to select oos2 at all because he cannot review his own submission. When a person who has oss1 as topic goes to this page and he asks to get a new submission for review it will give another slot’s oss1. If he does it again it says there are no more submissions available to review for this topic. So if he has already reviewed all the other submissions of his topic oss1 and he requests for one more submission of oss1 , he will not get any more submission to review because the only one left is his own submission.

Issue #969 :Set allowed number and required number of reviews per submission as 3 and 3 respectively. Now it will display "you should perform exactly 3 reviews."

Before:

After:

Issue #965 :We created a dummy assignment with a team size of 2, 3 such teams are created. Maximum allowed reviews is 2 now. Previously for a submission after 2 people begin review its does not give any further submission to review to that user because it already reached the limit.

On changing the method, when we try to get a new submission we receive one. This is because the previous reviewers have only begun their reviews but have not submitted them yet.

Now when the previous two reviewers submit the reviews they began. Now the same new reviewer will not be able to submit his review

Modified Files

• app/views/assignments/edit/_review_strategy.html.erb
• app/models/review_assignment.rb
• app/views/student_review/list.html.erb

References

1. Expertiza on GitHub
2. GitHub Project Repository Fork
3. The live Expertiza website
4. Expertiza project documentation wiki
5. Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin