CSC/ECE 517 Spring 2014/oss E1402 mmb

Background

There are three different review strategies in Expertiza.

1. Auto Selected
2. Student Selected
3. Instructor Selected

In auto-selected reviewing, a reviewer picks the topic to review on, but not the individual submission. Student-selected allows the reviewer to pick from a list of submissions, not topics. Instructor-selected is where the instructor/TA assigns submissions to reviewers. In the case of auto-selected and student-selected reviewing, the student is not allowed to pick from any topic or submission, but only to choose among those that have received the fewest reviews so far. But this strategy may be modified by selecting a non-zero threshold k as follows. A topic is review able if the minimum number of reviews already done for the submissions on that topic is within k of the minimum number of reviews done on the least-reviewed submission on any topic.

Motivation

The goal of Expertiza is to support student-generated content through peer review and teamwork. It is a peer-review system that allows students to select a topic to work on, then submit their work and review their peers' submissions. As mentioned above there are different strategies available for reviewing other student;s work. However these strategies do not work as expected. It allowed students to choose many topics without submitting them. In some cases this causes exceeding the maximum limit for the assignment. Also allowing only the topic/submission with the minimum reviews to be allowed for review, did not allow users to select a topic which has a little more reviews than the minimum but has not crossed the maximum threshold. Certain changes in the algorithm for making a topic/submission allowable for review can help us overcome these problem.

Design

There were few faulty designs in the project which we have improvised.

1. The the view - _set_self_review was related to student review technique. However its name was confusing and misleading. We have refactored and replaced it with _set_student_review.
2. Ability to set maximum threshold for review for an assignment should be available for Auto Review and Student Review and not for Instructor Review. The existing design was faulty and we have made the required change so that this field is available only for student review and auto review. Appropriate changes are done in the jQuery method to implement this design change.
3. If no submission was available, still the topic was made available to the user and on selection of the topic, he was provided the message that no submission is available to review. The user then had to go back and select some other topic if available. This process was time consuming. We have replaced this design using Javascript and make it work faster. If the topic had no submissions for review, the topic was not presented to the user.
4. If the assignment was such that it had no topics, the user was directed to a page with a single button which further directed him to the page which listed available submissions for review. Although there is nothing wrong in this approach, the intermediate page was redundant and not performing any task. Thus we have replaced this design and the the intermediate page is bypassed.
5. The existing design had no restriction on the number of reviews the student is allowed to begin without submitting. This design had a lot of problem. One problem was as mentioned before it would evade the maximum number of reviews. Another problem it caused was some students were unable to perform reviews as others performed too many reviews. We have devise an improved strategy which overcomes these drawbacks.

Implementation

Objective 1

Update code in for Student Selected Reviewing to cycle through submissions and not topics.
> With the logical path, it is observed that student-selected reviewing cycles through submission. If there are topics to the a particular assignment, it is logical to make the user to first select the assignment, cycle through submissions for that particular assignment and display the same.

Objective 2

Make the ‘threshold box’ on the Edit /Create assignment - Review Strategy tab to appear only when Auto-selected or Student-selected reviewing is chosen and not when Instructor-selected is chosen.

> The Maximum Threshold Box was getting shown using JQuery. However the logic used to display was not proper. We modified the code in the jQuery so that the required implementation ie to show the box for Auto Selected and Student Selected and not for Instructor Selected.
views>assignments>edit>general.html.erb>reviewStrategyChanged() (called in _review_strategy.html.erb)

function reviewStrategyChanged() {
     var val = jQuery('#assignment_review_assignment_strategy').val();
        if (val == 'Student-Selected' || val == 'Auto-Selected') {
          jQuery('#assignment_review_topic_threshold_row').removeAttr('hidden');
        } else {
          jQuery('#assignment_review_topic_threshold_row').attr('hidden', true);
        }
   }

> We also removed JavaScript functions repeated twice in this view.

Objective 3

Allowing a student to choose any number of reviews to do, without submitting any of them, allows a student to work on multiple reviews at the same time, but it also allows students to evade the limit on number of reviews that can be done. Devise a strategy to introduce certain threshold so that the student does not evades this limit.

> We implemented the following strategy in out implementation. Introduction of a fixed threshold value - '3' which limits the number of reviews the user can begin without submitting previously selected reviews. Although this value can be made dynamic depending on the instructor or the assignment, we chose to have it static because it is not dependent on any assignment and would be a good practice for restricting the user at a static value for every assignment. We have calculated this condition in the list action of student_review_controller.rb and checked it on page student_review/list.html.erb. This will help in limiting the evasion of maximum threshold.

<% if @num_reviews_in_progress < 3 %>
            <% if @assignment.review_assignment_strategy == Assignment::RS_AUTO_SELECTED %>
                <%= render :partial => 'set_dynamic_review', :locals => {:assignment => @assignment} %>
            <% elsif @assignment.review_assignment_strategy == Assignment::RS_STUDENT_SELECTED %>
                <%= render :partial => 'set_student_review', :locals => {:assignment => @assignment} %>
            <% end %>
        <% else%>
        <br><br>
        <span>Note: You can't have more than 3 outstanding review. Complete existing reviews.</span>
        <% end %>

Also we have introduced code to reject user's own submission and submissions already reviewed by user so that user has right choice of submission to review every time mentioned in Objective 5.

Objective 4

If there are no topics, Auto-selected reviewing should still work, but it should select one of the submissions that has the fewest reviews so far. Perform Testing for this.

> Successfully tested with threshold zero and it works as expected.

Objective 5

If the reviewer selects his own submission for review, he is not allowed to perform the review and presented with message saying “ There are no more submissions to review on that topic.” Instead it would be better not to allow him to select his own submission for review. Also, if the user has reviewed the submission/topic with the least reviews, he should be allowed to perform review on other topics which have not reached their maximum threshold.

Objective 6

Provide a comprehensive suite of tests so that we can be sure that all review strategies are working.

Issues

Initial Setup

There were many issues faced in the process of fixing the review strategies. The issues are as follos:

• Unsuccessful bundler install

We faced the issue of raspell.c, which is the dictionary for auto-completing the English words. This was resolved by commenting out the following gems from the gem file gem 'automated_metareview', github: 'expertiza/automated_metareview' gem 'raspell'

• Databases for Expertiza project

We initially had to set up several databases like student, instructor, assignments, review rubrics. This was resolved with the help of 'expertiza_scrubbed' versions provided by the instructor and TA.

• Troubleshooting the Expertiza project.

We had troubles to run the database dump. Constantly rake db:migrate was failing, tree display was not working and we were always prompted with "No_method" error. This was again resolved by completely replacing the existing database-dump

Core tasks

• Permission setting for review strategy

Fistly, we tried to understand the working of the present code. We created a new assignment with 5 topics. We logged in as a student to select the topics, but we were prompted with the 'Permission Denied to sign_up_sheet#add_signup_topics'. This was resolved by changing the ation_allowed method of sign_up_sheet_controller

• Missing partials

There were some of the partials missing in the given project code like the submitted_content/_self_review.html.

• Missing table attributes

There were missing some table attributes in the given database which resulted in "No_Method" error.We had to create a few attributes like created_at and updated_at in the ResponseMap table.

• Tweaking the code

There were some of the non-existing/irrelevant attributes accessed to perform the required action. These minor issues were resolved by tweaking the code and accessing the right attributes. The corresponding action were tested for correctness.

• Incomplete views

The student_review/_response.html view had incomplete condition code. We fixed the code manually by understanding and testing the code flow.

• Duplicate methods

There were present multiple definitions for the same function (e.g., new,edit) in response controller. We had to discard the method that was not referenced.

Testing

Testing was done with the Unit tests. Test cases were written for the corresponding models. The following commands were performed before writing the test cases:
rake db:test:clone
rake db:test:clone_structure
rake db:test:load
rake db:test:prepare
rake db:test:purge
Generally one can write test cases to check the following:

• Web request successful
• User redirected to the right page
• Successfully authenticated the user/object
• Correct object stored in response
• Appropriate message displayed for the user.
  

The following test cases were written for the review_strategies:

test_candidate_topics_to_review

A test case was written to test if candidate_topics_to_review returns nil if there are no topics to the assignment submission.

test_signup_topics

A test case was written to test if the existing signup_topic method in the expertiza code. The functionality of signup_topic method was tested:

signup_topics return non-empty set of values if it has topics for the assignment.

signup_topics returns a empty set of values if it has no topics for the assignment.

tests for candidate_topics_to_review

We re-factored the candiate_topics_to_review method in the assignment.rb model. contributor set is build according to a set of conditions. The modified candiate_topics_to_review is more modular and clean.

Test cases are written for each of the method that describes each condition.

Future Work

• Fix control flow of student-selected reviewing similar/through auto-selected reviewing. Currently, student-selected reviewing calls review_assignment() method in DynamicReviewAssignmentHelper which filters topics/submissions similar to auto-selected but not as efficiently. As per Professor's comments, student-selected reviewing process is the same as auto-selected except steps 3 and 5 are not needed, and in step 2, the check is only for whether they have submitted. Steps are as follows:
  • A contributor set is initialized with all of the teams participating in the assignment.
  • Contributors are removed if they have not selected a topic or not submitted.
  • Contributors are removed if their topic is not reviewable at the current time.
  • Contributors are removed if they have “too many” reviews (as determined by the review threshold).
  • Then we cycle through the remaining contributors, adding their topics to candidate_topics set.
• Logic to filter candidates is repeated in candidate_topics_to_review() and contributor_to_review() methods. We could skip filtering in the contributor_to_review() method.
• For student-selected assignments with no topics, a single button is displayed to proceed to submission-selection page. We have currently implemented it using JavaScript to call the click() function of this button. This could be implemented in a better way by calling the next controller action if no topics are present.
• The view views/student_review/_response.html.erb contains a lot of procedural code. This should ideally be moved from view to controller.