CSC/ECE 517 Spring 2014/oss E1402 mmb

Background

There are three different review strategies in Expertiza.

1. Auto Selected
2. Student Selected
3. Instructor Selected

In auto-selected reviewing, a reviewer picks the topic to review on, but not the individual submission. Student-selected allows the reviewer to pick from a list of submissions, not topics. Instructor-selected is where the instructor/TA assigns submissions to reviewers. In the case of auto-selected and student-selected reviewing, the student is not allowed to pick from any topic or submission, but only to choose among those that have received the fewest reviews so far. But this strategy may be modified by selecting a non-zero threshold k as follows. A topic is review able if the minimum number of reviews already done for the submissions on that topic is within k of the minimum number of reviews done on the least-reviewed submission on any topic.

Motivation

The goal of Expertiza is to support student-generated content through peer review and teamwork. It is a peer-review system that allows students to select a topic to work on, then submit their work and review their peers' submissions. As mentioned above there are different strategies available for reviewing other student;s work. However these strategies do not work as expected. It allowed students to choose many topics without submitting them. In some cases this causes exceeding the maximum limit for the assignment. Also allowing only the topic/submission with the minimum reviews to be allowed for review, did not allow users to select a topic which has a little more reviews than the minimum but has not crossed the maximum threshold. Certain changes in the algorithm for making a topic/submission allowable for review can help us overcome these problem.

Design

There were few faulty designs in the project which we have improvised.

1. The the view - _set_self_review was related to student review technique. However its name was confusing and misleading. We have refactored and replaced it with _set_student_review.
2. Ability to set maximum threshold for review for an assignment should be available for Auto Review and Student Review and not for Instructor Review. The existing design was faulty and we have made the required change so that this field is available only for student review and auto review. Appropriate changes are done in the jQuery method to implement this design change.
3. If no submission was available, still the topic was made available to the user and on selection of the topic, he was provided the message that no submission is available to review. The user then had to go back and select some other topic if available. This process was time consuming. We have replaced this design using Javascript and make it work faster. If the topic had no submissions for review, the topic was not presented to the user.
4. If the assignment was such that it had no topics, the user was directed to a page with a single button which further directed him to the page which listed available submissions for review. Although there is nothing wrong in this approach, the intermediate page was redundant and not performing any task. Thus we have replaced this design and the the intermediate page is bypassed.
5. The existing design had no restriction on the number of reviews the student is allowed to begin without submitting. This design had a lot of problem. One problem was as mentioned before it would evade the maximum number of reviews. Another problem it caused was some students were unable to perform reviews as others performed too many reviews. We have devise an improved strategy which overcomes these drawbacks.

Implementation

Objective 1

Update code in for Student Selected Reviewing to cycle through submissions and not topics.

Objective 2

Make the ‘threshold box’ on the Edit /Create assignment - Review Strategy tab to appear only when Auto-selected or Student-selected reviewing is chosen and not when Instructor-selected is chosen.

Objective 3

Allowing a student to choose any number of reviews to do, without submitting any of them, allows a student to work on multiple reviews at the same time, but it also allows students to evade the limit on number of reviews that can be done. Devise a strategy to introduce certain threshold so that the student does not evades this limit.

Objective 4

If there are no topics, Auto-selected reviewing should still work, but it should select one of the submissions that has the fewest reviews so far. Perform Testing for this.

Objective 5

If the reviewer selects his own submission for review, he is not allowed to perform the review and presented with message saying “ There are no more submissions to review on that topic.” Instead it would be better not to allow him to select his own submission for review. Also, if the user has reviewed the submission/topic with the least reviews, he should be allowed to perform review on other topics which have not reached their maximum threshold.

Objective 6

Provide a comprehensive suite of tests so that we can be sure that all review strategies are working.

Issues

Initial Setup

There were many issues faced in the process of fixing the review strategies. The issues are as follos:

• Unsuccessful bundler install

We faced the issue of raspell.c, which is the dictionary for auto-completing the English words. This was resolved by commenting out the following gems from the gem file gem 'automated_metareview', github: 'expertiza/automated_metareview' gem 'raspell'

• Databases for Expertiza project

We initially had to set up several databases like student, instructor, assignments, review rubrics. This was resolved with the help of 'expertiza_scrubbed' versions provided by the instructor and TA.

• Troubleshooting the Expertiza project.

We had troubles to run the database dump. Constantly rake db:migrate was failing, tree display was not working and we were always prompted with "No_method" error. This was again resolved by completely replacing the existing database-dump

Core tasks

• Permission setting for review strategy

Fistly, we tried to understand the working of the present code. We created a new assignment with 5 topics. We logged in as a student to select the topics, but we were prompted with the 'Permission Denied to sign_up_sheet#add_signup_topics'. This was resolved by changing the ation_allowed method of sign_up_sheet_controller

Future Work

1. Fix control flow of student-selected reviewing similar/through auto-selected reviewing. Currently, student-selected reviewing calls review_assignment() method in DynamicReviewAssignmentHelper which filters topics/submissions similar to auto-selected but not as efficiently. As per Professor's comments, student-selected reviewing process is the same as auto-selected except steps 3 and 5 are not needed, and in step 2, the check is only for whether they have submitted. Steps are as follows:
   1. A contributor set is initialized with all of the teams participating in the assignment.
   2. Contributors are removed if they have not selected a topic or not submitted.
   3. Contributors are removed if their topic is not reviewable at the current time.
   4. Contributors are removed if they have “too many” reviews (as determined by the review threshold).
   5. Then we cycle through the remaining contributors, adding their topics to candidate_topics set.
2. Logic to filter candidates is repeated in candidate_topics_to_review() and contributor_to_review() methods. We could skip filtering in the contributor_to_review() method.
3. For student-selected assignments with no topics, a single button is displayed to proceed to submission-selection page. We have currently implemented it using JavaScript to call the click() function of this button. This could be implemented in a better way by calling the next controller action if no topics are present.
4. The view views/student_review/_response.html.erb contains a lot of procedural code. This should ideally be moved from view to controller.