Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T21:56:58Z

Cmfoley2: /* Coverage */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

== '''[https://github.com/RPHolloway/expertiza/issues/32 #32 - Remove all debug comments in the UI]''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the model for an Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''[https://github.com/RPHolloway/expertiza/issues/27 #27 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database]''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Changes===

We created a migration that defaulted the metareviews_allowed and metareviews_required in the Assignment table to nil instead of what the previous team had as 3.

[[File:E2025_13.png]]

===Testing===

Tests have been created that test if a metareview can have nil limits. In this case, no limit will be shown to the user. Before this change that was impossible

== '''Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

===Changes===

The changes were too long to screenshot, so here is the specific diff section:

https://github.com/expertiza/expertiza/pull/1715/files#diff-540fd2bd6c34b48213716b1dd713733a
https://github.com/expertiza/expertiza/pull/1715/files#diff-80f39fa5f62c73ff8741b02d3b11656c

== '''Discovered Bugs:''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 0 reviewers. The user should NOT be able to see the 'Request a new meta-review to perform' button when there are no reviews available.
** with 1 reviewer.
*** A student should NOT be able to request a meta-review on their own review.
*** When the meta-review limit on the assignment is set to 1 then a student should NOT be able to request a second meta-review.
*** The user should not be able to see the 'Request a new meta-review to perform' button when they have performed all valid reviews.
** with 4 reviewers.
*** The user should be able to click the 'Begin' button and still see the 'Request a new meta-review to perform' button.
*** The user should NOT be able to see the 'Request a new meta-review to perform' button when they have reached the meta-review limit.
*** If the user requests two meta-reviews but has not submitted, then they should not be able to request a new meta-review.
*** The user should be able to request a meta-review if they are above their required but below the allowed number of meta-reviews.

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 1 reviewer.
*** The user should see the number of meta-reviews left decrement after a new meta-review is requested.
*** The user should see the number of meta-reviews left decrement after a new meta-review is completed.

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. All meta-review tests make meta-reviews allowed only during the meta-review round. This means that if the simple meta-review test is passing then this feature is working as intended.

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 1 reviewer.
*** The user is able to SUBMIT a simple meta-review.

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and then if the failing test case described below would be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

File:E2025 13.png

2020-04-25T21:55:29Z

Cmfoley2:

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T21:55:14Z

Cmfoley2: /* Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

== '''[https://github.com/RPHolloway/expertiza/issues/32 #32 - Remove all debug comments in the UI]''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the model for an Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''[https://github.com/RPHolloway/expertiza/issues/27 #27 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database]''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Changes===

We created a migration that defaulted the metareviews_allowed and metareviews_required in the Assignment table to nil instead of what the previous team had as 3.

[[File:E2025_13.png]]

===Testing===

Tests have been created that test if a metareview can have nil limits. In this case, no limit will be shown to the user. Before this change that was impossible

== '''Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

===Changes===

The changes were too long to screenshot, so here is the specific diff section:

https://github.com/expertiza/expertiza/pull/1715/files#diff-540fd2bd6c34b48213716b1dd713733a
https://github.com/expertiza/expertiza/pull/1715/files#diff-80f39fa5f62c73ff8741b02d3b11656c

== '''Discovered Bugs:''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 0 reviewers. The user should NOT be able to see the 'Request a new meta-review to perform' button when there are no reviews available.
** with 1 reviewer.
*** A student should NOT be able to request a meta-review on their own review.
*** When the meta-review limit on the assignment is set to 1 then a student should NOT be able to request a second meta-review.
*** The user should not be able to see the 'Request a new meta-review to perform' button when they have performed all valid reviews.
** with 4 reviewers.
*** The user should be able to click the 'Begin' button and still see the 'Request a new meta-review to perform' button.
*** The user should NOT be able to see the 'Request a new meta-review to perform' button when they have reached the meta-review limit.
*** If the user requests two meta-reviews but has not submitted, then they should not be able to request a new meta-review.
*** The user should be able to request a meta-review if they are above their required but below the allowed number of meta-reviews.

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 1 reviewer.
*** The user should see the number of meta-reviews left decrement after a new meta-review is requested.
*** The user should see the number of meta-reviews left decrement after a new meta-review is completed.

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. All meta-review tests make meta-reviews allowed only during the meta-review round. This means that if the simple meta-review test is passing then this feature is working as intended.

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 1 reviewer.
*** The user is able to SUBMIT a simple meta-review.

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and the failing test case described below will be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T21:52:44Z

Cmfoley2: /* #27 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

== '''[https://github.com/RPHolloway/expertiza/issues/32 #32 - Remove all debug comments in the UI]''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the model for an Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''[https://github.com/RPHolloway/expertiza/issues/27 #27 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database]''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Changes===

We created a migration that defaulted the metareviews_allowed and metareviews_required in the Assignment table to nil instead of what the previous team had as 3.

[[File:E2025_13.png]]

===Testing===

Tests have been created that test if a metareview can have nil limits. In this case, no limit will be shown to the user. Before this change that was impossible

== '''Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

== '''Discovered Bugs:''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 0 reviewers. The user should NOT be able to see the 'Request a new meta-review to perform' button when there are no reviews available.
** with 1 reviewer.
*** A student should NOT be able to request a meta-review on their own review.
*** When the meta-review limit on the assignment is set to 1 then a student should NOT be able to request a second meta-review.
*** The user should not be able to see the 'Request a new meta-review to perform' button when they have performed all valid reviews.
** with 4 reviewers.
*** The user should be able to click the 'Begin' button and still see the 'Request a new meta-review to perform' button.
*** The user should NOT be able to see the 'Request a new meta-review to perform' button when they have reached the meta-review limit.
*** If the user requests two meta-reviews but has not submitted, then they should not be able to request a new meta-review.
*** The user should be able to request a meta-review if they are above their required but below the allowed number of meta-reviews.

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 1 reviewer.
*** The user should see the number of meta-reviews left decrement after a new meta-review is requested.
*** The user should see the number of meta-reviews left decrement after a new meta-review is completed.

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. All meta-review tests make meta-reviews allowed only during the meta-review round. This means that if the simple meta-review test is passing then this feature is working as intended.

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

===Testing===
Functional tests from a student perspective are in /spec/features/meta_review_spec.rb. The tests assume the allowed number of meta-reviews is 5 and the required number of meta-reviews is 3. Tests for this feature are explained in summary below:
* For assignments that have a meta-review deadline...
** with 1 reviewer.
*** The user is able to SUBMIT a simple meta-review.

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and the failing test case described below will be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T20:24:21Z

Cmfoley2: /* Discovered Bugs */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

===Meta-Review Flow Design===

This diagram describes the interactions of the meta-review system.(From E1997)

[[File:E1997_1.PNG]]

== '''Issue 1 - Removing Debugging Comments''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the controller for Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''Issue 3 - Private Methods''' ==

===Issue Description===
Within the code for the meta-reviewing, it is noted that many statements could be moved to private methods to avoid conflating the purpose of the method with meta-reviewing specifics. We will use the code climate as a guide to reduce the block size and use existing or new private methods when possible to make the logic of the program easier to follow.

===Proposed Changes===
On first inspection, this will mainly apply to the code that is located in the Response Controller (response_conotroller.rb) and the view for "list.html.erb" under app/views/student_review.

For example: The "new" method has meta review checks that should probably be abstracted and placed as private methods.

We will also continue applying this to any changes that we make across the course of the project. This also will apply to the scripts that are used in the _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder. They should be extracted to somewhere else.

== '''Issue 4 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and confirm that "Use meta-review deadline" checkbox is unchecked

3. Use rails c to access the DB

4. Observe that all assignments contain the incorrect db value (3 for all) for Assignments in the db

===Proposed Change===

This will just require making a new migration in the Assignment database to set the default to NULL instead of 3, which we believe it is now on default. This may also bring up other issues in the code in the view by allowing assignments without meta-review limits, but we are unable to see that at this time.

file: schema.rb

create_table "assignments", force: :cascade do |t|
...
t.boolean "is_anonymous", default: true
t.integer "num_reviews_required", limit: 4, default: 3
t.integer "num_metareviews_required", limit: 4, default: '''NULL'''
t.integer "num_metareviews_allowed", limit: 4, default: '''NULL'''
t.integer "num_reviews_allowed", limit: 4, default: 3
t.integer "simicheck", limit: 4, default: -1
...
end

file: _review_strategy.html.erb

This file has all the logic to populate the view with a default value once the "Use meta-review deadline" is checked. No modifications needed.

== '''Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

== '''Discovered Bugs:''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

[https://github.com/RPHolloway/expertiza/issues/31 #31 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

[https://github.com/RPHolloway/expertiza/issues/29 #29 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and the failing test case described below will be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T20:24:15Z

Cmfoley2: /* #48 - Number of meta-reviews allowed for an assignment is always nil */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

===Meta-Review Flow Design===

This diagram describes the interactions of the meta-review system.(From E1997)

[[File:E1997_1.PNG]]

== '''Issue 1 - Removing Debugging Comments''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the controller for Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''Issue 3 - Private Methods''' ==

===Issue Description===
Within the code for the meta-reviewing, it is noted that many statements could be moved to private methods to avoid conflating the purpose of the method with meta-reviewing specifics. We will use the code climate as a guide to reduce the block size and use existing or new private methods when possible to make the logic of the program easier to follow.

===Proposed Changes===
On first inspection, this will mainly apply to the code that is located in the Response Controller (response_conotroller.rb) and the view for "list.html.erb" under app/views/student_review.

For example: The "new" method has meta review checks that should probably be abstracted and placed as private methods.

We will also continue applying this to any changes that we make across the course of the project. This also will apply to the scripts that are used in the _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder. They should be extracted to somewhere else.

== '''Issue 4 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and confirm that "Use meta-review deadline" checkbox is unchecked

3. Use rails c to access the DB

4. Observe that all assignments contain the incorrect db value (3 for all) for Assignments in the db

===Proposed Change===

This will just require making a new migration in the Assignment database to set the default to NULL instead of 3, which we believe it is now on default. This may also bring up other issues in the code in the view by allowing assignments without meta-review limits, but we are unable to see that at this time.

file: schema.rb

create_table "assignments", force: :cascade do |t|
...
t.boolean "is_anonymous", default: true
t.integer "num_reviews_required", limit: 4, default: 3
t.integer "num_metareviews_required", limit: 4, default: '''NULL'''
t.integer "num_metareviews_allowed", limit: 4, default: '''NULL'''
t.integer "num_reviews_allowed", limit: 4, default: 3
t.integer "simicheck", limit: 4, default: -1
...
end

file: _review_strategy.html.erb

This file has all the logic to populate the view with a default value once the "Use meta-review deadline" is checked. No modifications needed.

== '''Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

== '''Discovered Bugs''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

[https://github.com/RPHolloway/expertiza/issues/31 #31 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

[https://github.com/RPHolloway/expertiza/issues/29 #29 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and the failing test case described below will be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T20:23:54Z

Cmfoley2: /* Issue 6 - Meta Review Limit is Not Functioning Correctly in Assignment Creation */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

===Meta-Review Flow Design===

This diagram describes the interactions of the meta-review system.(From E1997)

[[File:E1997_1.PNG]]

== '''Issue 1 - Removing Debugging Comments''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the controller for Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''Issue 3 - Private Methods''' ==

===Issue Description===
Within the code for the meta-reviewing, it is noted that many statements could be moved to private methods to avoid conflating the purpose of the method with meta-reviewing specifics. We will use the code climate as a guide to reduce the block size and use existing or new private methods when possible to make the logic of the program easier to follow.

===Proposed Changes===
On first inspection, this will mainly apply to the code that is located in the Response Controller (response_conotroller.rb) and the view for "list.html.erb" under app/views/student_review.

For example: The "new" method has meta review checks that should probably be abstracted and placed as private methods.

We will also continue applying this to any changes that we make across the course of the project. This also will apply to the scripts that are used in the _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder. They should be extracted to somewhere else.

== '''Issue 4 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and confirm that "Use meta-review deadline" checkbox is unchecked

3. Use rails c to access the DB

4. Observe that all assignments contain the incorrect db value (3 for all) for Assignments in the db

===Proposed Change===

This will just require making a new migration in the Assignment database to set the default to NULL instead of 3, which we believe it is now on default. This may also bring up other issues in the code in the view by allowing assignments without meta-review limits, but we are unable to see that at this time.

file: schema.rb

create_table "assignments", force: :cascade do |t|
...
t.boolean "is_anonymous", default: true
t.integer "num_reviews_required", limit: 4, default: 3
t.integer "num_metareviews_required", limit: 4, default: '''NULL'''
t.integer "num_metareviews_allowed", limit: 4, default: '''NULL'''
t.integer "num_reviews_allowed", limit: 4, default: 3
t.integer "simicheck", limit: 4, default: -1
...
end

file: _review_strategy.html.erb

This file has all the logic to populate the view with a default value once the "Use meta-review deadline" is checked. No modifications needed.

== '''Issue 5 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

[https://github.com/RPHolloway/expertiza/issues/31 #31 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

[https://github.com/RPHolloway/expertiza/issues/29 #29 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and the failing test case described below will be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-25T20:23:47Z

Cmfoley2: /* Issue 5 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database */

== '''Overview of Project''' ==

In Expertiza, meta-review is a feature that enables a third party to evaluate the quality of feedback that a reviewer gives a reviewee. meta-review is an important feature that can help students become better reviewers for their peers by providing detailed feedback that is actionable and positively formulated. Unfortunately, this feature is broken and the following issues were identified. The task is to fix these issues so that this feature can be used again.

* An assignment is deemed to have meta-reviews if and only if there is a meta-review deadline for this assignment in the `assignments_questionnaires` table.
* If there is no meta-review deadline, then don’t show the question about whether there is a max meta-review limit.
* In the DB the default number of meta-reviews allowed and required should be null.
* As soon as someone checks the Has meta-review Limit box on the Review Strategy tab, the UI should fill in 3 and 3 as the required and allowed number.
* Student pages should show the list of reviews that have been performed.
* When a user requests a new meta-review, the field which says the number of meta-reviews left ( refer to above image “Numbers of Meta-reviews left:”) does not change and the button to request a new meta-review disappears.

===Previous Work===

This assignment is a repeat of E1997 which was assigned to students last semester, Fall 2019.

Details of the previous implementation can be found here:
[http://https://expertiza.csc.ncsu.edu/index.php/CSC/ECE_517_Fall_2019_-_E1997._Issues_related_to_meta-reviewing E1997]

Our goals are to update the logic, refactor non-DRY code, and fix bugs.

===Meta-Review Flow Design===

This diagram describes the interactions of the meta-review system.(From E1997)

[[File:E1997_1.PNG]]

== '''Issue 1 - Removing Debugging Comments''' ==

===Issue Description===

It was noted in the previous team's implementation that multiple debugging comments were showing up during their video testing.

Debug messages shown when viewing a completed meta review:
[[File:Debug_comments.png]]

===Changes===

There was a mistake in <code>app/views/response/_review.html.erb</code> where "<<nowiki/>br/><<nowiki/>hr/><<nowiki/>br/>" was being displayed under the review being meta-reviewed. This was due to being included as a string constant rather than html code.

[[File:E2025_debug_comments_response_review.PNG]]

There was an instance in <code>app/views/response/view.html.erb</code> where ruby code responsible for getting a review to be displayed was being interpreted and displayed, even though the response was being displayed as html in the line before. This resulted in the response details being listed in the UI erroneously.

[[File:E2025_debug_response_view.PNG]]

== '''[https://github.com/RPHolloway/expertiza/issues/49 #49 - "Request a new meta-review" button crashes when there are no available reviews]''' ==

===Issue Description===

When there are no valid metareviews to perform (either there are no reivews performed on the assignment, the only valid reviews concern your own work or own reviews, you have completed all allowed metareviews, or you have reviewed all valid reviews already) then the "Request a New Review" button still is visible and displays the following error (correctly) when pressed.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and check "Use meta-review deadline" checkbox

3. Select "Yes" for all submission allowed, Review allowed and Meta-review allowed in all of the deadline types

4. Navigate back to Assignment Tree View and click on "Add participant"

5. Add couple of students(say student1 and student2) to the assignment

6. Impersonate/login as student1 and make a submission

7. Log in as instructor6 as previously described and change the submission due date to be passed so we are currently in the review stage

8. Impersonate/login as student2 and Request for a new review under others work, submit the review

9. Log in as instructor6 as previously described and change the review due date to be passed so we are currently in the meta-review stage

10. Impersonate/login back as student1 (the one with the submission) and Request for a new meta review

[[File:E2025_outofreviews.png]]

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

Here we conditionally hide the button based on whether or not the participant has reviews available to perform. These checks are located in assignment.rb which is the controller for Assignment. The following method was added to assignment.rb and returns true when the participant has a valid meta-review available.

# Returns true if there is a meta-review task available for the user
def metareview_available?(user_id)
metareviewer = AssignmentParticipant.where(user_id: user_id, parent_id: self.id).first
begin
self.response_map_to_metareview(metareviewer)
true
rescue
false
end
end

== '''Issue 3 - Private Methods''' ==

===Issue Description===
Within the code for the meta-reviewing, it is noted that many statements could be moved to private methods to avoid conflating the purpose of the method with meta-reviewing specifics. We will use the code climate as a guide to reduce the block size and use existing or new private methods when possible to make the logic of the program easier to follow.

===Proposed Changes===
On first inspection, this will mainly apply to the code that is located in the Response Controller (response_conotroller.rb) and the view for "list.html.erb" under app/views/student_review.

For example: The "new" method has meta review checks that should probably be abstracted and placed as private methods.

We will also continue applying this to any changes that we make across the course of the project. This also will apply to the scripts that are used in the _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder. They should be extracted to somewhere else.

== '''Issue 4 - Defaulting Required and Allowed Meta-reviews to NULL in Assignment Database''' ==

===Issue Description===

Currently, the number of required and allowed meta-reviews is always being set to 3 in the database. Per the specs of the feature, it should default the database values to NULL. When checkbox "Use meta-review deadline", in the "Due Dates" tab, is selected then the view should populate required and allowed meta-reviews to 3.

===Reproducing Existing Issue===

1. Log in as instructor6, click on mange...-> Assignments

2. Click on edit assignment, navigate to Due Dates and confirm that "Use meta-review deadline" checkbox is unchecked

3. Use rails c to access the DB

4. Observe that all assignments contain the incorrect db value (3 for all) for Assignments in the db

===Proposed Change===

This will just require making a new migration in the Assignment database to set the default to NULL instead of 3, which we believe it is now on default. This may also bring up other issues in the code in the view by allowing assignments without meta-review limits, but we are unable to see that at this time.

file: schema.rb

create_table "assignments", force: :cascade do |t|
...
t.boolean "is_anonymous", default: true
t.integer "num_reviews_required", limit: 4, default: 3
t.integer "num_metareviews_required", limit: 4, default: '''NULL'''
t.integer "num_metareviews_allowed", limit: 4, default: '''NULL'''
t.integer "num_reviews_allowed", limit: 4, default: 3
t.integer "simicheck", limit: 4, default: -1
...
end

file: _review_strategy.html.erb

This file has all the logic to populate the view with a default value once the "Use meta-review deadline" is checked. No modifications needed.

== '''Issue 6 - Meta Review Limit is Not Functioning Correctly in Assignment Creation''' ==

===Issue Description===

The "has meta review limit" should determine whether a limit or requirement is placed on the student to complete meta reviews. If disabled, then they should be able to complete as many as they want. This is independent of the Meta Review STAGE that is enabled/disabled in the due dates tab. They should not impact each other and currently do within the assignment creation.

===Reproducing Existing Issue===

1. If the "Use meta-review deadline" is disabled, then the Meta Review Limit panel disappears

2. "Has meta-review limit?" is hidden on edit

3. Possibility the "Has meta-review limit?" not being saved on assignment creation?

4. If the "Use meta-review deadline" is toggled (disabled, enabled), then the "Has meta-review limit?" resets to disabled

===Proposed Changes===

The code in _due_dates.html.erb and the _review_strategy.html.erb within the /app/app/views/assignments/edit folder is lazily written and does not really adhere to the project specifications of how metareviews should work on assignment creation. These views need to be edited to work correctly and also respond to the changes in issue 5.

== '''[https://github.com/RPHolloway/expertiza/issues/48 #48 - Number of meta-reviews allowed for an assignment is always nil]''' ==

== '''[https://github.com/RPHolloway/expertiza/issues/45 #45 - "Request a new meta-review" button bugs]''' ==

===Issue Description===

The "Request new a meta-review" button should be hidden when:
# The number of meta-reviews in-progress is greater than the maximum allowed outstanding reviews.
# The number of reviews allowed is limited AND the total number of meta-reviews completed or in-progress is greater than the number allowed.
# There are no valid reviews available for the participant.

===Changes===

This issue deals with the view "list.html.erb" under /app/views/student_review. The code below shows the conditional used to display the "Request new meta-review" button.

<% if @num_metareviews_in_progress >= Assignment.max_outstanding_reviews %>
Note: You can't have more than 2 outstanding meta-reviews. You must complete one of your
outstanding meta-reviews before selecting another.
<% elsif @assignment.num_metareviews_allowed && @num_metareviews_total == @assignment.num_metareviews_allowed %>
Note: You can not do more than <%=@assignment.num_metareviews_allowed%> meta-reviews according
to assignment policy.
<% elsif (!@assignment.metareview_available?(session[:user].id)) %>
Meta-reviews cannot be performed at this time.
<% else %>
<%= render :partial => 'set_dynamic_metareview', :locals => {:assignment => @assignment} %>
<% end %>

[https://github.com/RPHolloway/expertiza/issues/31 #31 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/30 #30 - “Numbers of Meta-reviews left:” does not update after a user requests a review.]''' ==
===Issue Description===
Number of meta-reviews left should decrement when a user requests a meta-review. Current behavior only decremented when a review was submitted.

===Changes===

Previous behavior:

Number of Meta-Reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_completed %>

Fixed behavior:

Number of meta-reviews left: <%= @assignment.num_metareviews_allowed - @num_metareviews_total %>

== '''[https://github.com/RPHolloway/expertiza/issues/43 #43 - "Review Allowed" flag is required to be set on a due_date for a user to perform a meta-review.]''' ==
===Issue Description===
When an assignment with a meta-review is created the round that meta-reviews will be performed in should have the "Meta-review allowed" option set to yes. This should be the only requirement for making meta-reviews available in that round. The problem with the current implementation is that meta-reviews can only be performed in rounds where the "Review allowed" option is set.

[[File:E2025 Issue43.png]]

===Changes===
Code changes were applied to app/views/student_review/_responses.html.erb. Since this partial is shared by both reviews and meta-reviews in list.html.erb the title variable was used to determine if the begin option should be active or not.

Previous behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" && @assignment.can_review(topic_id) %>
Current behavior:
<% elsif @assignment.get_current_stage(topic_id) != "Complete" &&
((@assignment.can_review(topic_id) && title == "Review") ||
(@assignment.metareview_allowed(topic_id) && title == "Meta-review")) %>

[https://github.com/RPHolloway/expertiza/issues/29 #29 - Tests]

== '''[https://github.com/RPHolloway/expertiza/issues/50 #50 - The submit button is not available for meta-reviews.]''' ==
===Issue Description===
Users should see save and submit buttons as part of the meta-review form. These buttons should have the same behavior as the review form. The problem right now is there is not submit button for meta-reviews.
===Changes===
Code changes were made adding 'Metareview' to the conditional that determines if the submit button should be shown or not.

Previous behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review')) %>
New behavior:
<% if @title && ((@title.eql? 'Review') || (@title.eql? 'Self Review') || (@title.eql? 'Metareview'))%>
<%= submit_tag "Submit "+@title, :name => "Submit", :id => "Submit" %>
<img src="/assets/info.png" title='- SAVE will only store the peer-review as draft and the peer-review will not be displayed to the reviewee. After clicking SUBMIT, the peer-review will be displayed to reviewee, and you will not be able to update it any more.'>
<% end %>

== '''Testing''' ==

=== Manual Testing ===

1. Log in as super_administrator2. Password is "password". Click on mange...-> Assignments.

2. Create a new Assignment by clicking the "+" button. Give the assignment these fields: Name, Course and Submission Directory.

[[File:New_assignment_general.png]]

3. Go to "Due Dates" tab and check "Use meta-review deadline" and then set the due dates 1 into the FUTURE for each review round and meta-review. Leave everything else as default as shown below:

[[File:New_assignment_duedates.png]]

4. Go to the "Rubrics" tab and populate as shown below:

[[File:New_assignment_rubrics.png]]

5. OPTIONAL: Go to "Review strategy page and select "Has meta-review limit?" and set the number of allowed and required meta-reviews.

6. Once done, "Create" the assignment.

7. Go back to Assignments. manage...-> Assignments.

8. Find the newly created assignment and click on "Add participant". Add the students below:

[[File:add_participants.png]]

9. Go to manage...Impersonate User and impersonate "Student485"

10. Go to the newly created assignment and submit a link for the assignment.

11. Revert back to super_administrator2 and change the submission due date to a date in the PAST so we are currently in the review stage.

12. Impersonate student486 and request for a new review under others work, submit the review.

13. Revert back to super_administrator2 as previously described and change the review due date to be passed so we are currently in the meta-review stage.

14. Impersonate as student485 and Request for a new meta review.

15. Press "Begin" and complete the meta-review

16. When the review is complete, the "Begin" should disappear and be replaced with "View" and "Edit"

===Automated Test Plan===
1. If I create an assignment, then the number of meta-reviews is set to null.

2. If 'Use metareview deadline' is not enabled in the due dates tab, then the meta-review fields in the review strategy tab are not visible.

3. If the metareview limit on the assignment is set to 3, then a student will see they need to submit 3 meta reviews

4. If the metareview deadline is enabled, the assignment cannot be created without a metareview rubric

5. If the assignment created is in a stage that does not have metareview enabled, a metareview cannot be created

6. If the assignment does not have any legal meta reviews, then the "Request a new metareview to perform" button is not visible

7. A student should not be able to request a metareview if they have reached the limit of their allowed reviews

8. A student should be able to request a metareview if they are above their required but below their allowed reviews

9. A student should not be able to request a metareview about their own work.

10. A student should not be able to request a metareview about themselves.

11. If there is a valid metareview to perform related to two other students, then the metareview should be able to be performed.

12. If a student has requested two metareviewes but have not submitted it, then they should not be able to request a new metareview.

13. If the metareview limit on the assignment is set to 1 then a student should not be able to request a second meta review

===Coverage===
Current test coverage is 41.348%. Code coverage will increase based on the test plan above and the failing test case described below will be fixed.

Travis CI is currently failing because of:
Failures:
1) assignment function general tab should edit quiz number available to students
Failure/Error:
child_nodes.each do |node|
initialize_fnode_update_children(params, node, tmp_res)
end
NoMethodError:
undefined method `each' for "":String
# ./app/controllers/tree_display_controller.rb:209:in `children_node_ng'
# ./config/diagnostic.rb:11:in `call'
# ------------------
# --- Caused by: ---
# Capybara::CapybaraError:
# Your application server raised an error - It has been raised in your test code because Capybara.raise_server_errors == true
# /home/travis/build/expertiza/expertiza/vendor/bundle/ruby/2.2.0/gems/capybara-2.17.0/lib/capybara/session.rb:145:in `raise_server_error!'

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-20T21:20:42Z

Cmfoley2: /* Proposed Changes */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-20T21:03:56Z

Cmfoley2: /* Proposed Changes */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:13:15Z

Cmfoley2: /* Existing Design */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:11:27Z

Cmfoley2: /* Existing Design */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:11:14Z

Cmfoley2: /* Existing Design */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:10:57Z

Cmfoley2: /* Existing Design */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:09:31Z

Cmfoley2: /* Proposed Changes */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:07:54Z

Cmfoley2: /* Issue 6 - Meta Review Limit is Not Functioning Correctly in Assignment Creation */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:05:01Z

Cmfoley2: /* Proposed Changes */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:02:10Z

Cmfoley2: /* Issue 6 - Meta Review Limit Table is Not Functioning Correctly */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:01:23Z

Cmfoley2: /* Manual Testing */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:01:08Z

Cmfoley2: /* Manual Testing */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T02:00:48Z

Cmfoley2: /* Testing */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T01:56:26Z

Cmfoley2: /* Issue 5 - Defaulting Required and Allowed Meta-reviews to NULL in Database */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T01:54:12Z

Cmfoley2: /* Issue 5 - Defaulting Required and Allowed Meta-reviews to NULL in Database */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T01:51:38Z

Cmfoley2: /* Issue 4 - Creating Tests */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T01:42:12Z

Cmfoley2: /* Issue 3 - Private Methods */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T01:36:59Z

Cmfoley2: /* Issue 2 - Request a new Review Button */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T01:17:50Z

Cmfoley2: /* Proposed Changes */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-12T00:48:52Z

Cmfoley2: /* Issue 1 - Removing Debugging Comments */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:57:19Z

Cmfoley2: /* Reproducing Existing Issue */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:57:01Z

Cmfoley2: /* Reproducing Existing Issue */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:56:32Z

Cmfoley2: /* Reproducing Existing Issue */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:56:26Z

Cmfoley2: /* Issue Description */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:54:36Z

Cmfoley2:

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:30:47Z

Cmfoley2: /* Reproducing Existing Issue */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:28:36Z

Cmfoley2:

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-08T16:24:00Z

Cmfoley2: /* Issue Description */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-07T22:18:24Z

Cmfoley2: /* Manual Testing */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-07T22:17:13Z

Cmfoley2: /* Issue 2 - Change Update to Edit */

File:E2025 1.png

2020-04-07T22:17:03Z

Cmfoley2:

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-07T21:57:47Z

Cmfoley2: /* Automated Test Plan */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-07T01:56:56Z

Cmfoley2: /* Existing Design */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-07T01:56:41Z

Cmfoley2: /* Existing Design */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-06T16:52:02Z

Cmfoley2:

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-06T16:45:25Z

Cmfoley2: /* Reproducing Existing Issue */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-06T16:45:13Z

Cmfoley2: /* Manual Testing */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-06T16:43:57Z

Cmfoley2: /* Overview of Project */

CSC/ECE 517 Spring 2020 - E2025: Issues Related to Meta - Reviewing

2020-04-06T16:11:40Z

Cmfoley2: /* Intro */

CSC/ECE 517 Spring 2020 - Project E2011. Refactor assignment creation spec.rb

2020-03-26T15:27:00Z

Cmfoley2: /* Refactoring Process */

== E2011. Refactoring assignment_creation_spec.rb ==

This page provides a description of the Expertiza based OSS project.

=== Introduction ===
[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.
 

=== Problem Statement ===
The two major tasks for this project were the following:

* Refactor existing test cases to follow good coding practices. This includes variable names, following the Don't Repeat Yourself (DRY) principle, and more.
* Check coverage of assignment_creation_spec.rb and improve upon it through additional testing.

=== Current Implementation ===
The assignment_creation_spec file contains test cases that ensure proper functionality of an instructor's workflow in regards to the creation of assignments. An instructor, when logged into Expertiza, should be able to create new assignments where each assignment has it's own assigned parameters, rubrics, review strategies and due dates. At the start of this project, all logic and handling of testing this instructor workflow was handled within a single file. This single-file implementation leads to problems with maintenance, and such problems within this implementation can be abstracted into one of these forms:

*'''File is too long''': This file exceeds 250 lines of code, and should not.
*'''Block has too many lines''': Within the file, the code used to handle a certain operation is implemented with too many subroutines within one block.
*'''Similar/Exact block(s) of code found''': Across the file, there were duplicate/near-duplicate blocks of code being used.
*'''General syntax issues''': This could include unnecessary whitespace, use of deprecated classes or things of the like.

==Refactoring Process==
The first refactor that our team did was to extract methods from what otherwise would be duplicate code and large block sizes. The previous team had started this process, but we were able to follow it to completion. We also then moved all of these extracted methods into the helper class assignment_creation_helper.rb. Here is an example of extracted methods that the previous team had created:

<pre>
def get_questionnaire(finder_var = nil)
if finder_var.nil?
AssignmentQuestionnaire.find_by(assignment_id: @assignment.id)
else
AssignmentQuestionnaire.where(assignment_id: @assignment.id).where(questionnaire_id: get_selected_id(finder_var))
end
end

def get_selected_id(finder_var)
if finder_var == "ReviewQuestionnaire2"
ReviewQuestionnaire.find_by(name: finder_var).id
elsif finder_var == "AuthorFeedbackQuestionnaire2"
AuthorFeedbackQuestionnaire.find_by(name: finder_var).id
elsif finder_var == "TeammateReviewQuestionnaire2"
TeammateReviewQuestionnaire.find_by(name: finder_var).id
end
end

def fill_assignment_form
fill_in 'assignment_form_assignment_name', with: 'edit assignment for test'
select('Course 2', from: 'assignment_form_assignment_course_id')
fill_in 'assignment_form_assignment_directory_path', with: 'testDirectory1'
fill_in 'assignment_form_assignment_spec_location', with: 'testLocation1'
end

def assignment_creation_setup(privacy,name)
login_as("instructor6")
new_assignment_url = "/assignments/new?private=#{privacy}"
visit new_assignment_url

fill_in 'assignment_form_assignment_name', with: name
select('Course 2', from: 'assignment_form_assignment_course_id')
fill_in 'assignment_form_assignment_directory_path', with: 'testDirectory'
end
</pre>

The following is an example of how we were able to reduce block size and duplicate code with further extraction of methods into assignment_creation_helper.rb:

[[File:E2011RefactoringDiff.png]]

Another major refactor we did was to break up the one large 'describe' block in assignment_creation_spec, which contained all of the tests for that controller. Since assignment creation has many different pages, we created a separate file for testing each page or responsibility.

Beyond the creation of the helper file, assignment_creation_spec.rb was broken up into the following files:
*assignment_creation_dates_spec.rb
*assignment_creation_deadlines_spec.rb
*assignment_creation_general_tab_spec.rb
*assignment_creation_page_spec.rb
*assignment_creation_participants_spec.rb
*assignment_creation_review_strategy_spec.rb
*assignment_creation_rubrics_spec.rb
*assignment_creation_topics_spec.rb

In the relocation of assignment_creation_spec's functionality and creation of new constituent files, additional refactoring was done to ensure that trailing whitespace was deleted, files ended with a final newline character, and other unnecessary characters were removed. All of this was to remove any other problems that code climate detected.

=== Additional Code Climate Issues ===

There were a few things that code climate detected that we decided did not need to be refactored. The first was a complaint about blocks being too large, even after we divided the one large 'describe' statement into many smaller ones. This is because, in rspec, each test has its own 'it' block, but all of those must be grouped in a larger 'describe' block. I suggest that this be discussed with the teams who may refactor the other test files or have code climate to be configured to instead look just at the 'it' block size.

This is an diagram on how otherwise good ruby code may be flagged by code climate:
<pre>
'describe' block
'it' block
(23 lines of code) - Code climate: √
'it' block
(15 lines of code) - Code climate: √
'it' block
(11 lines of code) - Code climate: √
end - Code climate: X (23+15+11 = 49 lines)
</pre>

Additionally, code climate was suggesting to change DateTime to Date or Time. As assignments are due not only on a certain day but at a specific time and that this would require a large refactoring of the entire Expertiza system for tracking submission times, we decided it was beyond the scope of this refactor.

==Testing Plan==
As part of the refactoring, we fixed a few broken tests in the original assignments_controller_spec.rb and created additional ones to increase coverage. We used TDD to see which tests that were created were breaking and used that to drive our implementation of the controller.

Here is the extensive list of test cases handled

'''1. assignments_controller_spec.rb'''

-allows edit when an admin

-allows edit when an instructor

-does not allow edit when TA of course but not TA of current assignment

-allows new when admin/instructor/ta

-does not allow new when student

-#new creates a new AssignmentForm object and renders assignment#new page

-when assignment_form is saved successfully, it redirects to
assignment#edit page

-when assignment_form is not saved successfully, it renders assignment#new page

-when assignment has staggered deadlines, it shows a flash message and renders edit page

-during update when params does not have key :assignment_form, it shows a note flash message and redirects to tree_display#index page

-when assignment is not saved successfully, it shoes an error flash message and redirects to assignments#edit page

-when the timezone preference of current user is nil and assignment form updates attributes successfully, it shows an error message and redirects to assignments#edit page

-when the timezone preference of current user is not nil and assignment form updates attributes successfully, it shows an error message and redirects to assignments#edit page

-on action #show, it renders the #show page

-when new assignment id fetches successfully, it 'redirects to assignments#edit page

-when new assignment directory is same as old it should show an error and redirect to assignments#edit page

-when new assignment id does not fetch successfully, it shows an error flash message and redirects to assignments#edit page

-when assignment is deleted successfully, it shows a success flash message and redirects to tree_display#list page

-when assignment is not deleted successfully, it shows an error flash message and redirects to tree_display#list page

-when assignment is removed from course successfully, it removes assignment and redirects to tree_display#list page

-showing list submissions

(the following are the assignment_creation_spec.rb split into files by responsability)
'''2. assignment_creation_dates_spec.rb'''

-it is able to create assignment with a new late policy

-it is able to set the deadline for an assignment review

'''3. assignment_creation_deadlines_spec.rb'''

-it is able to set the deadline for an assignment review

'''4. assignment_creation_general_tab.rb'''

-it should edit assignment available to students

-it should edit quiz number available to students

-it should edit number of members per team

-it should edit review visible to all other reviewers

-it should check if checking calibration shows the tab

'''5. assignment_creation_page_spec.rb'''

-it is able to create a public assignment

-it is able to create with teams

-it is able to create with quiz

-it is able to create with staggered deadline

-it is able to create with review visible to all reviewers

-it is able to create public micro-task assignment

-it is able to create calibrated public assignment

-it is able to show tab review strategy

-it is able to show tab due deadlines

-it should set the deadline for an assignment review

-it is able to show tab rubrics

-it is able to show attributes in rubrics

-it should sets attributes for review strategy auto selects

-it should check to find if the assignment can be added to a course

'''6. assignment_creation_participants_spec.rb'''

-it should check to see if participants can be added

-it should display newly created assignment
'''
7. assignment_creation_review_strategy_tab.rb'''

-it should auto selects correct values

-it should set number of reviews by each student

'''8. assignment_creation_rubrics_strategy_tab.rb'''

-it should update review questionnaire

-it should update scored question dropdown

-it should update author feedback questionnaire

-it should update teammate review questionnaire

'''9. assignment_creation_topics_spec.rb'''

-it can edit topics properties

-it procedes without topics properties

-it can create new topics

-it can delete existing topic

-it hides topics tab when has topics is un-checked

==Code of Additional Rspec Tests Created==
Looking at the methods that were covered in assignments_controller.rb we decided to create the following additional test cases for assignment creation.

These are additions to the existing assignments_controller_spec.rb.

[[File:E2011NewTests.png]]

==RSpec Video ScreenCast Demos==
Links to RSpec screencasts for new test cases.
# [https://drive.google.com/open?id=1AIi3MtD_TtUN5F1k3kzgqIgi8ficz_Ar Assignment Creation General Tab]
# [https://drive.google.com/open?id=1VnH_4NSLBX1x7xeYnH1Zsy6XG_nWD7Vk Assignment Creation Topic]
# [https://drive.google.com/open?id=1jUQSgXjjI5jQBTWAkD8YGhYnYIIXPtnq Assignment Creation Page]

==Coverage==
92.86% for assignments_controller.rb

Overall coverage increased (+28.04%) to 52.334%

== Project Mentors ==
#Dr. Edward Gehringer (efg@ncsu.edu)
#Srujana Rachakonda (srachak@ncsu.edu)

== Team Members ==
#Colin Foley (cmfoley2@ncsu.edu)
#Ayush Khot (akhot@ncsu.edu)
#Christian Morris (cgmorris@ncsu.edu)

== References ==

#[https://github.com/expertiza/expertiza Expertiza on GitHub]

CSC/ECE 517 Spring 2020 - Project E2011. Refactor assignment creation spec.rb

2020-03-26T15:26:49Z

Cmfoley2: /* Testing Plan */

== E2011. Refactoring assignment_creation_spec.rb ==

This page provides a description of the Expertiza based OSS project.

=== Introduction ===
[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.
 

=== Problem Statement ===
The two major tasks for this project were the following:

* Refactor existing test cases to follow good coding practices. This includes variable names, following the Don't Repeat Yourself (DRY) principle, and more.
* Check coverage of assignment_creation_spec.rb and improve upon it through additional testing.

=== Current Implementation ===
The assignment_creation_spec file contains test cases that ensure proper functionality of an instructor's workflow in regards to the creation of assignments. An instructor, when logged into Expertiza, should be able to create new assignments where each assignment has it's own assigned parameters, rubrics, review strategies and due dates. At the start of this project, all logic and handling of testing this instructor workflow was handled within a single file. This single-file implementation leads to problems with maintenance, and such problems within this implementation can be abstracted into one of these forms:

*'''File is too long''': This file exceeds 250 lines of code, and should not.
*'''Block has too many lines''': Within the file, the code used to handle a certain operation is implemented with too many subroutines within one block.
*'''Similar/Exact block(s) of code found''': Across the file, there were duplicate/near-duplicate blocks of code being used.
*'''General syntax issues''': This could include unnecessary whitespace, use of deprecated classes or things of the like.

=== Refactoring Process ===
The first refactor that our team did was to extract methods from what otherwise would be duplicate code and large block sizes. The previous team had started this process, but we were able to follow it to completion. We also then moved all of these extracted methods into the helper class assignment_creation_helper.rb. Here is an example of extracted methods that the previous team had created:

<pre>
def get_questionnaire(finder_var = nil)
if finder_var.nil?
AssignmentQuestionnaire.find_by(assignment_id: @assignment.id)
else
AssignmentQuestionnaire.where(assignment_id: @assignment.id).where(questionnaire_id: get_selected_id(finder_var))
end
end

def get_selected_id(finder_var)
if finder_var == "ReviewQuestionnaire2"
ReviewQuestionnaire.find_by(name: finder_var).id
elsif finder_var == "AuthorFeedbackQuestionnaire2"
AuthorFeedbackQuestionnaire.find_by(name: finder_var).id
elsif finder_var == "TeammateReviewQuestionnaire2"
TeammateReviewQuestionnaire.find_by(name: finder_var).id
end
end

def fill_assignment_form
fill_in 'assignment_form_assignment_name', with: 'edit assignment for test'
select('Course 2', from: 'assignment_form_assignment_course_id')
fill_in 'assignment_form_assignment_directory_path', with: 'testDirectory1'
fill_in 'assignment_form_assignment_spec_location', with: 'testLocation1'
end

def assignment_creation_setup(privacy,name)
login_as("instructor6")
new_assignment_url = "/assignments/new?private=#{privacy}"
visit new_assignment_url

fill_in 'assignment_form_assignment_name', with: name
select('Course 2', from: 'assignment_form_assignment_course_id')
fill_in 'assignment_form_assignment_directory_path', with: 'testDirectory'
end
</pre>

The following is an example of how we were able to reduce block size and duplicate code with further extraction of methods into assignment_creation_helper.rb:

[[File:E2011RefactoringDiff.png]]

Another major refactor we did was to break up the one large 'describe' block in assignment_creation_spec, which contained all of the tests for that controller. Since assignment creation has many different pages, we created a separate file for testing each page or responsibility.

Beyond the creation of the helper file, assignment_creation_spec.rb was broken up into the following files:
*assignment_creation_dates_spec.rb
*assignment_creation_deadlines_spec.rb
*assignment_creation_general_tab_spec.rb
*assignment_creation_page_spec.rb
*assignment_creation_participants_spec.rb
*assignment_creation_review_strategy_spec.rb
*assignment_creation_rubrics_spec.rb
*assignment_creation_topics_spec.rb

In the relocation of assignment_creation_spec's functionality and creation of new constituent files, additional refactoring was done to ensure that trailing whitespace was deleted, files ended with a final newline character, and other unnecessary characters were removed. All of this was to remove any other problems that code climate detected.

=== Additional Code Climate Issues ===

There were a few things that code climate detected that we decided did not need to be refactored. The first was a complaint about blocks being too large, even after we divided the one large 'describe' statement into many smaller ones. This is because, in rspec, each test has its own 'it' block, but all of those must be grouped in a larger 'describe' block. I suggest that this be discussed with the teams who may refactor the other test files or have code climate to be configured to instead look just at the 'it' block size.

This is an diagram on how otherwise good ruby code may be flagged by code climate:
<pre>
'describe' block
'it' block
(23 lines of code) - Code climate: √
'it' block
(15 lines of code) - Code climate: √
'it' block
(11 lines of code) - Code climate: √
end - Code climate: X (23+15+11 = 49 lines)
</pre>

Additionally, code climate was suggesting to change DateTime to Date or Time. As assignments are due not only on a certain day but at a specific time and that this would require a large refactoring of the entire Expertiza system for tracking submission times, we decided it was beyond the scope of this refactor.

==Testing Plan==
As part of the refactoring, we fixed a few broken tests in the original assignments_controller_spec.rb and created additional ones to increase coverage. We used TDD to see which tests that were created were breaking and used that to drive our implementation of the controller.

Here is the extensive list of test cases handled

'''1. assignments_controller_spec.rb'''

-allows edit when an admin

-allows edit when an instructor

-does not allow edit when TA of course but not TA of current assignment

-allows new when admin/instructor/ta

-does not allow new when student

-#new creates a new AssignmentForm object and renders assignment#new page

-when assignment_form is saved successfully, it redirects to
assignment#edit page

-when assignment_form is not saved successfully, it renders assignment#new page

-when assignment has staggered deadlines, it shows a flash message and renders edit page

-during update when params does not have key :assignment_form, it shows a note flash message and redirects to tree_display#index page

-when assignment is not saved successfully, it shoes an error flash message and redirects to assignments#edit page

-when the timezone preference of current user is nil and assignment form updates attributes successfully, it shows an error message and redirects to assignments#edit page

-when the timezone preference of current user is not nil and assignment form updates attributes successfully, it shows an error message and redirects to assignments#edit page

-on action #show, it renders the #show page

-when new assignment id fetches successfully, it 'redirects to assignments#edit page

-when new assignment directory is same as old it should show an error and redirect to assignments#edit page

-when new assignment id does not fetch successfully, it shows an error flash message and redirects to assignments#edit page

-when assignment is deleted successfully, it shows a success flash message and redirects to tree_display#list page

-when assignment is not deleted successfully, it shows an error flash message and redirects to tree_display#list page

-when assignment is removed from course successfully, it removes assignment and redirects to tree_display#list page

-showing list submissions

(the following are the assignment_creation_spec.rb split into files by responsability)
'''2. assignment_creation_dates_spec.rb'''

-it is able to create assignment with a new late policy

-it is able to set the deadline for an assignment review

'''3. assignment_creation_deadlines_spec.rb'''

-it is able to set the deadline for an assignment review

'''4. assignment_creation_general_tab.rb'''

-it should edit assignment available to students

-it should edit quiz number available to students

-it should edit number of members per team

-it should edit review visible to all other reviewers

-it should check if checking calibration shows the tab

'''5. assignment_creation_page_spec.rb'''

-it is able to create a public assignment

-it is able to create with teams

-it is able to create with quiz

-it is able to create with staggered deadline

-it is able to create with review visible to all reviewers

-it is able to create public micro-task assignment

-it is able to create calibrated public assignment

-it is able to show tab review strategy

-it is able to show tab due deadlines

-it should set the deadline for an assignment review

-it is able to show tab rubrics

-it is able to show attributes in rubrics

-it should sets attributes for review strategy auto selects

-it should check to find if the assignment can be added to a course

'''6. assignment_creation_participants_spec.rb'''

-it should check to see if participants can be added

-it should display newly created assignment
'''
7. assignment_creation_review_strategy_tab.rb'''

-it should auto selects correct values

-it should set number of reviews by each student

'''8. assignment_creation_rubrics_strategy_tab.rb'''

-it should update review questionnaire

-it should update scored question dropdown

-it should update author feedback questionnaire

-it should update teammate review questionnaire

'''9. assignment_creation_topics_spec.rb'''

-it can edit topics properties

-it procedes without topics properties

-it can create new topics

-it can delete existing topic

-it hides topics tab when has topics is un-checked

==Code of Additional Rspec Tests Created==
Looking at the methods that were covered in assignments_controller.rb we decided to create the following additional test cases for assignment creation.

These are additions to the existing assignments_controller_spec.rb.

[[File:E2011NewTests.png]]

==RSpec Video ScreenCast Demos==
Links to RSpec screencasts for new test cases.
# [https://drive.google.com/open?id=1AIi3MtD_TtUN5F1k3kzgqIgi8ficz_Ar Assignment Creation General Tab]
# [https://drive.google.com/open?id=1VnH_4NSLBX1x7xeYnH1Zsy6XG_nWD7Vk Assignment Creation Topic]
# [https://drive.google.com/open?id=1jUQSgXjjI5jQBTWAkD8YGhYnYIIXPtnq Assignment Creation Page]

==Coverage==
92.86% for assignments_controller.rb

Overall coverage increased (+28.04%) to 52.334%

== Project Mentors ==
#Dr. Edward Gehringer (efg@ncsu.edu)
#Srujana Rachakonda (srachak@ncsu.edu)

== Team Members ==
#Colin Foley (cmfoley2@ncsu.edu)
#Ayush Khot (akhot@ncsu.edu)
#Christian Morris (cgmorris@ncsu.edu)

== References ==

#[https://github.com/expertiza/expertiza Expertiza on GitHub]

CSC/ECE 517 Spring 2020 - Project E2011. Refactor assignment creation spec.rb

2020-03-26T15:26:38Z

Cmfoley2: /* Code of Additional Rspec Tests Created */

== E2011. Refactoring assignment_creation_spec.rb ==

This page provides a description of the Expertiza based OSS project.

=== Introduction ===
[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.
 

=== Problem Statement ===
The two major tasks for this project were the following:

* Refactor existing test cases to follow good coding practices. This includes variable names, following the Don't Repeat Yourself (DRY) principle, and more.
* Check coverage of assignment_creation_spec.rb and improve upon it through additional testing.

=== Current Implementation ===
The assignment_creation_spec file contains test cases that ensure proper functionality of an instructor's workflow in regards to the creation of assignments. An instructor, when logged into Expertiza, should be able to create new assignments where each assignment has it's own assigned parameters, rubrics, review strategies and due dates. At the start of this project, all logic and handling of testing this instructor workflow was handled within a single file. This single-file implementation leads to problems with maintenance, and such problems within this implementation can be abstracted into one of these forms:

*'''File is too long''': This file exceeds 250 lines of code, and should not.
*'''Block has too many lines''': Within the file, the code used to handle a certain operation is implemented with too many subroutines within one block.
*'''Similar/Exact block(s) of code found''': Across the file, there were duplicate/near-duplicate blocks of code being used.
*'''General syntax issues''': This could include unnecessary whitespace, use of deprecated classes or things of the like.

=== Refactoring Process ===
The first refactor that our team did was to extract methods from what otherwise would be duplicate code and large block sizes. The previous team had started this process, but we were able to follow it to completion. We also then moved all of these extracted methods into the helper class assignment_creation_helper.rb. Here is an example of extracted methods that the previous team had created:

<pre>
def get_questionnaire(finder_var = nil)
if finder_var.nil?
AssignmentQuestionnaire.find_by(assignment_id: @assignment.id)
else
AssignmentQuestionnaire.where(assignment_id: @assignment.id).where(questionnaire_id: get_selected_id(finder_var))
end
end

def get_selected_id(finder_var)
if finder_var == "ReviewQuestionnaire2"
ReviewQuestionnaire.find_by(name: finder_var).id
elsif finder_var == "AuthorFeedbackQuestionnaire2"
AuthorFeedbackQuestionnaire.find_by(name: finder_var).id
elsif finder_var == "TeammateReviewQuestionnaire2"
TeammateReviewQuestionnaire.find_by(name: finder_var).id
end
end

def fill_assignment_form
fill_in 'assignment_form_assignment_name', with: 'edit assignment for test'
select('Course 2', from: 'assignment_form_assignment_course_id')
fill_in 'assignment_form_assignment_directory_path', with: 'testDirectory1'
fill_in 'assignment_form_assignment_spec_location', with: 'testLocation1'
end

def assignment_creation_setup(privacy,name)
login_as("instructor6")
new_assignment_url = "/assignments/new?private=#{privacy}"
visit new_assignment_url

fill_in 'assignment_form_assignment_name', with: name
select('Course 2', from: 'assignment_form_assignment_course_id')
fill_in 'assignment_form_assignment_directory_path', with: 'testDirectory'
end
</pre>

The following is an example of how we were able to reduce block size and duplicate code with further extraction of methods into assignment_creation_helper.rb:

[[File:E2011RefactoringDiff.png]]

Another major refactor we did was to break up the one large 'describe' block in assignment_creation_spec, which contained all of the tests for that controller. Since assignment creation has many different pages, we created a separate file for testing each page or responsibility.

Beyond the creation of the helper file, assignment_creation_spec.rb was broken up into the following files:
*assignment_creation_dates_spec.rb
*assignment_creation_deadlines_spec.rb
*assignment_creation_general_tab_spec.rb
*assignment_creation_page_spec.rb
*assignment_creation_participants_spec.rb
*assignment_creation_review_strategy_spec.rb
*assignment_creation_rubrics_spec.rb
*assignment_creation_topics_spec.rb

In the relocation of assignment_creation_spec's functionality and creation of new constituent files, additional refactoring was done to ensure that trailing whitespace was deleted, files ended with a final newline character, and other unnecessary characters were removed. All of this was to remove any other problems that code climate detected.

=== Additional Code Climate Issues ===

There were a few things that code climate detected that we decided did not need to be refactored. The first was a complaint about blocks being too large, even after we divided the one large 'describe' statement into many smaller ones. This is because, in rspec, each test has its own 'it' block, but all of those must be grouped in a larger 'describe' block. I suggest that this be discussed with the teams who may refactor the other test files or have code climate to be configured to instead look just at the 'it' block size.

This is an diagram on how otherwise good ruby code may be flagged by code climate:
<pre>
'describe' block
'it' block
(23 lines of code) - Code climate: √
'it' block
(15 lines of code) - Code climate: √
'it' block
(11 lines of code) - Code climate: √
end - Code climate: X (23+15+11 = 49 lines)
</pre>

Additionally, code climate was suggesting to change DateTime to Date or Time. As assignments are due not only on a certain day but at a specific time and that this would require a large refactoring of the entire Expertiza system for tracking submission times, we decided it was beyond the scope of this refactor.

=== Testing Plan ===
As part of the refactoring, we fixed a few broken tests in the original assignments_controller_spec.rb and created additional ones to increase coverage. We used TDD to see which tests that were created were breaking and used that to drive our implementation of the controller.

Here is the extensive list of test cases handled

'''1. assignments_controller_spec.rb'''

-allows edit when an admin

-allows edit when an instructor

-does not allow edit when TA of course but not TA of current assignment

-allows new when admin/instructor/ta

-does not allow new when student

-#new creates a new AssignmentForm object and renders assignment#new page

-when assignment_form is saved successfully, it redirects to
assignment#edit page

-when assignment_form is not saved successfully, it renders assignment#new page

-when assignment has staggered deadlines, it shows a flash message and renders edit page

-during update when params does not have key :assignment_form, it shows a note flash message and redirects to tree_display#index page

-when assignment is not saved successfully, it shoes an error flash message and redirects to assignments#edit page

-when the timezone preference of current user is nil and assignment form updates attributes successfully, it shows an error message and redirects to assignments#edit page

-when the timezone preference of current user is not nil and assignment form updates attributes successfully, it shows an error message and redirects to assignments#edit page

-on action #show, it renders the #show page

-when new assignment id fetches successfully, it 'redirects to assignments#edit page

-when new assignment directory is same as old it should show an error and redirect to assignments#edit page

-when new assignment id does not fetch successfully, it shows an error flash message and redirects to assignments#edit page

-when assignment is deleted successfully, it shows a success flash message and redirects to tree_display#list page

-when assignment is not deleted successfully, it shows an error flash message and redirects to tree_display#list page

-when assignment is removed from course successfully, it removes assignment and redirects to tree_display#list page

-showing list submissions

(the following are the assignment_creation_spec.rb split into files by responsability)
'''2. assignment_creation_dates_spec.rb'''

-it is able to create assignment with a new late policy

-it is able to set the deadline for an assignment review

'''3. assignment_creation_deadlines_spec.rb'''

-it is able to set the deadline for an assignment review

'''4. assignment_creation_general_tab.rb'''

-it should edit assignment available to students

-it should edit quiz number available to students

-it should edit number of members per team

-it should edit review visible to all other reviewers

-it should check if checking calibration shows the tab

'''5. assignment_creation_page_spec.rb'''

-it is able to create a public assignment

-it is able to create with teams

-it is able to create with quiz

-it is able to create with staggered deadline

-it is able to create with review visible to all reviewers

-it is able to create public micro-task assignment

-it is able to create calibrated public assignment

-it is able to show tab review strategy

-it is able to show tab due deadlines

-it should set the deadline for an assignment review

-it is able to show tab rubrics

-it is able to show attributes in rubrics

-it should sets attributes for review strategy auto selects

-it should check to find if the assignment can be added to a course

'''6. assignment_creation_participants_spec.rb'''

-it should check to see if participants can be added

-it should display newly created assignment
'''
7. assignment_creation_review_strategy_tab.rb'''

-it should auto selects correct values

-it should set number of reviews by each student

'''8. assignment_creation_rubrics_strategy_tab.rb'''

-it should update review questionnaire

-it should update scored question dropdown

-it should update author feedback questionnaire

-it should update teammate review questionnaire

'''9. assignment_creation_topics_spec.rb'''

-it can edit topics properties

-it procedes without topics properties

-it can create new topics

-it can delete existing topic

-it hides topics tab when has topics is un-checked

==Code of Additional Rspec Tests Created==
Looking at the methods that were covered in assignments_controller.rb we decided to create the following additional test cases for assignment creation.

These are additions to the existing assignments_controller_spec.rb.

[[File:E2011NewTests.png]]

==RSpec Video ScreenCast Demos==
Links to RSpec screencasts for new test cases.
# [https://drive.google.com/open?id=1AIi3MtD_TtUN5F1k3kzgqIgi8ficz_Ar Assignment Creation General Tab]
# [https://drive.google.com/open?id=1VnH_4NSLBX1x7xeYnH1Zsy6XG_nWD7Vk Assignment Creation Topic]
# [https://drive.google.com/open?id=1jUQSgXjjI5jQBTWAkD8YGhYnYIIXPtnq Assignment Creation Page]

==Coverage==
92.86% for assignments_controller.rb

Overall coverage increased (+28.04%) to 52.334%

== Project Mentors ==
#Dr. Edward Gehringer (efg@ncsu.edu)
#Srujana Rachakonda (srachak@ncsu.edu)

== Team Members ==
#Colin Foley (cmfoley2@ncsu.edu)
#Ayush Khot (akhot@ncsu.edu)
#Christian Morris (cgmorris@ncsu.edu)

== References ==

#[https://github.com/expertiza/expertiza Expertiza on GitHub]