Expertiza_Wiki - User contributions [en]

E1936 Specialized Rubrics

2019-04-26T22:16:07Z

Ngtitov: /* Sequence Diagram */

This wiki page describes the work and goals that have been completed and achieved while implementing Expertiza Project "''E1936. Specialized Rubrics for Different Topic Types (e.g., Servo project, refactoring project)''" for the CSC/ECE 517 "Object-Oriented Design and Development" class, in the Spring semester of the 2019.

__TOC__

== Introduction ==
This Wiki page is created per CSC/ECE 517: "Object-Oriented Design and Development", Expertiza Project requirements (Spring, 2019).

Our [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Team team] was assigned the project "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]". This is an Expertiza project.

In this wiki page we describe the problem to be solved, how we solve that problem, and how we test our solution. We also provide helpful links and contact information for our team.

== Project Description ==

In this section, we discuss the problem we need to solve, the Expertiza feature this problem is related to, what needs to be done to solve the problem (at a high level), and the previous attempt to solve this problem.

=== Problem Statement ===

This problem statement summarizes the problem as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment.
* Code base
** Expertiza
** Mozilla
** etc.
* Goal
** Refactoring
** Testing
** etc.
However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

=== Feature Description ===

The Expertiza project already has a feature that allows rubrics to vary by project phase or "round". The feature we will add in this project is a feature to allow rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment.

When this feature is complete, the following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

=== What Needs to be Done ===

This section summarizes the work as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

* In assignment#edit page "Rubrics" tab, add a checkbox to specify whether rubric should vary by topic.
* In assignment#edit page "Topics" tab, add dropdown(s) next to each topic to specify which rubric(s) are associated with the topic.
** The dropdown(s) should appear only if rubric should vary by topic, per the Rubrics tab.
** The topic should have a multiple dropdowns (one for each round) only if rubrics vary by round, per the Rubrics tab.
** By default, the rubric(s) for each course project will be the one(s) specified in “Rubrics” tab.
** The dropdown value can overwrite the default rubric.
* Be careful when making changes to the code
** The signup_sheet_controller should have as little to do as possible with the selection of rubrics.
** Anything not germane to selecting topics should be in another controller or (more probably) a helper.
* Create a DB migration to allow rubric to vary by topic as well as by round.

==== Screenshots ====
'''Before: (Rubrics tab for an Assignment)'''

[[File:E1936 Screenshot Assignment Edit Rubrics.png]]

'''Before: (Topics tab for an Assignment)'''

[[File:E1936 Screenshot Assignment Edit Topics.png]]

'''After: (Rubrics tab for an Assignment)'''

[[File:E1936 Specialized Rubric checkbox after.png]]

'''After: (Topics tab for an Assignment)'''

[[File:E1936 Specialized Rubrics dropdown after.png]]

== Previous Work ==

There was previous attempt to implement this project in the Fall of 2018 semester by different team, but it was rejected due to several reasons. As a result our team was assigned to complete this project with the given implementation. However, after detailed investigation of the proposed design that team has made in the previous semester and review of the code changes, we proposed different architectural solution for the problem, which we believe is simpler, more efficient, and elegant solution.

=== Previous Project Implementation ===

Our team was provided with all relevant information and materials about the design and implementation of the previously attempted project. We would like to acknowledge that we use previous implementation and design as reference and our starting point for our own implementation. The following is the list of links with the relevant materials that we have used for better project understanding, determining pros and cons of previously proposed changes, and establishing our design strategy and implementation:

# [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types Wiki]
# [https://github.com/expertiza/expertiza/pull/1327 Pull Request]
# [https://drive.google.com/file/d/1UrLb0Ula_Lb_F4MOrdC-s6xng6B08VYu/view Video Demo]

After reviewing all relevant materials above, we concluded that the project implementation was incomplete and proposed design had significant flaw.
We identified main drawbacks and flaw as follows:

* Establishing incorrect ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables:
** Creating a DB migration to add ''questionnaire_id'' in the ''sign_up_topics'' table to store the dropdown data presumes ''one-to-one'' or ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables
** Correct relationship between ''sign_up_topics'' and ''questionnaires'' tables is ''many-to-many''. For example, each project topic may have various rubrics depending on the round it is used (e.g., for round 1, topic may have one rubric; but for round 2, the same topic may have different rubric)
** Proposed previous migration changes would break DB design and would not allow to store correctly topic and its associated rubric per different round.
* Per project requirement specifications, "''The dropdown box should appear only if 'Vary rubric by topic' is selected in the Rubrics tab.''" This requirement was not met and ''Vary rubric by topic'' checkbox in the Rubrics tab was not implemented. Therefore, dropdown box with the list of rubrics appears regardless of whether "''Vary rubric by topic''" is selected or not.
* The dropdown box includes the list of all possible rubrics including default value. However, this condition can be improved. The dropdown box should list the rubrics that were created by the current user (professor who is logged in). This would allow to shorten the list of possible rubrics dramatically and help to avoid any misunderstanding and confusion while user selects desired rubric for the course project.

We found no further issues with the rest implementation and in other project materials. Wiki page is very useful for understanding design of proposed changes, finding drawbacks and the flaw. Considering all the pros and cons, and taking into account drawbacks of the proposed changes, our team evolved different approach that should successfully solve the problem stated for the project without without creating additional issues. Please see our [http://wiki.expertiza.ncsu.edu/index.php?title=E1936_Specialized_Rubrics#Project_Implementation Project Implementation] for more details.

=== Previous Project Requirements ===

The project requirements originally specified how to alter the databases to support varying rubrics by topic.

''Create a new table named topic_questionnaires with field:''
* ''id''
* ''topic_id''
* ''questionnaire_id''
* ''used_in_round''

However, the E1936 team successfully petitioned for this requirement to change to give us more freedom in the implementation. The original suggestion had a flaw. There is already an existing table in the Expertiza project called ''assignment_questionnaires'', which has columns including ''assignment_id'', ''questionnaire_id'', and ''used_in_round''. It is this table that maintains the knowledge of which rubric is used in which round of an assignment. If we were to add a new table to maintains the knowledge of which rubric is used in which topic of an assignment as originally suggested, then we would have a big problem. Knowledge of round-rubric relationship is duplicated. This would lead (best case scenario) to extra work, and (worst case scenario) to lots of bugs caused by inconsistencies in how these tables are handled with CRUD operations throughout the code. In other words, this is not a DRY solution.

== Project Implementation ==

In this section, we propose our solution to the problem with the set of UML diagrams, discuss implementation techniques and details, and list the goals we achieved.

We start by proposing our solution to the problem utilizing UML diagrams that include Use case, Entity Relationship, Class, and Sequence diagrams! Further, we discuss design strategy by breaking project problem into multiple staged sub-problems and provide solutions for each of these sub-problems. We also discuss some design strategies in the form of problem - solution, and provide the list of involved files. To illustrate implementation details for different tasks we decided to include selected (only) modified code where pseudocode-like solution is provided for each major task in this project. Finally, since we received some common questions and misunderstanding during first peer review period, we include Q/A section, where we provide an answer to each common question and concern we received.

=== UML Diagrams ===

Our solution and implementation of the project can be easily shown by discussing Use Case, Entity Relationship, and Class diagrams. First, for simplicity we discuss in general possible sequence of interactions between a user (in this case instructor) and the system (Expertiza) via presenting Use Case Diagram. Then we plan to show proposed database modification via Entity Relationship Diagram. At this point the high-level design and proposed changes should be clear. To make our design more concrete and dive into implementation level, we decided to provide UML Class diagram. Finally, to demonstrate our achievements, we complete this section with the UML Sequence diagram to illustrate interactive behavior of a system with utilized specialized rubrics feature.

==== Use Case Diagram ====

[[File: use_case_diagram_1936.png]]

The above Use Case Diagram shows general sequence of interactions between an instructor and the system (Expertiza) that defines where and how the feature can be utilized. Note that highlighted use cases are objects of our interest and work. Also please note that all of the shown use cases already exist in the system and we simply added a new feature into the system with existing use cases. Particularly, the rubric for each assignment and newly added ''Review rubric varies by topic'' checkbox with which instructor may choose to vary assignment by topic or not, by checking and unchecking the checkbox is displayed in the ''Rubrics'' tab. Hence, we show how instructor can get to the ''Rubrics'' tab of the assignments. Similarly, ''Topics'' tab of assignments has additional ''Questionnaires'' column for each topic, where instructor may specify which rubric associates with which topic. To perform this sequence of interactions and utilize the new feature, instructor must choose ''Edit'' any assignment from the list of all assignments displayed to him/her in the ''Manage Assignments'' page, which can be navigated by instructor by selecting it from the header of the system.

==== Entity Relationship Diagram ====

[[File: HW04_ERD_Closed_Arrow.png]]

The sections of the diagram highlighted in yellow show newly introduced items. That is, we are adding a new column called ''topic_id'' to ''assignment_questionnaires'', in order to link ''assignment_questionnaires'' to ''sign_up_topics''.

Any assignment, topic, or questionnaire may be represented by multiple records within ''assignment_questionnaires'', but each record within ''assignment_questionnaires'' refers to just one assignment, topic, or questionnaire. It is the ability to have an assignment represented by multiple records, and the new relationship to ''sign_up_topics'', that allows us to support all of the required scenarios:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

[[File: HW04_table.PNG]]

This table illustrates the possibilities that the system may have.

* Assignment 1 does not vary by round or by topic.
* Assignment 2 varies by round, but not by topic.
* Assignment 3 varies by topic, but not by round.
* Assignment 4 varies by both round and topic.

This change is done via database migration by running the following commands:

rails generate migration AddTopicIdToAssignmentQuestionnaires

Above command adds the ''topic_id'' column to the ''AssignmentQuestionnaires'' table and generates new migration file:

[[File: db_migration.png]]

To see the details of the entire database change please check this [https://github.com/gabalmat/expertiza/commit/d46dd57d35a21ab43efb82c28d3f14b75c695f2e commit].

==== Class Diagram ====

[[File: class_diagram_1936.png]]

The above UML Class diagram describes the changes our team made in the Ruby classes, methods and attributes. Please note that all provided classes in the diagram had already existed in the system, and we only added new relationship, modified existing methods (and some method signatures) and utilized given attributes. All of the class's attributes and methods that are shown in the diagram are utilized (directly and indirectly) in our current implementation. Some of the methods and attributes provided within the classes in the diagram are newly developed. There is only one newly added aggregation relationship between the ''SingUpTopic'' class and ''AssignmentQuestionnaire'' class. Moreover, controllers and models for the certain classes were combined into a single class for simplicity to the design description. In reality, half of the methods shown in the ''Assignment'' class are implemented in the ''app/controllers/assignments_controller.rb'' file and other half is in the ''app/models/assignment.rb'' file. We decided to combine all of the functionalities of each ''relevant'' controller and model class into single class for the ease of design presentation and interpretation. We used Sequence diagram below to illustrate how MVC architectural pattern is used for a given classes.

==== Sequence Diagram ====

Below we provide UML Sequence diagram to illustrate the interactive behavior of a system utilizing the specialized rubric feature that we implemented. ''Note'' that the following UML Sequence Diagram still is considered to be a high-level diagram and it displays the behavior of the system that we observed while debugging and implementing our code. By presenting the following sequence diagram, we do not make a claim that this is how Expertiza works, we simply documented our observations and want to share it with the teaching staff and fellow students. It is possible that we could have missed something or incorrectly interpreted the source code, but this diagram is the best of our knowledge. We hope that this would be useful for the teams working on similar projects in the future!

[[File: sequence_diagram_1936.png]]

=== Design Strategy ===

As we have concrete understanding of the problem and proposed solution, we would like break this problem into multiple more manageable sub-problems with specific tasks and well-defined solutions to these tasks. Generally, each sub-problem requires modification of one or more files. We would like to propose our Design Strategy as follows:

==== Preparation ====

* Clean up code that our implementation will use
** PROBLEM: We need to add/change content to the edit assignment Topics tab, but unlike other tabs in the views, it does not have its own partial view to work with
*** SOLUTION: Refactor and move the code dealing with topics into separate ''app/views/assignments/edit/_topics.html.erb'' file
*** File: ''app/views/assignments/edit.html.erb''
** PROBLEM: In a section below, we discuss the need to modify existing methods to also accept an optional argument ''topic_id''. In many cases these existing method use confusing variable names or lack sufficient comments.
*** SOLUTION: Clean up methods that we plan to modify
*** FILE: (many)

==== Implementation ====

* Allow an instructor to indicate that rubric should vary by topic
** PROBLEM: There is no way for an instructor to indicate that an assignments' rubric should vary by topic
*** SOLUTION: Add a new checkbox in the edit assignment Rubrics tab to indicate this preference
*** FILE: ''app/views/assignments/edit/_rubrics.html.erb''
** PROBLEM: If the user changes the "Review rubric varies by round?" or "Review rubric varies by topic?" checkbox on the Rubrics tab, but leaves the page in an inappropriate state before leaving the tab (e.g. rubric weights that do not add up to 0% or 100%), then the user does not get feedback that they made a mistake. They only get such feedback when explicitly performing a "Save". Switching from tab to tab on the assignments edit page attempts to perform an implicit save, but there is no mechanism to update the flash messages onscreen if anything goes wrong. This becomes more problematic now that the Topics tab appearance depends on choices made on the Rubrics tab.
*** SOLUTION: Provide a route to an assignments controller method that can return a render of the flash messages. Add an AJAX call on the assignments edit view that can update the flash messages onscreen.
*** FILE: app/controllers/assignments_controller.rb
*** FILE: app/views/assignments/edit.html.erb

* Allow an instructor to choose different rubrics for different topics
** PROBLEM: There is no way for an instructor to choose different rubrics for different topics
*** SOLUTION: Add a new column in the edit assignment Topics tab for choosing rubric(s)
**** New column only visible if rubrics vary by topic, per Rubrics tab
**** New column has multiple rounds per topic only if rubrics vary by round, per Rubrics tab
**** Default value for any rubric is that rubric which is shown on Rubrics tab
**** Rubric(s) in the dropdown list is/are only those that were created by the currently logged in instructor
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The code in ''app/views/assignments/edit/_rubrics.html.erb'' includes dropdowns like the ones we need to add for topics, but that code is way too complex for its functionality (too heavy for a view, uses JavaScript)
*** SOLUTION: Use the code in ''app/views/assignments/edit/_rubrics.html.erb'' for inspiration only
**** Refactor and move as much actual work as possible out into helper methods that are simply called from this view
**** Views are not intended to do heavy lifting
**** Use as little JavaScript as possible, since this is a Ruby On Rails project, not a JS project
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The drop-downs for selecting rubrics should show only those rubrics created by the currently-logged in instructor (per project mentor). They currently include those created by the instructor of the relevant assignment.
*** SOLUTION: Change the filtering in the ''questionnaire_options'' method to reflect the desired filtering.
*** FILE: app/helpers/assignment_helper.rb

* Support the ability of the database to link rubrics to topics
** PROBLEM: There is no relationship in the database that can link together rubrics with topics
*** SOLUTION: Add a new migration that adds topic_id field to existing assignment_questionnaires table
**** This new field will reference the id field in the sign_up_topics table
*** FILE: ''db/migrate/[timestamp]_add_topic_id_to_assignment_questionnaires_table.rb''
** PROBLEM: The AssignmentQuestionnaire model does not have any knowledge of topics
*** SOLUTION: Add a new belongs_to reference in the model, to sign_up_topic
*** FILE: ''app/models/assignment_questionnaire.rb''
** PROBLEM: The SignUpTopic model does not have any knowledge of assignment_questionnaire records
*** SOLUTION: Add a new has_many reference in the model, to assignment_questionnaires
*** FILE: ''app/models/sign_up_topic.rb''

* React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox
** PROBLEM: There is nothing in place to remember when an instructor chooses to vary rubrics by topic, or chooses to not vary rubrics by topic
*** SOLUTION: Modify existing method ''update_assignment_questionnaires()''. This method executes upon assignment save (explicit or via tab-change). This method deletes all ''AssignmentQuestionnaire'' records before creating new correct records reflecting the user's selections in the edit assignment form. Modify such that this method can create multiple records reflecting multiple topics, in the case where rubric should vary by topic. With this modification, the correct records (with default rubrics) are present when the user goes to the Topics tab, so that the Topics tab can display properly.
*** FILE: app\models\assignment_form.rb
** PROBLEM: There is no way to refresh the Topics tab to show / hide drop-downs after the instructor changes checkbox on Rubrics tab
*** SOLUTION: Make use of existing "Save"-like functionality when changing tabs on the edit assignment page. When updating due to tab-change, cause a fresh render of the Topics tab. When updating due to "Save" button, keep functionality as-is.
*** FILE: ''app/controllers/assignments_controller.rb''
*** FILE: ''app/views/assignments/edit.html.erb''

* Support the ability to find the correct rubric for a particular topic
** PROBLEM: The existing methods allow us to find the correct rubric for a particular round, but not a particular topic
*** SOLUTION: Modify the existing method questionnaire(assignment, type, round_number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method assignment_questionnaire(assignment, type, number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method questionnaire(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/review_response_map.rb''
*** SOLUTION: Modify the existing method questionnaire(round) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/self_review_response_map.rb''
*** SOLUTION: Modify the existing method review_questionnaire_id(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/assignment.rb''

* Use the correct rubric for a particular topic
** PROBLEM: When finding rubrics to display, save results from, calculate scores from, etc. round may be used but topic is not
*** SOLUTION: Examine all callers of methods modified in the previous section and pass in ''topic_id'' where appropriate to grab the correct rubric
*** FILE: (many)

* Support the ability to determine if an assignment has rubrics that vary by topic
** PROBLEM: There is no handy method to call that can tell us if an assignment has rubrics that vary by topic
*** SOLUTION: Add a new method ''varying_rubrics_by_topic?'' based on the existing method ''varying_rubrics_by_round?''
*** FILE: ''app/models/assignment.rb''

=== Q&A ===

Here we paraphrase some questions we received during round 1 of the documentation review.

* Q: How will the new logic for varying rubric by topic tie in with the existing logic?
** A: We will add a new field ''topic_id'' into the existing table ''assignment_questionnaires'', as noted in the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram]. We will update methods that find questionnaires to allow these methods to also accept a ''topic_id'' argument, as noted in [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the "Rubric varies by topic" selection be stored?
** A: Indirectly, by saving records in the ''assignment_questionnaires'' table which either reference a ''topic_id'' or leave this field nil. For more details please see the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram] discussion.
* Q: What will happen if an instructor selects "Rubric varies by topic", makes selections on the "Topics" tab, and then deselects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will no longer show dropdowns for selecting rubrics by topic. When the assignment is saved, the appropriate not-varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: What will happen after an instructor selects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will now show dropdowns for selecting rubrics by topic. These rubrics will default to those shown on the Rubrics tab. When the assignment is saved, the appropriate varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the appropriate list of questionnaires for the new dropdowns on the Topics tab be determined?
** A: In the same way that the appropriate list of questionnaires for the existing dropdowns on the Rubrics tab is determined. Namely, by the ''questionnaire_options'' method in the file ''expertiza/app/helpers/assignment_helper.rb''.

=== (Selected) Modified Code ===

In this section, we provide ''selected portions'' of modified code to illustrate how we implemented the new feature.

Allow an instructor to indicate that rubric should vary by topic

''app\views\assignments\edit\_rubrics.html.erb''

<pre>
<input name="vary_by_topic" type="hidden" value="false" />
<%= check_box_tag('vary_by_topic', 'true', @review_vary_by_topic_check) %>
<%= label_tag('vary_by_topic', 'Review rubric varies by topic?') %>
</pre>

Allow an instructor to choose different rubrics for different topics

''app\views\sign_up_sheet\_table_line.html.erb''

<pre>
<% if @assignment_form.assignment.varying_rubrics_by_topic? %>
...
<td>
<table id = <%=table_id %>>
<tr>
<script>
...
<% if @assignment_form.assignment.varying_rubrics_by_round? %>
<% for round in 1..@assignment_form.assignment.rounds_of_reviews %>
...
<% end %>
<% else %>
...
<% end %>
</script>
</tr>
</table>
</td>
<% end %>
</pre>

Support the ability of the database to link rubrics to topics

''app\models\sign_up_topic.rb''

<pre>
class SignUpTopic < ActiveRecord::Base
...
has_many :assignment_questionnaires, class_name: 'AssignmentQuestionnaire', foreign_key: 'topic_id', dependent: :destroy
...
end
</pre>

React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox

''app/controllers/assignments_controller.rb''

<pre>
def update
...
# What to do next depends on how we got here
if params['button'].nil?
# REFRESH the topics tab after changing tabs
...
render :partial => "assignments/edit/topics"
else
# SAVE button was used (do a redirect)
redirect_to edit_assignment_path @assignment_form.assignment.id
end
end
</pre>

Support the ability to find the correct rubric for a particular topic

''app/helpers/assignment_helper.rb''

<pre>
def assignment_questionnaire(assignment, questionnaire_type, round_number = nil, topic_id = nil)
q_by_type = assignment.questionnaires.find_by(type: questionnaire_type)
if q_by_type.nil?
...
else
...
aq_by_type = assignment.assignment_questionnaires.find_by(questionnaire_id: q_by_type.id)
if round_number.nil? && topic_id.nil?
# Find by type
aq = aq_by_type
elsif topic_id.nil?
# Find by round
aq = assignment.assignment_questionnaires.find_by(used_in_round: round_number)
elsif round_number.nil?
# Find by topic
aq = assignment.assignment_questionnaires.find_by(topic_id: topic_id)
else
# Find by round and topic
aq = assignment.assignment_questionnaires.where(used_in_round: round_number, topic_id: topic_id).first
end
aq.nil? ? aq_by_type : aq
end
end
</pre>

Use the correct rubric for a particular topic

''app\controllers\response_controller.rb''

<pre>
def set_questionnaire_for_new_response
case @map.type
when "ReviewResponseMap", "SelfReviewResponseMap"
reviewees_topic = SignedUpTeam.topic_id_by_team_id(@contributor.id)
@current_round = @assignment.number_of_current_round(reviewees_topic)
@questionnaire = @map.questionnaire(@current_round, reviewees_topic)
when
...
end
end
</pre>

Support the ability to determine if an assignment has rubrics that vary by topic

''app\models\assignment.rb''

<pre>
def varying_rubrics_by_topic?
aq_no_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id, topic_id: nil).size >= 1
aq_with_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id).where.not(topic_id: nil).size >= 1
if !aq_no_topic_id && !aq_with_topic_id
return false
elsif aq_no_topic_id && !aq_with_topic_id
return false
elsif !aq_no_topic_id && aq_with_topic_id
return true
else
raise StandardError.new("Assignment with id " + self.id.to_s + " has a conflict about whether or not it varies by topic")
end
end
</pre>

== How To ==

In this section, we describe how to use the newly implemented feature.

(Log in as an Instructor)

=== Specify Rubrics should vary by Topic ===
# Navigate to Edit page for an Assignment.
# Click the Rubrics tab.
# Check the "Review rubric varies by topic?" checkbox.

=== Select a Rubric for a Topic ===
After checking the "Review rubric varies by topic?" checkbox,
# Click the Topics tab.
# Select the appropriate Rubric from the dropdown menu(s) beside each Topic.

See also: [https://www.youtube.com/watch?v=F7nQsIUspQM Video Demonstration of Specialized Rubrics]

== Project Testing ==

=== Strategy ===

To test code implementation, correctness of the added feature, verify that the team did not break any existing functionalities in the Expertiza, and ensure that no bugs were introduced in the code, we developed the following Test Strategy:

# Code inspection
# Run and pass existing RSpec Tests
# Develop New RSpec Tests
# Run through live UI to test a feature using Expertiza instance

=== Rspec Testing ===

==== Existing Tests ====

Run all existing RSpec tests on any changed files, after our changes, to ensure that we have not introduced any failures.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/ spec/controllers/ spec/models/
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 6978
[...]

Finished in 8 minutes 10 seconds (files took 12.32 seconds to load)
900 examples, 0 failures, 16 pending
</pre>

The features tests were extremely time-consuming so we did not routinely run them during development. However TravisCI results on our [https://github.com/expertiza/expertiza/pull/1444 Pull Request] demonstrate the passing status of all tests.

==== New Tests ====

Write comprehensive RSpec tests, for all new or modified methods.
* FILE: ''spec/helpers/assignment_helper_spec.rb''
** METHODS TO TEST:
*** questionnaire(assignment, type, round_number, topic_id)
*** assignment_questionnaire(assignment, type, number, topic_id)
* FILE: ''spec/models/review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/self_review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/assignment_spec.rb''
*** METHODS TO TEST:
**** review_questionnaire_id(round, topic_id)
**** varying_rubrics_by_topic?()

Run these tests, to ensure that the new code works as intended.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/assignment_helper_spec.rb spec/models/review_response_map_spec.rb spec/models/self_review_response_map_spec.rb spec/models/assignment_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 47508
....................................................................................................................

Finished in 1 minute 7.94 seconds (files took 10.85 seconds to load)
116 examples, 0 failures
</pre>

The test suite for a single new method is shown below.
* There are many such suites added in expertiza/spec
* This example illustrates our general strategy:
** test missing input
** test bad input
** test various acceptable forms of input
** test scenarios that lead to various return values

<pre>
describe "#questionnaire" do

...

it "throws exception if assignment argument nil" do
...
end

it "throws exception if all arguments nil except for assignment" do
...
end

it "finds by type if round number & topic id not given" do
...
end

it "throws exception if round number & topic id not given, no luck finding by type" do
...
end

it "finds by round number alone if round number alone is given" do
...
end

it "creates new questionnaire of given type if round number alone is given, no luck finding by round" do
...
end

it "finds by topic id alone if topic id alone is given" do
...
end

it "creates new questionnaire of given type if topic id alone is given, no luck finding by topic" do
...
end

it "finds by round number and topic id if both are given" do
...
end

it "creates new questionnaire of given type if round and topic are given, no luck finding by round and topic" do
...
end

end
</pre>

=== UI Testing ===

Here we describe manual UI Testing that was performed.

==== Rubric does not vary by round or by topic ====

* Go to Rubrics tab and verify that both "Review rubric varies by round?" and "Review rubric varies by topic?" checkboxes are unchecked.
* Go to Topics tab and verify that there are no dropdown menus beside each Topic.

==== Rubric varies by round, but not by topic ====

* Go to Rubrics tab and verify that "Review rubric varies by round?" is checked and "Review rubric varies by topic?" is unchecked.
* Go to Topics tab and verify that there are no dropdown menus beside each Topic.

==== Rubric varies by topic, but not by round ====

* Go to Rubrics tab and verify that "Review rubric varies by round?" is unchecked and "Review rubric varies by topic?" is checked.
* Go to Topics tab and verify that there is only 1 dropdown menu beside each Topic.

==== Rubric varies by both round and topic ====

* Go to Rubrics tab and verify that both "Review rubric varies by round?" and "Review rubric varies by topic?" checkboxes are checked.
* Go to Topics tab and verify that there is 1 dropdown menu per round for each Topic.

==== Updates rejected if Rubric weights do not add up to 100% or 0% ====

* Go to Rubrics tab and check the "Review rubric varies by topic?" checkbox.
* Change weight values for the Rubrics so that they add up to 97%.
* Go to Topics tab and verify that checking the "Review rubric varies by topic?" checkbox did not work.
* Verify that there are no dropdown menus beside each Topic.
* Verify that the proper error message is shown.

== Conclusion ==

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment. However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

In this project, we have solved this issue, by allowing rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment. The following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

The instructor may use the Rubrics tab of the assignments edit page to make changes to the "Review rubric varies by round?" and "Review rubric varies by topic?" checkboxes. The instructor may then use the Topics tab to select the appropriate rubric(s) for every topic in the assignment.

Various other Expertiza code has been updated to retrieve the correct rubric depending on round and topic after the instructor has made these selections.

== Future Work ==

This section discusses possible future work for another CSC 517 team to take on. Such a team can search the Expertiza codebase for "TODO E1936 (future work)" to find most of the places in the code that the notes below refer to.

* Reduce the delay between changing the state of the "Review rubric varies by topic?" checkbox on the Rubrics tab, and the show / hide of rubric drop-downs on the Topics tab.
* Redesign UI for tone analysis, heatmaps, and review scores pop-up. These areas are designed around the assumption that reviews do NOT vary by topic. For example, in the review scores pop-up, a reviewer can see the reviews they have given, arranged by question, across multiple reviewees.
* Consider whether the questionnaire options available for selection in drop-downs on the Topics and Rubrics tabs should include ''only'' those questionnaires created by the currently-logged-in instructor. Our work in this project changed the filtering from instructor-owning-assignment to currently-logged-in instructor - but left in place all non-private questionnaires. ''Consider'' whether non-private questionnaires should be excluded. This may result in some drop-downs not having any available questionnaires.

== Useful Links ==
In this section we provide useful links related to the work that has been completed by our team.
# [https://github.com/gabalmat/expertiza Forked Project Repository]
# [https://github.com/expertiza/expertiza/pull/1444 Pull Request]
# [https://www.youtube.com/watch?v=F7nQsIUspQM Video Demonstration of Specialized Rubrics]

== Team ==
=== Contacts ===
We have 4 (four) members in our team and 1 (one) mentor was also assigned to us. For any project-related questions/concerns please contact us:
* Aurora Tiffany-Davis - attiffan@ncsu.edu
* Ginger Balmat - gabalmat@ncsu.edu
* Joe Hutchinson - jehutch3@ncsu.edu
* Nikolay Titov - ngtitov@ncsu.edu
* Zhewei Hu (mentor) - zhu6@ncsu.edu

=== Contribution Based Inquiry ===
For any questions or concerns related to the project design, implementation, testing of the feature developed by our team, please contact an engineer who worked closely on the topic of your interest based on the [https://docs.google.com/spreadsheets/d/11uxuI_5gaC-FQDwIOxtCaN3HZ9bfWDJgIjFemtYV5jE/edit?ts=5ca222f3#gid=0 Contribution Assignment Grid] we created to keep track of our progress and contribution. Please carbon copy all team members and our mentor in the email.

== References ==
* Expertiza Web [https://github.com/expertiza/expertiza#expertiza]
* Expertiza on GitHub [https://github.com/expertiza/expertiza]
* Expertiza Wiki [http://wikis.lib.ncsu.edu/index.php/Expertiza]
* Rspec Documentation [https://relishapp.com/rspec]
* Wiki with Previous Work [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types]
* Pull Request of the Previous Work [https://github.com/expertiza/expertiza/pull/1327]

2019-04-26T14:59:22Z

Ngtitov: uploaded a new version of "File:Class diagram 1936.png"

File:Use case diagram 1936.png

2019-04-26T14:49:19Z

Ngtitov:

E1936 Specialized Rubrics

2019-04-26T14:47:17Z

Ngtitov: /* User Case Diagram */

This wiki page describes the work and goals that have been completed and achieved while implementing Expertiza Project "''E1936. Specialized Rubrics for Different Topic Types (e.g., Servo project, refactoring project)''" for the CSC/ECE 517 "Object-Oriented Design and Development" class, in the Spring semester of the 2019.

__TOC__

== Introduction ==
This Wiki page is created per CSC/ECE 517: "Object-Oriented Design and Development", Expertiza Project requirements (Spring, 2019).

Our [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Team team] was assigned the project "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]". This is an Expertiza project.

In this wiki page we describe the problem to be solved, how we solve that problem, and how we test our solution. We also provide helpful links and contact information for our team.

== Project Description ==

In this section, we discuss the problem we need to solve, the Expertiza feature this problem is related to, what needs to be done to solve the problem (at a high level), and the previous attempt to solve this problem.

=== Problem Statement ===

This problem statement summarizes the problem as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment.
* Code base
** Expertiza
** Mozilla
** etc.
* Goal
** Refactoring
** Testing
** etc.
However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

=== Feature Description ===

The Expertiza project already has a feature that allows rubrics to vary by project phase or "round". The feature we will add in this project is a feature to allow rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment.

When this feature is complete, the following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

=== What Needs to be Done ===

This section summarizes the work as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

* In assignment#edit page "Rubrics" tab, add a checkbox to specify whether rubric should vary by topic.
* In assignment#edit page "Topics" tab, add dropdown(s) next to each topic to specify which rubric(s) are associated with the topic.
** The dropdown(s) should appear only if rubric should vary by topic, per the Rubrics tab.
** The topic should have a multiple dropdowns (one for each round) only if rubrics vary by round, per the Rubrics tab.
** By default, the rubric(s) for each course project will be the one(s) specified in “Rubrics” tab.
** The dropdown value can overwrite the default rubric.
* Be careful when making changes to the code
** The signup_sheet_controller should have as little to do as possible with the selection of rubrics.
** Anything not germane to selecting topics should be in another controller or (more probably) a helper.
* Create a DB migration to allow rubric to vary by topic as well as by round.

==== Screenshots ====
'''Current screenshot of Rubrics tab for an Assignment'''
(need to add a checkbox to indicate that rubric should vary by topic)

[[File:E1936 Screenshot Assignment Edit Rubrics.png]]

'''Current screenshot of Topics tab for an Assignment'''
(need to add a way to select rubrics for particular topics)

[[File:E1936 Screenshot Assignment Edit Topics.png]]

'''Proposed screenshot of Rubrics tab for an Assignment'''

[[File:E1936 Specialized Rubric rubric checkbox.png]]

'''Proposed screenshot of Topics tab for an Assignment'''

(drop down shown for only one topic here, but final product will have one for each topic)

[[File:E1936 Specialized Rubric dropdown menu.png]]

== Previous Work ==

There was previous attempt to implement this project in the Fall of 2018 semester by different team, but it was rejected due to several reasons. Our team was provided with all relevant information and materials about the design and implementation of the previously attempted project. We would like to acknowledge that we use previous implementation and design as reference and our starting point for our own implementation. The following is the list of links with the relevant materials that we have used for better project understanding, determining pros and cons of previously proposed changes, and establishing our design strategy and implementation:

# [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types Wiki]
# [https://github.com/expertiza/expertiza/pull/1327 Pull Request]
# [https://drive.google.com/file/d/1UrLb0Ula_Lb_F4MOrdC-s6xng6B08VYu/view Video Demo]

After reviewing all relevant materials above, we concluded that the project implementation was incomplete and proposed design had significant flaw.
We identified main drawbacks and flaw as follows:

* Establishing incorrect ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables:
** Creating a DB migration to add ''questionnaire_id'' in the ''sign_up_topics'' table to store the dropdown data presumes ''one-to-one'' or ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables
** Correct relationship between ''sign_up_topics'' and ''questionnaires'' tables is ''many-to-many''. For example, each project topic may have various rubrics depending on the round it is used (e.g., for round 1, topic may have one rubric; but for round 2, the same topic may have different rubric)
** Proposed previous migration changes would break DB design and would not allow to store correctly topic and its associated rubric per different round.
* Per project requirement specifications, "''The dropdown box should appear only if 'Vary rubric by topic' is selected in the Rubrics tab.''" This requirement was not met and ''Vary rubric by topic'' checkbox in the Rubrics tab was not implemented. Therefore, dropdown box with the list of rubrics appears regardless of whether "''Vary rubric by topic''" is selected or not.
* The dropdown box includes the list of all possible rubrics including default value. However, this condition can be improved. The dropdown box should list the rubrics that were created by the current user (professor who is logged in). This would allow to shorten the list of possible rubrics dramatically and help to avoid any misunderstanding and confusion while user selects desired rubric for the course project.

We found no further issues with the rest implementation and in other project materials. Wiki page is very useful for understanding design of proposed changes, finding drawbacks and the flaw. Considering all the pros and cons, and taking into account drawbacks of the proposed changes, our team evolved different approach that should successfully solve the problem stated for the project without without creating additional issues. Please see our [http://wiki.expertiza.ncsu.edu/index.php?title=E1936_Specialized_Rubrics#Project_Implementation Project Implementation] for more details.

== Previous Project Requirements ==

The project requirements originally specified how to alter the databases to support varying rubrics by topic.

''Create a new table named topic_questionnaires with field:''
* ''id''
* ''topic_id''
* ''questionnaire_id''
* ''used_in_round''

However, the E1936 team successfully petitioned for this requirement to change to give us more freedom in the implementation. The original suggestion had a flaw. There is already an existing table in the Expertiza project called ''assignment_questionnaires'', which has columns including ''assignment_id'', ''questionnaire_id'', and ''used_in_round''. It is this table that maintains the knowledge of which rubric is used in which round of an assignment. If we were to add a new table to maintains the knowledge of which rubric is used in which topic of an assignment as originally suggested, then we would have a big problem. Knowledge of round-rubric relationship is duplicated. This would lead (best case scenario) to extra work, and (worst case scenario) to lots of bugs caused by inconsistencies in how these tables are handled with CRUD operations throughout the code. In other words, this is not a DRY solution.

== Project Implementation ==

In this section, we propose our solution to the problem, discuss implementation techniques and details, and list the goals we achieved.

We start by analyzing Entity Relationship Diagram where we state the solution to the problem. Further we discuss design strategy by breaking project problem into multiple staged sub-problems and provide solutions for each of these sub-problems. We also discuss some implementation details and provide the list of involved files. Finally, we demonstrate how the implemented feature can be used by providing a ''How To'' sub-section.

=== Diagrams ===

Our solution and implementation of the project can be easily shown by discussing Use Case, Entity Relationship, and Class diagrams. First, for simplicity we discuss in general possible sequence of interactions between a user (in this case instructor) and the system (Expertiza) via presenting Use Case Diagram. Then we plan to show proposed database modification via Entity Relationship Diagram.

==== User Case Diagram ====

[[File: use_case_diagram_1936.png]]

The above Use Case Diagram shows general sequence of interactions between an instructor and the system (Expertiza) that defines where and how the feature can be utilized. Note that highlighted use cases are objects of our interest and work. Also please note that all of the shown use cases already exist in the system and we simply adding new feature into the system with existing use cases. Particularly, the rubric for each assignment and newly added ''Review rubric varies by topic'' checkbox with which instructor may choose to vary assignment by topic or not by checking and unchecking the checkbox is displayed in the ''Rubrics'' tab. Hence, we show how instructor can get to the ''Rubrics'' tab of the assignments. Similarly, ''Topics'' tab of assignments has additional ''Questionnaires'' column for each topic, where instructor may specify which rubric associates with the topic. To perform such sequence of iterations and utilize the new feature, instructor must choose ''Edit'' any assignment from the list of all assignments displayed to him/her in the ''Manage Assignments'' boundary, which can be navigated by instructor by selecting it from the header of the page.

==== Entity Relationship Diagram ====

[[File: HW04_ERD.png]]

The sections of the diagram highlighted in yellow show newly introduced items. That is, we are adding a new column called ''topic_id'' to ''assignment_questionnaires'', in order to link ''assignment_questionnaires'' to ''sign_up_topics''.

Any assignment, topic, or questionnaire may be represented by multiple records within ''assignment_questionnaires'', but each record within ''assignment_questionnaires'' refers to just one assignment, topic, or questionnaire. It is the ability to have an assignment represented by multiple records, and the new relationship to ''sign_up_topics'', that allows us to support all of the required scenarios:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

[[File: HW04_table.PNG]]

This table illustrates the possibilities that the system may have.

* Assignment 1 does not vary by round or by topic.
* Assignment 2 varies by round, but not by topic.
* Assignment 3 varies by topic, but not by round.
* Assignment 4 varies by both round and topic.

This change is done via database migration by running the following commands:

rails generate migration AddTopicIdToAssignmentQuestionnaires

Above command adds the ''topic_id'' column to the ''AssignmentQuestionnaires'' table and generates new migration file:

[[File: db_migration.png]]

To see the details of the entire database change please check this [https://github.com/gabalmat/expertiza/commit/d46dd57d35a21ab43efb82c28d3f14b75c695f2e commit].

==== Class Diagram ====

[[File: class_diagram_1936.png]]

This is Class Class diagram.

=== Design Strategy ===

As we have concrete understanding of the problem and proposed solution, we would like break this problem into multiple more manageable sub-problems with specific tasks and well-defined solutions to these tasks. Generally, each sub-problem requires modification of one or more files. We would like to propose our Design Strategy as follows:

==== Preparation ====

* Clean up code that our implementation will use
** PROBLEM: We need to add/change content to the edit assignment Topics tab, but unlike other tabs in the views, it does not have its own partial view to work with
*** SOLUTION: Refactor and move the code dealing with topics into separate ''app/views/assignments/edit/_topics.html.erb'' file
*** File: ''app/views/assignments/edit.html.erb''
** PROBLEM: In a section below, we discuss the need to modify existing methods to also accept an optional argument ''topic_id''. In many cases these existing method use confusing variable names or lack sufficient comments.
*** SOLUTION: Clean up methods that we plan to modify
*** FILE: (many)

==== Implementation ====

* Allow an instructor to indicate that rubric should vary by topic
** PROBLEM: There is no way for an instructor to indicate that an assignments' rubric should vary by topic
*** SOLUTION: Add a new checkbox in the edit assignment Rubrics tab to indicate this preference
*** FILE: ''app/views/assignments/edit/_rubrics.html.erb''
** PROBLEM: If the user changes the "Review rubric varies by round?" or "Review rubric varies by topic?" checkbox on the Rubrics tab, but leaves the page in an inappropriate state before leaving the tab (e.g. rubric weights that do not add up to 0% or 100%), then the user does not get feedback that they made a mistake. They only get such feedback when explicitly performing a "Save". Switching from tab to tab on the assignments edit page attempts to perform an implicit save, but there is no mechanism to update the flash messages onscreen if anything goes wrong. This becomes more problematic now that the Topics tab appearance depends on choices made on the Rubrics tab.
*** SOLUTION: Provide a route to an assignments controller method that can return a render of the flash messages. Add an AJAX call on the assignments edit view that can update the flash messages onscreen.
*** FILE: app/controllers/assignments_controller.rb
*** FILE: app/views/assignments/edit.html.erb

* Allow an instructor to choose different rubrics for different topics
** PROBLEM: There is no way for an instructor to choose different rubrics for different topics
*** SOLUTION: Add a new column in the edit assignment Topics tab for choosing rubric(s)
**** New column only visible if rubrics vary by topic, per Rubrics tab
**** New column has multiple rounds per topic only if rubrics vary by round, per Rubrics tab
**** Default value for any rubric is that rubric which is shown on Rubrics tab
**** Rubric(s) in the dropdown list is/are only those that were created by the currently logged in instructor
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The code in ''app/views/assignments/edit/_rubrics.html.erb'' includes dropdowns like the ones we need to add for topics, but that code is way too complex for its functionality (too heavy for a view, uses JavaScript)
*** SOLUTION: Use the code in ''app/views/assignments/edit/_rubrics.html.erb'' for inspiration only
**** Refactor and move as much actual work as possible out into helper methods that are simply called from this view
**** Views are not intended to do heavy lifting
**** Use as little JavaScript as possible, since this is a Ruby On Rails project, not a JS project
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The drop-downs for selecting rubrics should show only those rubrics created by the currently-logged in instructor (per project mentor). They currently include those created by the instructor of the relevant assignment.
*** SOLUTION: Change the filtering in the ''questionnaire_options'' method to reflect the desired filtering.
*** FILE: app/helpers/assignment_helper.rb

* Support the ability of the database to link rubrics to topics
** PROBLEM: There is no relationship in the database that can link together rubrics with topics
*** SOLUTION: Add a new migration that adds topic_id field to existing assignment_questionnaires table
**** This new field will reference the id field in the sign_up_topics table
*** FILE: ''db/migrate/[timestamp]_add_topic_id_to_assignment_questionnaires_table.rb''
** PROBLEM: The AssignmentQuestionnaire model does not have any knowledge of topics
*** SOLUTION: Add a new belongs_to reference in the model, to sign_up_topic
*** FILE: ''app/models/assignment_questionnaire.rb''
** PROBLEM: The SignUpTopic model does not have any knowledge of assignment_questionnaire records
*** SOLUTION: Add a new has_many reference in the model, to assignment_questionnaires
*** FILE: ''app/models/sign_up_topic.rb''

* React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox
** PROBLEM: There is nothing in place to remember when an instructor chooses to vary rubrics by topic, or chooses to not vary rubrics by topic
*** SOLUTION: Modify existing method ''update_assignment_questionnaires()''. This method executes upon assignment save (explicit or via tab-change). This method deletes all ''AssignmentQuestionnaire'' records before creating new correct records reflecting the user's selections in the edit assignment form. Modify such that this method can create multiple records reflecting multiple topics, in the case where rubric should vary by topic. With this modification, the correct records (with default rubrics) are present when the user goes to the Topics tab, so that the Topics tab can display properly.
*** FILE: app\models\assignment_form.rb
** PROBLEM: There is no way to refresh the Topics tab to show / hide drop-downs after the instructor changes checkbox on Rubrics tab
*** SOLUTION: Make use of existing "Save"-like functionality when changing tabs on the edit assignment page. When updating due to tab-change, cause a fresh render of the Topics tab. When updating due to "Save" button, keep functionality as-is.
*** FILE: ''app/controllers/assignments_controller.rb''
*** FILE: ''app/views/assignments/edit.html.erb''

* Support the ability to find the correct rubric for a particular topic
** PROBLEM: The existing methods allow us to find the correct rubric for a particular round, but not a particular topic
*** SOLUTION: Modify the existing method questionnaire(assignment, type, round_number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method assignment_questionnaire(assignment, type, number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method questionnaire(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/review_response_map.rb''
*** SOLUTION: Modify the existing method questionnaire(round) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/self_review_response_map.rb''
*** SOLUTION: Modify the existing method review_questionnaire_id(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/assignment.rb''

* Use the correct rubric for a particular topic
** PROBLEM: When finding rubrics to display, save results from, calculate scores from, etc. round may be used but topic is not
*** SOLUTION: Examine all callers of methods modified in the previous section and pass in ''topic_id'' where appropriate to grab the correct rubric
*** FILE: (many)

* Support the ability to determine if an assignment has rubrics that vary by topic
** PROBLEM: There is no handy method to call that can tell us if an assignment has rubrics that vary by topic
*** SOLUTION: Add a new method ''varying_rubrics_by_topic?'' based on the existing method ''varying_rubrics_by_round?''
*** FILE: ''app/models/assignment.rb''

=== Q&A ===

Here we paraphrase some questions we received during round 1 of the documentation review.

* Q: How will the new logic for varying rubric by topic tie in with the existing logic?
** A: We will add a new field ''topic_id'' into the existing table ''assignment_questionnaires'', as noted in the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram]. We will update methods that find questionnaires to allow these methods to also accept a ''topic_id'' argument, as noted in [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the "Rubric varies by topic" selection be stored?
** A: Indirectly, by saving records in the ''assignment_questionnaires'' table which either reference a ''topic_id'' or leave this field nil. For more details please see the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram] discussion.
* Q: What will happen if an instructor selects "Rubric varies by topic", makes selections on the "Topics" tab, and then deselects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will no longer show dropdowns for selecting rubrics by topic. When the assignment is saved, the appropriate not-varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: What will happen after an instructor selects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will now show dropdowns for selecting rubrics by topic. These rubrics will default to those shown on the Rubrics tab. When the assignment is saved, the appropriate varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the appropriate list of questionnaires for the new dropdowns on the Topics tab be determined?
** A: In the same way that the appropriate list of questionnaires for the existing dropdowns on the Rubrics tab is determined. Namely, by the ''questionnaire_options'' method in the file ''expertiza/app/helpers/assignment_helper.rb''.

=== (Selected) Modified Code ===

In this section, we provide ''selected portions'' of modified code to illustrate how we implemented the new feature.

Allow an instructor to indicate that rubric should vary by topic

''app\views\assignments\edit\_rubrics.html.erb''

<pre>
<input name="vary_by_topic" type="hidden" value="false" />
<%= check_box_tag('vary_by_topic', 'true', @review_vary_by_topic_check) %>
<%= label_tag('vary_by_topic', 'Review rubric varies by topic?') %>
</pre>

Allow an instructor to choose different rubrics for different topics

''app\views\sign_up_sheet\_table_line.html.erb''

<pre>
<% if @assignment_form.assignment.varying_rubrics_by_topic? %>
...
<td>
<table id = <%=table_id %>>
<tr>
<script>
...
<% if @assignment_form.assignment.varying_rubrics_by_round? %>
<% for round in 1..@assignment_form.assignment.rounds_of_reviews %>
...
<% end %>
<% else %>
...
<% end %>
</script>
</tr>
</table>
</td>
<% end %>
</pre>

Support the ability of the database to link rubrics to topics

''app\models\sign_up_topic.rb''

<pre>
class SignUpTopic < ActiveRecord::Base
...
has_many :assignment_questionnaires, class_name: 'AssignmentQuestionnaire', foreign_key: 'topic_id', dependent: :destroy
...
end
</pre>

React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox

''app/controllers/assignments_controller.rb''

<pre>
def update
...
# What to do next depends on how we got here
if params['button'].nil?
# REFRESH the topics tab after changing tabs
...
render :partial => "assignments/edit/topics"
else
# SAVE button was used (do a redirect)
redirect_to edit_assignment_path @assignment_form.assignment.id
end
end
</pre>

Support the ability to find the correct rubric for a particular topic

''app/helpers/assignment_helper.rb''

<pre>
def assignment_questionnaire(assignment, questionnaire_type, round_number = nil, topic_id = nil)
q_by_type = assignment.questionnaires.find_by(type: questionnaire_type)
if q_by_type.nil?
...
else
...
aq_by_type = assignment.assignment_questionnaires.find_by(questionnaire_id: q_by_type.id)
if round_number.nil? && topic_id.nil?
# Find by type
aq = aq_by_type
elsif topic_id.nil?
# Find by round
aq = assignment.assignment_questionnaires.find_by(used_in_round: round_number)
elsif round_number.nil?
# Find by topic
aq = assignment.assignment_questionnaires.find_by(topic_id: topic_id)
else
# Find by round and topic
aq = assignment.assignment_questionnaires.where(used_in_round: round_number, topic_id: topic_id).first
end
aq.nil? ? aq_by_type : aq
end
end
</pre>

Use the correct rubric for a particular topic

''app\controllers\response_controller.rb''

<pre>
def set_questionnaire_for_new_response
case @map.type
when "ReviewResponseMap", "SelfReviewResponseMap"
reviewees_topic = SignedUpTeam.topic_id_by_team_id(@contributor.id)
@current_round = @assignment.number_of_current_round(reviewees_topic)
@questionnaire = @map.questionnaire(@current_round, reviewees_topic)
when
...
end
end
</pre>

Support the ability to determine if an assignment has rubrics that vary by topic

''app\models\assignment.rb''

<pre>
def varying_rubrics_by_topic?
aq_no_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id, topic_id: nil).size >= 1
aq_with_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id).where.not(topic_id: nil).size >= 1
if !aq_no_topic_id && !aq_with_topic_id
return false
elsif aq_no_topic_id && !aq_with_topic_id
return false
elsif !aq_no_topic_id && aq_with_topic_id
return true
else
raise StandardError.new("Assignment with id " + self.id.to_s + " has a conflict about whether or not it varies by topic")
end
end
</pre>

== How To ==

In this section, we describe how to use the newly implemented feature.

=== Add a New Rubric ===

* [to be populated after coding is complete]

=== Select a Rubric for a Topic ===

* [to be populated after coding is complete]

== Project Testing ==

=== Strategy ===

To test code implementation, correctness of the added feature, verify that the team did not break any existing functionalities in the Expertiza, and ensure that no bugs were introduced in the code, we developed the following Test Strategy:

# Code inspection
# Run and pass existing RSpec Tests
# Develop New RSpec Tests
# Run through live UI to test a feature using Expertiza instance

=== Rspec Testing ===

==== Existing Tests ====

Run all existing RSpec tests on any changed files, after our changes, to ensure that we have not introduced any failures.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/ spec/controllers/ spec/models/
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 6978
[...]

Finished in 8 minutes 10 seconds (files took 12.32 seconds to load)
900 examples, 0 failures, 16 pending
</pre>

The features tests were extremely time-consuming so we did not routinely run them during development. However TravisCI results on our [https://github.com/expertiza/expertiza/pull/1444 Pull Request] demonstrate the passing status of all tests.

==== New Tests ====

Write comprehensive RSpec tests, for all new or modified methods.
* FILE: ''spec/helpers/assignment_helper_spec.rb''
** METHODS TO TEST:
*** questionnaire(assignment, type, round_number, topic_id)
*** assignment_questionnaire(assignment, type, number, topic_id)
* FILE: ''spec/models/review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/self_review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/assignment_spec.rb''
*** METHODS TO TEST:
**** review_questionnaire_id(round, topic_id)
**** varying_rubrics_by_topic?()

Run these tests, to ensure that the new code works as intended.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/assignment_helper_spec.rb spec/models/review_response_map_spec.rb spec/models/self_review_response_map_spec.rb spec/models/assignment_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 47508
....................................................................................................................

Finished in 1 minute 7.94 seconds (files took 10.85 seconds to load)
116 examples, 0 failures
</pre>

The test suite for a single new method is shown below.
* There are many such suites added in expertiza/spec
* This example illustrates our general strategy:
** test missing input
** test bad input
** test various acceptable forms of input
** test scenarios that lead to various return values

<pre>
describe "#questionnaire" do

...

it "throws exception if assignment argument nil" do
...
end

it "throws exception if all arguments nil except for assignment" do
...
end

it "finds by type if round number & topic id not given" do
...
end

it "throws exception if round number & topic id not given, no luck finding by type" do
...
end

it "finds by round number alone if round number alone is given" do
...
end

it "creates new questionnaire of given type if round number alone is given, no luck finding by round" do
...
end

it "finds by topic id alone if topic id alone is given" do
...
end

it "creates new questionnaire of given type if topic id alone is given, no luck finding by topic" do
...
end

it "finds by round number and topic id if both are given" do
...
end

it "creates new questionnaire of given type if round and topic are given, no luck finding by round and topic" do
...
end

end
</pre>

=== UI Testing ===

Here we describe manual UI Testing that was performed.

==== Rubric does not vary by round or by topic ====

[to be populated after coding is complete]

==== Rubric varies by round, but not by topic ====

[to be populated after coding is complete]

==== Rubric varies by topic, but not by round ====

[to be populated after coding is complete]

==== Rubric varies by both round and topic ====

[to be populated after coding is complete]

== Conclusion ==

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment. However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

In this project, we have solved this issue, by allowing rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment. The following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

The instructor may use the Rubrics tab of the assignments edit page to make changes to the "Review rubric varies by round?" and "Review rubric varies by topic?" checkboxes. The instructor may then use the Topics tab to select the appropriate rubric(s) for every topic in the assignment.

Various other Expertiza code has been updated to retrieve the correct rubric depending on round and topic after the instructor has made these selections.

== Future Work ==

This section discusses possible future work for another CSC 517 team to take on. Such a team can search the Expertiza codebase for "TODO E1936 (future work)" to find most of the places in the code that the notes below refer to.

* Reduce the delay between changing the state of the "Review rubric varies by topic?" checkbox on the Rubrics tab, and the show / hide of rubric drop-downs on the Topics tab.
* Redesign UI for tone analysis, heatmaps, and review scores pop-up. These areas are designed around the assumption that reviews do NOT vary by topic. For example, in the review scores pop-up, a reviewer can see the reviews they have given, arranged by question, across multiple reviewees.
* Consider whether the questionnaire options available for selection in drop-downs on the Topics and Rubrics tabs should include ''only'' those questionnaires created by the currently-logged-in instructor. Our work in this project changed the filtering from instructor-owning-assignment to currently-logged-in instructor - but left in place all non-private questionnaires. ''Consider'' whether non-private questionnaires should be excluded. This may result in some drop-downs not having any available questionnaires.

== Useful Links ==
In this section we provide useful links related to the work that has been completed by our team.
# [https://github.com/gabalmat/expertiza Forked Project Repository]
# [https://github.com/expertiza/expertiza/pull/1444 Pull Request]
# [(add link here) Video Demonstration of Specialized Rubrics]
# [(add link here) Video Demonstration of Passing Rspec Tests]

== Team ==
=== Contacts ===
We have 4 (four) members in our team and 1 (one) mentor was also assigned to us. For any project-related questions/concerns please contact us:
* Aurora Tiffany-Davis - attiffan@ncsu.edu
* Ginger Balmat - gabalmat@ncsu.edu
* Joe Hutchinson - jehutch3@ncsu.edu
* Nikolay Titov - ngtitov@ncsu.edu
* Zhewei Hu (mentor) - zhu6@ncsu.edu

=== Contribution Based Inquiry ===
For any questions or concerns related to the project design, implementation, testing of the feature developed by our team, please contact an engineer who worked closely on the topic of your interest based on the [https://docs.google.com/spreadsheets/d/11uxuI_5gaC-FQDwIOxtCaN3HZ9bfWDJgIjFemtYV5jE/edit?ts=5ca222f3#gid=0 Contribution Assignment Grid] we created to keep track of our progress and contribution. Please carbon copy all team members and our mentor in the email.

== References ==
* Expertiza Web [https://github.com/expertiza/expertiza#expertiza]
* Expertiza on GitHub [https://github.com/expertiza/expertiza]
* Expertiza Wiki [http://wikis.lib.ncsu.edu/index.php/Expertiza]
* Rspec Documentation [https://relishapp.com/rspec]
* Wiki with Previous Work [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types]
* Pull Request of the Previous Work [https://github.com/expertiza/expertiza/pull/1327]

File:Use case diagram.png

2019-04-26T14:46:01Z

Ngtitov: uploaded a new version of "File:Use case diagram.png"

File:Use case diagram.png

2019-04-26T14:44:16Z

Ngtitov: uploaded a new version of "File:Use case diagram.png": Reverted to version as of 14:32, 26 April 2019

File:Class diagram 1936.png

2019-04-26T14:42:09Z

Ngtitov: uploaded a new version of "File:Class diagram 1936.png"

File:Class diagram 1936.png

2019-04-26T14:38:57Z

Ngtitov:

E1936 Specialized Rubrics

2019-04-26T14:37:10Z

Ngtitov: /* Class Diagram */

This wiki page describes the work and goals that have been completed and achieved while implementing Expertiza Project "''E1936. Specialized Rubrics for Different Topic Types (e.g., Servo project, refactoring project)''" for the CSC/ECE 517 "Object-Oriented Design and Development" class, in the Spring semester of the 2019.

__TOC__

== Introduction ==
This Wiki page is created per CSC/ECE 517: "Object-Oriented Design and Development", Expertiza Project requirements (Spring, 2019).

Our [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Team team] was assigned the project "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]". This is an Expertiza project.

In this wiki page we describe the problem to be solved, how we solve that problem, and how we test our solution. We also provide helpful links and contact information for our team.

== Project Description ==

In this section, we discuss the problem we need to solve, the Expertiza feature this problem is related to, what needs to be done to solve the problem (at a high level), and the previous attempt to solve this problem.

=== Problem Statement ===

This problem statement summarizes the problem as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment.
* Code base
** Expertiza
** Mozilla
** etc.
* Goal
** Refactoring
** Testing
** etc.
However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

=== Feature Description ===

The Expertiza project already has a feature that allows rubrics to vary by project phase or "round". The feature we will add in this project is a feature to allow rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment.

When this feature is complete, the following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

=== What Needs to be Done ===

This section summarizes the work as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

* In assignment#edit page "Rubrics" tab, add a checkbox to specify whether rubric should vary by topic.
* In assignment#edit page "Topics" tab, add dropdown(s) next to each topic to specify which rubric(s) are associated with the topic.
** The dropdown(s) should appear only if rubric should vary by topic, per the Rubrics tab.
** The topic should have a multiple dropdowns (one for each round) only if rubrics vary by round, per the Rubrics tab.
** By default, the rubric(s) for each course project will be the one(s) specified in “Rubrics” tab.
** The dropdown value can overwrite the default rubric.
* Be careful when making changes to the code
** The signup_sheet_controller should have as little to do as possible with the selection of rubrics.
** Anything not germane to selecting topics should be in another controller or (more probably) a helper.
* Create a DB migration to allow rubric to vary by topic as well as by round.

==== Screenshots ====
'''Current screenshot of Rubrics tab for an Assignment'''
(need to add a checkbox to indicate that rubric should vary by topic)

[[File:E1936 Screenshot Assignment Edit Rubrics.png]]

'''Current screenshot of Topics tab for an Assignment'''
(need to add a way to select rubrics for particular topics)

[[File:E1936 Screenshot Assignment Edit Topics.png]]

'''Proposed screenshot of Rubrics tab for an Assignment'''

[[File:E1936 Specialized Rubric rubric checkbox.png]]

'''Proposed screenshot of Topics tab for an Assignment'''

(drop down shown for only one topic here, but final product will have one for each topic)

[[File:E1936 Specialized Rubric dropdown menu.png]]

== Previous Work ==

There was previous attempt to implement this project in the Fall of 2018 semester by different team, but it was rejected due to several reasons. Our team was provided with all relevant information and materials about the design and implementation of the previously attempted project. We would like to acknowledge that we use previous implementation and design as reference and our starting point for our own implementation. The following is the list of links with the relevant materials that we have used for better project understanding, determining pros and cons of previously proposed changes, and establishing our design strategy and implementation:

# [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types Wiki]
# [https://github.com/expertiza/expertiza/pull/1327 Pull Request]
# [https://drive.google.com/file/d/1UrLb0Ula_Lb_F4MOrdC-s6xng6B08VYu/view Video Demo]

After reviewing all relevant materials above, we concluded that the project implementation was incomplete and proposed design had significant flaw.
We identified main drawbacks and flaw as follows:

* Establishing incorrect ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables:
** Creating a DB migration to add ''questionnaire_id'' in the ''sign_up_topics'' table to store the dropdown data presumes ''one-to-one'' or ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables
** Correct relationship between ''sign_up_topics'' and ''questionnaires'' tables is ''many-to-many''. For example, each project topic may have various rubrics depending on the round it is used (e.g., for round 1, topic may have one rubric; but for round 2, the same topic may have different rubric)
** Proposed previous migration changes would break DB design and would not allow to store correctly topic and its associated rubric per different round.
* Per project requirement specifications, "''The dropdown box should appear only if 'Vary rubric by topic' is selected in the Rubrics tab.''" This requirement was not met and ''Vary rubric by topic'' checkbox in the Rubrics tab was not implemented. Therefore, dropdown box with the list of rubrics appears regardless of whether "''Vary rubric by topic''" is selected or not.
* The dropdown box includes the list of all possible rubrics including default value. However, this condition can be improved. The dropdown box should list the rubrics that were created by the current user (professor who is logged in). This would allow to shorten the list of possible rubrics dramatically and help to avoid any misunderstanding and confusion while user selects desired rubric for the course project.

We found no further issues with the rest implementation and in other project materials. Wiki page is very useful for understanding design of proposed changes, finding drawbacks and the flaw. Considering all the pros and cons, and taking into account drawbacks of the proposed changes, our team evolved different approach that should successfully solve the problem stated for the project without without creating additional issues. Please see our [http://wiki.expertiza.ncsu.edu/index.php?title=E1936_Specialized_Rubrics#Project_Implementation Project Implementation] for more details.

== Previous Project Requirements ==

The project requirements originally specified how to alter the databases to support varying rubrics by topic.

''Create a new table named topic_questionnaires with field:''
* ''id''
* ''topic_id''
* ''questionnaire_id''
* ''used_in_round''

However, the E1936 team successfully petitioned for this requirement to change to give us more freedom in the implementation. The original suggestion had a flaw. There is already an existing table in the Expertiza project called ''assignment_questionnaires'', which has columns including ''assignment_id'', ''questionnaire_id'', and ''used_in_round''. It is this table that maintains the knowledge of which rubric is used in which round of an assignment. If we were to add a new table to maintains the knowledge of which rubric is used in which topic of an assignment as originally suggested, then we would have a big problem. Knowledge of round-rubric relationship is duplicated. This would lead (best case scenario) to extra work, and (worst case scenario) to lots of bugs caused by inconsistencies in how these tables are handled with CRUD operations throughout the code. In other words, this is not a DRY solution.

== Project Implementation ==

In this section, we propose our solution to the problem, discuss implementation techniques and details, and list the goals we achieved.

We start by analyzing Entity Relationship Diagram where we state the solution to the problem. Further we discuss design strategy by breaking project problem into multiple staged sub-problems and provide solutions for each of these sub-problems. We also discuss some implementation details and provide the list of involved files. Finally, we demonstrate how the implemented feature can be used by providing a ''How To'' sub-section.

=== Diagrams ===

Our solution and implementation of the project can be easily shown by discussing Use Case, Entity Relationship, and Class diagrams. First, for simplicity we discuss in general possible sequence of interactions between a user (in this case instructor) and the system (Expertiza) via presenting Use Case Diagram. Then we plan to show proposed database modification via Entity Relationship Diagram.

==== User Case Diagram ====

[[File: use_case_diagram.png]]

The above Use Case Diagram shows general sequence of interactions between an instructor and the system (Expertiza) that defines where and how the feature can be utilized. Note that highlighted use cases are objects of our interest and work. Also please note that all of the shown use cases already exist in the system and we simply adding new feature into the system with existing use cases. Particularly, the rubric for each assignment and newly added ''Review rubric varies by topic'' checkbox with which instructor may choose to vary assignment by topic or not by checking and unchecking the checkbox is displayed in the ''Rubrics'' tab. Hence, we show how instructor can get to the ''Rubrics'' tab of the assignments. Similarly, ''Topics'' tab of assignments has additional ''Questionnaires'' column for each topic, where instructor may specify which rubric associates with the topic. To perform such sequence of iterations and utilize the new feature, instructor must choose ''Edit'' any assignment from the list of all assignments displayed to him/her in the ''Manage Assignments'' boundary, which can be navigated by instructor by selecting it from the header of the page.

==== Entity Relationship Diagram ====

[[File: HW04_ERD.png]]

The sections of the diagram highlighted in yellow show newly introduced items. That is, we are adding a new column called ''topic_id'' to ''assignment_questionnaires'', in order to link ''assignment_questionnaires'' to ''sign_up_topics''.

Any assignment, topic, or questionnaire may be represented by multiple records within ''assignment_questionnaires'', but each record within ''assignment_questionnaires'' refers to just one assignment, topic, or questionnaire. It is the ability to have an assignment represented by multiple records, and the new relationship to ''sign_up_topics'', that allows us to support all of the required scenarios:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

[[File: HW04_table.PNG]]

This table illustrates the possibilities that the system may have.

* Assignment 1 does not vary by round or by topic.
* Assignment 2 varies by round, but not by topic.
* Assignment 3 varies by topic, but not by round.
* Assignment 4 varies by both round and topic.

This change is done via database migration by running the following commands:

rails generate migration AddTopicIdToAssignmentQuestionnaires

Above command adds the ''topic_id'' column to the ''AssignmentQuestionnaires'' table and generates new migration file:

[[File: db_migration.png]]

To see the details of the entire database change please check this [https://github.com/gabalmat/expertiza/commit/d46dd57d35a21ab43efb82c28d3f14b75c695f2e commit].

==== Class Diagram ====

[[File: class_diagram_1936.png]]

This is Class Class diagram.

=== Design Strategy ===

As we have concrete understanding of the problem and proposed solution, we would like break this problem into multiple more manageable sub-problems with specific tasks and well-defined solutions to these tasks. Generally, each sub-problem requires modification of one or more files. We would like to propose our Design Strategy as follows:

==== Preparation ====

* Clean up code that our implementation will use
** PROBLEM: We need to add/change content to the edit assignment Topics tab, but unlike other tabs in the views, it does not have its own partial view to work with
*** SOLUTION: Refactor and move the code dealing with topics into separate ''app/views/assignments/edit/_topics.html.erb'' file
*** File: ''app/views/assignments/edit.html.erb''
** PROBLEM: In a section below, we discuss the need to modify existing methods to also accept an optional argument ''topic_id''. In many cases these existing method use confusing variable names or lack sufficient comments.
*** SOLUTION: Clean up methods that we plan to modify
*** FILE: (many)

==== Implementation ====

* Allow an instructor to indicate that rubric should vary by topic
** PROBLEM: There is no way for an instructor to indicate that an assignments' rubric should vary by topic
*** SOLUTION: Add a new checkbox in the edit assignment Rubrics tab to indicate this preference
*** FILE: ''app/views/assignments/edit/_rubrics.html.erb''
** PROBLEM: If the user changes the "Review rubric varies by round?" or "Review rubric varies by topic?" checkbox on the Rubrics tab, but leaves the page in an inappropriate state before leaving the tab (e.g. rubric weights that do not add up to 0% or 100%), then the user does not get feedback that they made a mistake. They only get such feedback when explicitly performing a "Save". Switching from tab to tab on the assignments edit page attempts to perform an implicit save, but there is no mechanism to update the flash messages onscreen if anything goes wrong. This becomes more problematic now that the Topics tab appearance depends on choices made on the Rubrics tab.
*** SOLUTION: Provide a route to an assignments controller method that can return a render of the flash messages. Add an AJAX call on the assignments edit view that can update the flash messages onscreen.
*** FILE: app/controllers/assignments_controller.rb
*** FILE: app/views/assignments/edit.html.erb

* Allow an instructor to choose different rubrics for different topics
** PROBLEM: There is no way for an instructor to choose different rubrics for different topics
*** SOLUTION: Add a new column in the edit assignment Topics tab for choosing rubric(s)
**** New column only visible if rubrics vary by topic, per Rubrics tab
**** New column has multiple rounds per topic only if rubrics vary by round, per Rubrics tab
**** Default value for any rubric is that rubric which is shown on Rubrics tab
**** Rubric(s) in the dropdown list is/are only those that were created by the currently logged in instructor
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The code in ''app/views/assignments/edit/_rubrics.html.erb'' includes dropdowns like the ones we need to add for topics, but that code is way too complex for its functionality (too heavy for a view, uses JavaScript)
*** SOLUTION: Use the code in ''app/views/assignments/edit/_rubrics.html.erb'' for inspiration only
**** Refactor and move as much actual work as possible out into helper methods that are simply called from this view
**** Views are not intended to do heavy lifting
**** Use as little JavaScript as possible, since this is a Ruby On Rails project, not a JS project
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The drop-downs for selecting rubrics should show only those rubrics created by the currently-logged in instructor (per project mentor). They currently include those created by the instructor of the relevant assignment.
*** SOLUTION: Change the filtering in the ''questionnaire_options'' method to reflect the desired filtering.
*** FILE: app/helpers/assignment_helper.rb

* Support the ability of the database to link rubrics to topics
** PROBLEM: There is no relationship in the database that can link together rubrics with topics
*** SOLUTION: Add a new migration that adds topic_id field to existing assignment_questionnaires table
**** This new field will reference the id field in the sign_up_topics table
*** FILE: ''db/migrate/[timestamp]_add_topic_id_to_assignment_questionnaires_table.rb''
** PROBLEM: The AssignmentQuestionnaire model does not have any knowledge of topics
*** SOLUTION: Add a new belongs_to reference in the model, to sign_up_topic
*** FILE: ''app/models/assignment_questionnaire.rb''
** PROBLEM: The SignUpTopic model does not have any knowledge of assignment_questionnaire records
*** SOLUTION: Add a new has_many reference in the model, to assignment_questionnaires
*** FILE: ''app/models/sign_up_topic.rb''

* React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox
** PROBLEM: There is nothing in place to remember when an instructor chooses to vary rubrics by topic, or chooses to not vary rubrics by topic
*** SOLUTION: Modify existing method ''update_assignment_questionnaires()''. This method executes upon assignment save (explicit or via tab-change). This method deletes all ''AssignmentQuestionnaire'' records before creating new correct records reflecting the user's selections in the edit assignment form. Modify such that this method can create multiple records reflecting multiple topics, in the case where rubric should vary by topic. With this modification, the correct records (with default rubrics) are present when the user goes to the Topics tab, so that the Topics tab can display properly.
*** FILE: app\models\assignment_form.rb
** PROBLEM: There is no way to refresh the Topics tab to show / hide drop-downs after the instructor changes checkbox on Rubrics tab
*** SOLUTION: Make use of existing "Save"-like functionality when changing tabs on the edit assignment page. When updating due to tab-change, cause a fresh render of the Topics tab. When updating due to "Save" button, keep functionality as-is.
*** FILE: ''app/controllers/assignments_controller.rb''
*** FILE: ''app/views/assignments/edit.html.erb''

* Support the ability to find the correct rubric for a particular topic
** PROBLEM: The existing methods allow us to find the correct rubric for a particular round, but not a particular topic
*** SOLUTION: Modify the existing method questionnaire(assignment, type, round_number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method assignment_questionnaire(assignment, type, number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method questionnaire(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/review_response_map.rb''
*** SOLUTION: Modify the existing method questionnaire(round) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/self_review_response_map.rb''
*** SOLUTION: Modify the existing method review_questionnaire_id(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/assignment.rb''

* Use the correct rubric for a particular topic
** PROBLEM: When finding rubrics to display, save results from, calculate scores from, etc. round may be used but topic is not
*** SOLUTION: Examine all callers of methods modified in the previous section and pass in ''topic_id'' where appropriate to grab the correct rubric
*** FILE: (many)

* Support the ability to determine if an assignment has rubrics that vary by topic
** PROBLEM: There is no handy method to call that can tell us if an assignment has rubrics that vary by topic
*** SOLUTION: Add a new method ''varying_rubrics_by_topic?'' based on the existing method ''varying_rubrics_by_round?''
*** FILE: ''app/models/assignment.rb''

=== Q&A ===

Here we paraphrase some questions we received during round 1 of the documentation review.

* Q: How will the new logic for varying rubric by topic tie in with the existing logic?
** A: We will add a new field ''topic_id'' into the existing table ''assignment_questionnaires'', as noted in the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram]. We will update methods that find questionnaires to allow these methods to also accept a ''topic_id'' argument, as noted in [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the "Rubric varies by topic" selection be stored?
** A: Indirectly, by saving records in the ''assignment_questionnaires'' table which either reference a ''topic_id'' or leave this field nil. For more details please see the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram] discussion.
* Q: What will happen if an instructor selects "Rubric varies by topic", makes selections on the "Topics" tab, and then deselects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will no longer show dropdowns for selecting rubrics by topic. When the assignment is saved, the appropriate not-varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: What will happen after an instructor selects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will now show dropdowns for selecting rubrics by topic. These rubrics will default to those shown on the Rubrics tab. When the assignment is saved, the appropriate varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the appropriate list of questionnaires for the new dropdowns on the Topics tab be determined?
** A: In the same way that the appropriate list of questionnaires for the existing dropdowns on the Rubrics tab is determined. Namely, by the ''questionnaire_options'' method in the file ''expertiza/app/helpers/assignment_helper.rb''.

=== (Selected) Modified Code ===

In this section, we provide ''selected portions'' of modified code to illustrate how we implemented the new feature.

Allow an instructor to indicate that rubric should vary by topic

''app\views\assignments\edit\_rubrics.html.erb''

<pre>
<input name="vary_by_topic" type="hidden" value="false" />
<%= check_box_tag('vary_by_topic', 'true', @review_vary_by_topic_check) %>
<%= label_tag('vary_by_topic', 'Review rubric varies by topic?') %>
</pre>

Allow an instructor to choose different rubrics for different topics

''app\views\sign_up_sheet\_table_line.html.erb''

<pre>
<% if @assignment_form.assignment.varying_rubrics_by_topic? %>
...
<td>
<table id = <%=table_id %>>
<tr>
<script>
...
<% if @assignment_form.assignment.varying_rubrics_by_round? %>
<% for round in 1..@assignment_form.assignment.rounds_of_reviews %>
...
<% end %>
<% else %>
...
<% end %>
</script>
</tr>
</table>
</td>
<% end %>
</pre>

Support the ability of the database to link rubrics to topics

''app\models\sign_up_topic.rb''

<pre>
class SignUpTopic < ActiveRecord::Base
...
has_many :assignment_questionnaires, class_name: 'AssignmentQuestionnaire', foreign_key: 'topic_id', dependent: :destroy
...
end
</pre>

React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox

''app/controllers/assignments_controller.rb''

<pre>
def update
...
# What to do next depends on how we got here
if params['button'].nil?
# REFRESH the topics tab after changing tabs
...
render :partial => "assignments/edit/topics"
else
# SAVE button was used (do a redirect)
redirect_to edit_assignment_path @assignment_form.assignment.id
end
end
</pre>

Support the ability to find the correct rubric for a particular topic

''app/helpers/assignment_helper.rb''

<pre>
def assignment_questionnaire(assignment, questionnaire_type, round_number = nil, topic_id = nil)
q_by_type = assignment.questionnaires.find_by(type: questionnaire_type)
if q_by_type.nil?
...
else
...
aq_by_type = assignment.assignment_questionnaires.find_by(questionnaire_id: q_by_type.id)
if round_number.nil? && topic_id.nil?
# Find by type
aq = aq_by_type
elsif topic_id.nil?
# Find by round
aq = assignment.assignment_questionnaires.find_by(used_in_round: round_number)
elsif round_number.nil?
# Find by topic
aq = assignment.assignment_questionnaires.find_by(topic_id: topic_id)
else
# Find by round and topic
aq = assignment.assignment_questionnaires.where(used_in_round: round_number, topic_id: topic_id).first
end
aq.nil? ? aq_by_type : aq
end
end
</pre>

Use the correct rubric for a particular topic

''app\controllers\response_controller.rb''

<pre>
def set_questionnaire_for_new_response
case @map.type
when "ReviewResponseMap", "SelfReviewResponseMap"
reviewees_topic = SignedUpTeam.topic_id_by_team_id(@contributor.id)
@current_round = @assignment.number_of_current_round(reviewees_topic)
@questionnaire = @map.questionnaire(@current_round, reviewees_topic)
when
...
end
end
</pre>

Support the ability to determine if an assignment has rubrics that vary by topic

''app\models\assignment.rb''

<pre>
def varying_rubrics_by_topic?
aq_no_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id, topic_id: nil).size >= 1
aq_with_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id).where.not(topic_id: nil).size >= 1
if !aq_no_topic_id && !aq_with_topic_id
return false
elsif aq_no_topic_id && !aq_with_topic_id
return false
elsif !aq_no_topic_id && aq_with_topic_id
return true
else
raise StandardError.new("Assignment with id " + self.id.to_s + " has a conflict about whether or not it varies by topic")
end
end
</pre>

== How To ==

In this section, we describe how to use the newly implemented feature.

=== Add a New Rubric ===

* [to be populated after coding is complete]

=== Select a Rubric for a Topic ===

* [to be populated after coding is complete]

== Project Testing ==

=== Strategy ===

To test code implementation, correctness of the added feature, verify that the team did not break any existing functionalities in the Expertiza, and ensure that no bugs were introduced in the code, we developed the following Test Strategy:

# Code inspection
# Run and pass existing RSpec Tests
# Develop New RSpec Tests
# Run through live UI to test a feature using Expertiza instance

=== Rspec Testing ===

==== Existing Tests ====

Run all existing RSpec tests on any changed files, after our changes, to ensure that we have not introduced any failures.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/ spec/controllers/ spec/models/
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 6978
[...]

Finished in 8 minutes 10 seconds (files took 12.32 seconds to load)
900 examples, 0 failures, 16 pending
</pre>

The features tests were extremely time-consuming so we did not routinely run them during development. However TravisCI results on our [https://github.com/expertiza/expertiza/pull/1444 Pull Request] demonstrate the passing status of all tests.

==== New Tests ====

Write comprehensive RSpec tests, for all new or modified methods.
* FILE: ''spec/helpers/assignment_helper_spec.rb''
** METHODS TO TEST:
*** questionnaire(assignment, type, round_number, topic_id)
*** assignment_questionnaire(assignment, type, number, topic_id)
* FILE: ''spec/models/review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/self_review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/assignment_spec.rb''
*** METHODS TO TEST:
**** review_questionnaire_id(round, topic_id)
**** varying_rubrics_by_topic?()

Run these tests, to ensure that the new code works as intended.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/assignment_helper_spec.rb spec/models/review_response_map_spec.rb spec/models/self_review_response_map_spec.rb spec/models/assignment_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 47508
....................................................................................................................

Finished in 1 minute 7.94 seconds (files took 10.85 seconds to load)
116 examples, 0 failures
</pre>

The test suite for a single new method is shown below.
* There are many such suites added in expertiza/spec
* This example illustrates our general strategy:
** test missing input
** test bad input
** test various acceptable forms of input
** test scenarios that lead to various return values

<pre>
describe "#questionnaire" do

...

it "throws exception if assignment argument nil" do
...
end

it "throws exception if all arguments nil except for assignment" do
...
end

it "finds by type if round number & topic id not given" do
...
end

it "throws exception if round number & topic id not given, no luck finding by type" do
...
end

it "finds by round number alone if round number alone is given" do
...
end

it "creates new questionnaire of given type if round number alone is given, no luck finding by round" do
...
end

it "finds by topic id alone if topic id alone is given" do
...
end

it "creates new questionnaire of given type if topic id alone is given, no luck finding by topic" do
...
end

it "finds by round number and topic id if both are given" do
...
end

it "creates new questionnaire of given type if round and topic are given, no luck finding by round and topic" do
...
end

end
</pre>

=== UI Testing ===

Here we describe manual UI Testing that was performed.

==== Rubric does not vary by round or by topic ====

[to be populated after coding is complete]

==== Rubric varies by round, but not by topic ====

[to be populated after coding is complete]

==== Rubric varies by topic, but not by round ====

[to be populated after coding is complete]

==== Rubric varies by both round and topic ====

[to be populated after coding is complete]

== Conclusion ==

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment. However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

In this project, we have solved this issue, by allowing rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment. The following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

The instructor may use the Rubrics tab of the assignments edit page to make changes to the "Review rubric varies by round?" and "Review rubric varies by topic?" checkboxes. The instructor may then use the Topics tab to select the appropriate rubric(s) for every topic in the assignment.

Various other Expertiza code has been updated to retrieve the correct rubric depending on round and topic after the instructor has made these selections.

== Future Work ==

This section discusses possible future work for another CSC 517 team to take on. Such a team can search the Expertiza codebase for "TODO E1936 (future work)" to find most of the places in the code that the notes below refer to.

* Reduce the delay between changing the state of the "Review rubric varies by topic?" checkbox on the Rubrics tab, and the show / hide of rubric drop-downs on the Topics tab.
* Redesign UI for tone analysis, heatmaps, and review scores pop-up. These areas are designed around the assumption that reviews do NOT vary by topic. For example, in the review scores pop-up, a reviewer can see the reviews they have given, arranged by question, across multiple reviewees.
* Consider whether the questionnaire options available for selection in drop-downs on the Topics and Rubrics tabs should include ''only'' those questionnaires created by the currently-logged-in instructor. Our work in this project changed the filtering from instructor-owning-assignment to currently-logged-in instructor - but left in place all non-private questionnaires. ''Consider'' whether non-private questionnaires should be excluded. This may result in some drop-downs not having any available questionnaires.

== Useful Links ==
In this section we provide useful links related to the work that has been completed by our team.
# [https://github.com/gabalmat/expertiza Forked Project Repository]
# [https://github.com/expertiza/expertiza/pull/1444 Pull Request]
# [(add link here) Video Demonstration of Specialized Rubrics]
# [(add link here) Video Demonstration of Passing Rspec Tests]

== Team ==
=== Contacts ===
We have 4 (four) members in our team and 1 (one) mentor was also assigned to us. For any project-related questions/concerns please contact us:
* Aurora Tiffany-Davis - attiffan@ncsu.edu
* Ginger Balmat - gabalmat@ncsu.edu
* Joe Hutchinson - jehutch3@ncsu.edu
* Nikolay Titov - ngtitov@ncsu.edu
* Zhewei Hu (mentor) - zhu6@ncsu.edu

=== Contribution Based Inquiry ===
For any questions or concerns related to the project design, implementation, testing of the feature developed by our team, please contact an engineer who worked closely on the topic of your interest based on the [https://docs.google.com/spreadsheets/d/11uxuI_5gaC-FQDwIOxtCaN3HZ9bfWDJgIjFemtYV5jE/edit?ts=5ca222f3#gid=0 Contribution Assignment Grid] we created to keep track of our progress and contribution. Please carbon copy all team members and our mentor in the email.

== References ==
* Expertiza Web [https://github.com/expertiza/expertiza#expertiza]
* Expertiza on GitHub [https://github.com/expertiza/expertiza]
* Expertiza Wiki [http://wikis.lib.ncsu.edu/index.php/Expertiza]
* Rspec Documentation [https://relishapp.com/rspec]
* Wiki with Previous Work [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types]
* Pull Request of the Previous Work [https://github.com/expertiza/expertiza/pull/1327]

E1936 Specialized Rubrics

2019-04-26T14:36:15Z

Ngtitov: /* Class Diagram */

This wiki page describes the work and goals that have been completed and achieved while implementing Expertiza Project "''E1936. Specialized Rubrics for Different Topic Types (e.g., Servo project, refactoring project)''" for the CSC/ECE 517 "Object-Oriented Design and Development" class, in the Spring semester of the 2019.

__TOC__

== Introduction ==
This Wiki page is created per CSC/ECE 517: "Object-Oriented Design and Development", Expertiza Project requirements (Spring, 2019).

Our [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Team team] was assigned the project "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]". This is an Expertiza project.

In this wiki page we describe the problem to be solved, how we solve that problem, and how we test our solution. We also provide helpful links and contact information for our team.

== Project Description ==

In this section, we discuss the problem we need to solve, the Expertiza feature this problem is related to, what needs to be done to solve the problem (at a high level), and the previous attempt to solve this problem.

=== Problem Statement ===

This problem statement summarizes the problem as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment.
* Code base
** Expertiza
** Mozilla
** etc.
* Goal
** Refactoring
** Testing
** etc.
However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

=== Feature Description ===

The Expertiza project already has a feature that allows rubrics to vary by project phase or "round". The feature we will add in this project is a feature to allow rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment.

When this feature is complete, the following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

=== What Needs to be Done ===

This section summarizes the work as described in the project description "[https://docs.google.com/document/d/1Ozw2Bj2u_LyeKUdDqAyQtz1stEIKrxkmRnwmNyUgmzw/edit#heading=h.z86zstcoj6u7 E1936. Specialized rubrics for different topic types (e.g., Servo project, refactoring project)]".

* In assignment#edit page "Rubrics" tab, add a checkbox to specify whether rubric should vary by topic.
* In assignment#edit page "Topics" tab, add dropdown(s) next to each topic to specify which rubric(s) are associated with the topic.
** The dropdown(s) should appear only if rubric should vary by topic, per the Rubrics tab.
** The topic should have a multiple dropdowns (one for each round) only if rubrics vary by round, per the Rubrics tab.
** By default, the rubric(s) for each course project will be the one(s) specified in “Rubrics” tab.
** The dropdown value can overwrite the default rubric.
* Be careful when making changes to the code
** The signup_sheet_controller should have as little to do as possible with the selection of rubrics.
** Anything not germane to selecting topics should be in another controller or (more probably) a helper.
* Create a DB migration to allow rubric to vary by topic as well as by round.

==== Screenshots ====
'''Current screenshot of Rubrics tab for an Assignment'''
(need to add a checkbox to indicate that rubric should vary by topic)

[[File:E1936 Screenshot Assignment Edit Rubrics.png]]

'''Current screenshot of Topics tab for an Assignment'''
(need to add a way to select rubrics for particular topics)

[[File:E1936 Screenshot Assignment Edit Topics.png]]

'''Proposed screenshot of Rubrics tab for an Assignment'''

[[File:E1936 Specialized Rubric rubric checkbox.png]]

'''Proposed screenshot of Topics tab for an Assignment'''

(drop down shown for only one topic here, but final product will have one for each topic)

[[File:E1936 Specialized Rubric dropdown menu.png]]

== Previous Work ==

There was previous attempt to implement this project in the Fall of 2018 semester by different team, but it was rejected due to several reasons. Our team was provided with all relevant information and materials about the design and implementation of the previously attempted project. We would like to acknowledge that we use previous implementation and design as reference and our starting point for our own implementation. The following is the list of links with the relevant materials that we have used for better project understanding, determining pros and cons of previously proposed changes, and establishing our design strategy and implementation:

# [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types Wiki]
# [https://github.com/expertiza/expertiza/pull/1327 Pull Request]
# [https://drive.google.com/file/d/1UrLb0Ula_Lb_F4MOrdC-s6xng6B08VYu/view Video Demo]

After reviewing all relevant materials above, we concluded that the project implementation was incomplete and proposed design had significant flaw.
We identified main drawbacks and flaw as follows:

* Establishing incorrect ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables:
** Creating a DB migration to add ''questionnaire_id'' in the ''sign_up_topics'' table to store the dropdown data presumes ''one-to-one'' or ''many-to-one'' relationship between ''sign_up_topics'' and ''questionnaires'' tables
** Correct relationship between ''sign_up_topics'' and ''questionnaires'' tables is ''many-to-many''. For example, each project topic may have various rubrics depending on the round it is used (e.g., for round 1, topic may have one rubric; but for round 2, the same topic may have different rubric)
** Proposed previous migration changes would break DB design and would not allow to store correctly topic and its associated rubric per different round.
* Per project requirement specifications, "''The dropdown box should appear only if 'Vary rubric by topic' is selected in the Rubrics tab.''" This requirement was not met and ''Vary rubric by topic'' checkbox in the Rubrics tab was not implemented. Therefore, dropdown box with the list of rubrics appears regardless of whether "''Vary rubric by topic''" is selected or not.
* The dropdown box includes the list of all possible rubrics including default value. However, this condition can be improved. The dropdown box should list the rubrics that were created by the current user (professor who is logged in). This would allow to shorten the list of possible rubrics dramatically and help to avoid any misunderstanding and confusion while user selects desired rubric for the course project.

We found no further issues with the rest implementation and in other project materials. Wiki page is very useful for understanding design of proposed changes, finding drawbacks and the flaw. Considering all the pros and cons, and taking into account drawbacks of the proposed changes, our team evolved different approach that should successfully solve the problem stated for the project without without creating additional issues. Please see our [http://wiki.expertiza.ncsu.edu/index.php?title=E1936_Specialized_Rubrics#Project_Implementation Project Implementation] for more details.

== Previous Project Requirements ==

The project requirements originally specified how to alter the databases to support varying rubrics by topic.

''Create a new table named topic_questionnaires with field:''
* ''id''
* ''topic_id''
* ''questionnaire_id''
* ''used_in_round''

However, the E1936 team successfully petitioned for this requirement to change to give us more freedom in the implementation. The original suggestion had a flaw. There is already an existing table in the Expertiza project called ''assignment_questionnaires'', which has columns including ''assignment_id'', ''questionnaire_id'', and ''used_in_round''. It is this table that maintains the knowledge of which rubric is used in which round of an assignment. If we were to add a new table to maintains the knowledge of which rubric is used in which topic of an assignment as originally suggested, then we would have a big problem. Knowledge of round-rubric relationship is duplicated. This would lead (best case scenario) to extra work, and (worst case scenario) to lots of bugs caused by inconsistencies in how these tables are handled with CRUD operations throughout the code. In other words, this is not a DRY solution.

== Project Implementation ==

In this section, we propose our solution to the problem, discuss implementation techniques and details, and list the goals we achieved.

We start by analyzing Entity Relationship Diagram where we state the solution to the problem. Further we discuss design strategy by breaking project problem into multiple staged sub-problems and provide solutions for each of these sub-problems. We also discuss some implementation details and provide the list of involved files. Finally, we demonstrate how the implemented feature can be used by providing a ''How To'' sub-section.

=== Diagrams ===

Our solution and implementation of the project can be easily shown by discussing Use Case, Entity Relationship, and Class diagrams. First, for simplicity we discuss in general possible sequence of interactions between a user (in this case instructor) and the system (Expertiza) via presenting Use Case Diagram. Then we plan to show proposed database modification via Entity Relationship Diagram.

==== User Case Diagram ====

[[File: use_case_diagram.png]]

The above Use Case Diagram shows general sequence of interactions between an instructor and the system (Expertiza) that defines where and how the feature can be utilized. Note that highlighted use cases are objects of our interest and work. Also please note that all of the shown use cases already exist in the system and we simply adding new feature into the system with existing use cases. Particularly, the rubric for each assignment and newly added ''Review rubric varies by topic'' checkbox with which instructor may choose to vary assignment by topic or not by checking and unchecking the checkbox is displayed in the ''Rubrics'' tab. Hence, we show how instructor can get to the ''Rubrics'' tab of the assignments. Similarly, ''Topics'' tab of assignments has additional ''Questionnaires'' column for each topic, where instructor may specify which rubric associates with the topic. To perform such sequence of iterations and utilize the new feature, instructor must choose ''Edit'' any assignment from the list of all assignments displayed to him/her in the ''Manage Assignments'' boundary, which can be navigated by instructor by selecting it from the header of the page.

==== Entity Relationship Diagram ====

[[File: HW04_ERD.png]]

The sections of the diagram highlighted in yellow show newly introduced items. That is, we are adding a new column called ''topic_id'' to ''assignment_questionnaires'', in order to link ''assignment_questionnaires'' to ''sign_up_topics''.

Any assignment, topic, or questionnaire may be represented by multiple records within ''assignment_questionnaires'', but each record within ''assignment_questionnaires'' refers to just one assignment, topic, or questionnaire. It is the ability to have an assignment represented by multiple records, and the new relationship to ''sign_up_topics'', that allows us to support all of the required scenarios:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

[[File: HW04_table.PNG]]

This table illustrates the possibilities that the system may have.

* Assignment 1 does not vary by round or by topic.
* Assignment 2 varies by round, but not by topic.
* Assignment 3 varies by topic, but not by round.
* Assignment 4 varies by both round and topic.

This change is done via database migration by running the following commands:

rails generate migration AddTopicIdToAssignmentQuestionnaires

Above command adds the ''topic_id'' column to the ''AssignmentQuestionnaires'' table and generates new migration file:

[[File: db_migration.png]]

To see the details of the entire database change please check this [https://github.com/gabalmat/expertiza/commit/d46dd57d35a21ab43efb82c28d3f14b75c695f2e commit].

==== Class Diagram ====

[[File: class_diagram.png]]

This is Class Class diagram.

=== Design Strategy ===

As we have concrete understanding of the problem and proposed solution, we would like break this problem into multiple more manageable sub-problems with specific tasks and well-defined solutions to these tasks. Generally, each sub-problem requires modification of one or more files. We would like to propose our Design Strategy as follows:

==== Preparation ====

* Clean up code that our implementation will use
** PROBLEM: We need to add/change content to the edit assignment Topics tab, but unlike other tabs in the views, it does not have its own partial view to work with
*** SOLUTION: Refactor and move the code dealing with topics into separate ''app/views/assignments/edit/_topics.html.erb'' file
*** File: ''app/views/assignments/edit.html.erb''
** PROBLEM: In a section below, we discuss the need to modify existing methods to also accept an optional argument ''topic_id''. In many cases these existing method use confusing variable names or lack sufficient comments.
*** SOLUTION: Clean up methods that we plan to modify
*** FILE: (many)

==== Implementation ====

* Allow an instructor to indicate that rubric should vary by topic
** PROBLEM: There is no way for an instructor to indicate that an assignments' rubric should vary by topic
*** SOLUTION: Add a new checkbox in the edit assignment Rubrics tab to indicate this preference
*** FILE: ''app/views/assignments/edit/_rubrics.html.erb''
** PROBLEM: If the user changes the "Review rubric varies by round?" or "Review rubric varies by topic?" checkbox on the Rubrics tab, but leaves the page in an inappropriate state before leaving the tab (e.g. rubric weights that do not add up to 0% or 100%), then the user does not get feedback that they made a mistake. They only get such feedback when explicitly performing a "Save". Switching from tab to tab on the assignments edit page attempts to perform an implicit save, but there is no mechanism to update the flash messages onscreen if anything goes wrong. This becomes more problematic now that the Topics tab appearance depends on choices made on the Rubrics tab.
*** SOLUTION: Provide a route to an assignments controller method that can return a render of the flash messages. Add an AJAX call on the assignments edit view that can update the flash messages onscreen.
*** FILE: app/controllers/assignments_controller.rb
*** FILE: app/views/assignments/edit.html.erb

* Allow an instructor to choose different rubrics for different topics
** PROBLEM: There is no way for an instructor to choose different rubrics for different topics
*** SOLUTION: Add a new column in the edit assignment Topics tab for choosing rubric(s)
**** New column only visible if rubrics vary by topic, per Rubrics tab
**** New column has multiple rounds per topic only if rubrics vary by round, per Rubrics tab
**** Default value for any rubric is that rubric which is shown on Rubrics tab
**** Rubric(s) in the dropdown list is/are only those that were created by the currently logged in instructor
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The code in ''app/views/assignments/edit/_rubrics.html.erb'' includes dropdowns like the ones we need to add for topics, but that code is way too complex for its functionality (too heavy for a view, uses JavaScript)
*** SOLUTION: Use the code in ''app/views/assignments/edit/_rubrics.html.erb'' for inspiration only
**** Refactor and move as much actual work as possible out into helper methods that are simply called from this view
**** Views are not intended to do heavy lifting
**** Use as little JavaScript as possible, since this is a Ruby On Rails project, not a JS project
*** FILE: ''app/views/assignments/edit/_topics.html.erb''
** PROBLEM: The drop-downs for selecting rubrics should show only those rubrics created by the currently-logged in instructor (per project mentor). They currently include those created by the instructor of the relevant assignment.
*** SOLUTION: Change the filtering in the ''questionnaire_options'' method to reflect the desired filtering.
*** FILE: app/helpers/assignment_helper.rb

* Support the ability of the database to link rubrics to topics
** PROBLEM: There is no relationship in the database that can link together rubrics with topics
*** SOLUTION: Add a new migration that adds topic_id field to existing assignment_questionnaires table
**** This new field will reference the id field in the sign_up_topics table
*** FILE: ''db/migrate/[timestamp]_add_topic_id_to_assignment_questionnaires_table.rb''
** PROBLEM: The AssignmentQuestionnaire model does not have any knowledge of topics
*** SOLUTION: Add a new belongs_to reference in the model, to sign_up_topic
*** FILE: ''app/models/assignment_questionnaire.rb''
** PROBLEM: The SignUpTopic model does not have any knowledge of assignment_questionnaire records
*** SOLUTION: Add a new has_many reference in the model, to assignment_questionnaires
*** FILE: ''app/models/sign_up_topic.rb''

* React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox
** PROBLEM: There is nothing in place to remember when an instructor chooses to vary rubrics by topic, or chooses to not vary rubrics by topic
*** SOLUTION: Modify existing method ''update_assignment_questionnaires()''. This method executes upon assignment save (explicit or via tab-change). This method deletes all ''AssignmentQuestionnaire'' records before creating new correct records reflecting the user's selections in the edit assignment form. Modify such that this method can create multiple records reflecting multiple topics, in the case where rubric should vary by topic. With this modification, the correct records (with default rubrics) are present when the user goes to the Topics tab, so that the Topics tab can display properly.
*** FILE: app\models\assignment_form.rb
** PROBLEM: There is no way to refresh the Topics tab to show / hide drop-downs after the instructor changes checkbox on Rubrics tab
*** SOLUTION: Make use of existing "Save"-like functionality when changing tabs on the edit assignment page. When updating due to tab-change, cause a fresh render of the Topics tab. When updating due to "Save" button, keep functionality as-is.
*** FILE: ''app/controllers/assignments_controller.rb''
*** FILE: ''app/views/assignments/edit.html.erb''

* Support the ability to find the correct rubric for a particular topic
** PROBLEM: The existing methods allow us to find the correct rubric for a particular round, but not a particular topic
*** SOLUTION: Modify the existing method questionnaire(assignment, type, round_number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method assignment_questionnaire(assignment, type, number) to also accept an optional argument ''topic_id''
*** FILE: ''app/helpers/assignment_helper.rb''
*** SOLUTION: Modify the existing method questionnaire(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/review_response_map.rb''
*** SOLUTION: Modify the existing method questionnaire(round) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/self_review_response_map.rb''
*** SOLUTION: Modify the existing method review_questionnaire_id(round = nil) to also accept an optional argument ''topic_id''
*** FILE: ''app/models/assignment.rb''

* Use the correct rubric for a particular topic
** PROBLEM: When finding rubrics to display, save results from, calculate scores from, etc. round may be used but topic is not
*** SOLUTION: Examine all callers of methods modified in the previous section and pass in ''topic_id'' where appropriate to grab the correct rubric
*** FILE: (many)

* Support the ability to determine if an assignment has rubrics that vary by topic
** PROBLEM: There is no handy method to call that can tell us if an assignment has rubrics that vary by topic
*** SOLUTION: Add a new method ''varying_rubrics_by_topic?'' based on the existing method ''varying_rubrics_by_round?''
*** FILE: ''app/models/assignment.rb''

=== Q&A ===

Here we paraphrase some questions we received during round 1 of the documentation review.

* Q: How will the new logic for varying rubric by topic tie in with the existing logic?
** A: We will add a new field ''topic_id'' into the existing table ''assignment_questionnaires'', as noted in the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram]. We will update methods that find questionnaires to allow these methods to also accept a ''topic_id'' argument, as noted in [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the "Rubric varies by topic" selection be stored?
** A: Indirectly, by saving records in the ''assignment_questionnaires'' table which either reference a ''topic_id'' or leave this field nil. For more details please see the [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Entity_Relationship_Diagram E/R Diagram] discussion.
* Q: What will happen if an instructor selects "Rubric varies by topic", makes selections on the "Topics" tab, and then deselects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will no longer show dropdowns for selecting rubrics by topic. When the assignment is saved, the appropriate not-varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: What will happen after an instructor selects "Rubric varies by topic"?
** A: The Topics tab, when next visited, will now show dropdowns for selecting rubrics by topic. These rubrics will default to those shown on the Rubrics tab. When the assignment is saved, the appropriate varying-by-topic records will be saved in the ''assignment_questionnaires'' table. For more details please see [http://wiki.expertiza.ncsu.edu/index.php/E1936_Specialized_Rubrics#Implementation Implementation].
* Q: How will the appropriate list of questionnaires for the new dropdowns on the Topics tab be determined?
** A: In the same way that the appropriate list of questionnaires for the existing dropdowns on the Rubrics tab is determined. Namely, by the ''questionnaire_options'' method in the file ''expertiza/app/helpers/assignment_helper.rb''.

=== (Selected) Modified Code ===

In this section, we provide ''selected portions'' of modified code to illustrate how we implemented the new feature.

Allow an instructor to indicate that rubric should vary by topic

''app\views\assignments\edit\_rubrics.html.erb''

<pre>
<input name="vary_by_topic" type="hidden" value="false" />
<%= check_box_tag('vary_by_topic', 'true', @review_vary_by_topic_check) %>
<%= label_tag('vary_by_topic', 'Review rubric varies by topic?') %>
</pre>

Allow an instructor to choose different rubrics for different topics

''app\views\sign_up_sheet\_table_line.html.erb''

<pre>
<% if @assignment_form.assignment.varying_rubrics_by_topic? %>
...
<td>
<table id = <%=table_id %>>
<tr>
<script>
...
<% if @assignment_form.assignment.varying_rubrics_by_round? %>
<% for round in 1..@assignment_form.assignment.rounds_of_reviews %>
...
<% end %>
<% else %>
...
<% end %>
</script>
</tr>
</table>
</td>
<% end %>
</pre>

Support the ability of the database to link rubrics to topics

''app\models\sign_up_topic.rb''

<pre>
class SignUpTopic < ActiveRecord::Base
...
has_many :assignment_questionnaires, class_name: 'AssignmentQuestionnaire', foreign_key: 'topic_id', dependent: :destroy
...
end
</pre>

React to an instructor saving an assignment after altering the rubrics-vary-by-topic checkbox

''app/controllers/assignments_controller.rb''

<pre>
def update
...
# What to do next depends on how we got here
if params['button'].nil?
# REFRESH the topics tab after changing tabs
...
render :partial => "assignments/edit/topics"
else
# SAVE button was used (do a redirect)
redirect_to edit_assignment_path @assignment_form.assignment.id
end
end
</pre>

Support the ability to find the correct rubric for a particular topic

''app/helpers/assignment_helper.rb''

<pre>
def assignment_questionnaire(assignment, questionnaire_type, round_number = nil, topic_id = nil)
q_by_type = assignment.questionnaires.find_by(type: questionnaire_type)
if q_by_type.nil?
...
else
...
aq_by_type = assignment.assignment_questionnaires.find_by(questionnaire_id: q_by_type.id)
if round_number.nil? && topic_id.nil?
# Find by type
aq = aq_by_type
elsif topic_id.nil?
# Find by round
aq = assignment.assignment_questionnaires.find_by(used_in_round: round_number)
elsif round_number.nil?
# Find by topic
aq = assignment.assignment_questionnaires.find_by(topic_id: topic_id)
else
# Find by round and topic
aq = assignment.assignment_questionnaires.where(used_in_round: round_number, topic_id: topic_id).first
end
aq.nil? ? aq_by_type : aq
end
end
</pre>

Use the correct rubric for a particular topic

''app\controllers\response_controller.rb''

<pre>
def set_questionnaire_for_new_response
case @map.type
when "ReviewResponseMap", "SelfReviewResponseMap"
reviewees_topic = SignedUpTeam.topic_id_by_team_id(@contributor.id)
@current_round = @assignment.number_of_current_round(reviewees_topic)
@questionnaire = @map.questionnaire(@current_round, reviewees_topic)
when
...
end
end
</pre>

Support the ability to determine if an assignment has rubrics that vary by topic

''app\models\assignment.rb''

<pre>
def varying_rubrics_by_topic?
aq_no_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id, topic_id: nil).size >= 1
aq_with_topic_id = AssignmentQuestionnaire.where(assignment_id: self.id).where.not(topic_id: nil).size >= 1
if !aq_no_topic_id && !aq_with_topic_id
return false
elsif aq_no_topic_id && !aq_with_topic_id
return false
elsif !aq_no_topic_id && aq_with_topic_id
return true
else
raise StandardError.new("Assignment with id " + self.id.to_s + " has a conflict about whether or not it varies by topic")
end
end
</pre>

== How To ==

In this section, we describe how to use the newly implemented feature.

=== Add a New Rubric ===

* [to be populated after coding is complete]

=== Select a Rubric for a Topic ===

* [to be populated after coding is complete]

== Project Testing ==

=== Strategy ===

To test code implementation, correctness of the added feature, verify that the team did not break any existing functionalities in the Expertiza, and ensure that no bugs were introduced in the code, we developed the following Test Strategy:

# Code inspection
# Run and pass existing RSpec Tests
# Develop New RSpec Tests
# Run through live UI to test a feature using Expertiza instance

=== Rspec Testing ===

==== Existing Tests ====

Run all existing RSpec tests on any changed files, after our changes, to ensure that we have not introduced any failures.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/ spec/controllers/ spec/models/
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 6978
[...]

Finished in 8 minutes 10 seconds (files took 12.32 seconds to load)
900 examples, 0 failures, 16 pending
</pre>

The features tests were extremely time-consuming so we did not routinely run them during development. However TravisCI results on our [https://github.com/expertiza/expertiza/pull/1444 Pull Request] demonstrate the passing status of all tests.

==== New Tests ====

Write comprehensive RSpec tests, for all new or modified methods.
* FILE: ''spec/helpers/assignment_helper_spec.rb''
** METHODS TO TEST:
*** questionnaire(assignment, type, round_number, topic_id)
*** assignment_questionnaire(assignment, type, number, topic_id)
* FILE: ''spec/models/review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/self_review_response_map_spec.rb''
*** METHODS TO TEST:
**** questionnaire(round, topic_id)
* FILE: ''spec/models/assignment_spec.rb''
*** METHODS TO TEST:
**** review_questionnaire_id(round, topic_id)
**** varying_rubrics_by_topic?()

Run these tests, to ensure that the new code works as intended.
The commands and results are shown below.

<pre>
e@ubuntu:~/Desktop/CSC517_HW04/expertiza$ bundle exec rspec spec/helpers/assignment_helper_spec.rb spec/models/review_response_map_spec.rb spec/models/self_review_response_map_spec.rb spec/models/assignment_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

Randomized with seed 47508
....................................................................................................................

Finished in 1 minute 7.94 seconds (files took 10.85 seconds to load)
116 examples, 0 failures
</pre>

The test suite for a single new method is shown below.
* There are many such suites added in expertiza/spec
* This example illustrates our general strategy:
** test missing input
** test bad input
** test various acceptable forms of input
** test scenarios that lead to various return values

<pre>
describe "#questionnaire" do

...

it "throws exception if assignment argument nil" do
...
end

it "throws exception if all arguments nil except for assignment" do
...
end

it "finds by type if round number & topic id not given" do
...
end

it "throws exception if round number & topic id not given, no luck finding by type" do
...
end

it "finds by round number alone if round number alone is given" do
...
end

it "creates new questionnaire of given type if round number alone is given, no luck finding by round" do
...
end

it "finds by topic id alone if topic id alone is given" do
...
end

it "creates new questionnaire of given type if topic id alone is given, no luck finding by topic" do
...
end

it "finds by round number and topic id if both are given" do
...
end

it "creates new questionnaire of given type if round and topic are given, no luck finding by round and topic" do
...
end

end
</pre>

=== UI Testing ===

Here we describe manual UI Testing that was performed.

==== Rubric does not vary by round or by topic ====

[to be populated after coding is complete]

==== Rubric varies by round, but not by topic ====

[to be populated after coding is complete]

==== Rubric varies by topic, but not by round ====

[to be populated after coding is complete]

==== Rubric varies by both round and topic ====

[to be populated after coding is complete]

== Conclusion ==

In CSC/ECE 517, there are several types of topics that could be covered in a single class assignment. However, currently we can only specify one kind of rubric for an assignment. This means that teams working on different topics will be evaluated using the same rubric. With this setup, it's not possible to fine-tune rubrics for different topics - rubrics tend to be overly general.

In this project, we have solved this issue, by allowing rubrics to vary not only by round, but also by topic. This allows more flexibility to the instructor when setting up an assignment, so that they can use rubrics better suited to the tasks that students are performing in an assignment. The following scenarios will be possible, and the instructor will be the judge of which scenario is the best fit to the assignment:

* Rubric does not vary by round or by topic.
* Rubric varies by round, but not by topic.
* Rubric varies by topic, but not by round.
* Rubric varies by both round and topic.

The instructor may use the Rubrics tab of the assignments edit page to make changes to the "Review rubric varies by round?" and "Review rubric varies by topic?" checkboxes. The instructor may then use the Topics tab to select the appropriate rubric(s) for every topic in the assignment.

Various other Expertiza code has been updated to retrieve the correct rubric depending on round and topic after the instructor has made these selections.

== Future Work ==

This section discusses possible future work for another CSC 517 team to take on. Such a team can search the Expertiza codebase for "TODO E1936 (future work)" to find most of the places in the code that the notes below refer to.

* Reduce the delay between changing the state of the "Review rubric varies by topic?" checkbox on the Rubrics tab, and the show / hide of rubric drop-downs on the Topics tab.
* Redesign UI for tone analysis, heatmaps, and review scores pop-up. These areas are designed around the assumption that reviews do NOT vary by topic. For example, in the review scores pop-up, a reviewer can see the reviews they have given, arranged by question, across multiple reviewees.
* Consider whether the questionnaire options available for selection in drop-downs on the Topics and Rubrics tabs should include ''only'' those questionnaires created by the currently-logged-in instructor. Our work in this project changed the filtering from instructor-owning-assignment to currently-logged-in instructor - but left in place all non-private questionnaires. ''Consider'' whether non-private questionnaires should be excluded. This may result in some drop-downs not having any available questionnaires.

== Useful Links ==
In this section we provide useful links related to the work that has been completed by our team.
# [https://github.com/gabalmat/expertiza Forked Project Repository]
# [https://github.com/expertiza/expertiza/pull/1444 Pull Request]
# [(add link here) Video Demonstration of Specialized Rubrics]
# [(add link here) Video Demonstration of Passing Rspec Tests]

== Team ==
=== Contacts ===
We have 4 (four) members in our team and 1 (one) mentor was also assigned to us. For any project-related questions/concerns please contact us:
* Aurora Tiffany-Davis - attiffan@ncsu.edu
* Ginger Balmat - gabalmat@ncsu.edu
* Joe Hutchinson - jehutch3@ncsu.edu
* Nikolay Titov - ngtitov@ncsu.edu
* Zhewei Hu (mentor) - zhu6@ncsu.edu

=== Contribution Based Inquiry ===
For any questions or concerns related to the project design, implementation, testing of the feature developed by our team, please contact an engineer who worked closely on the topic of your interest based on the [https://docs.google.com/spreadsheets/d/11uxuI_5gaC-FQDwIOxtCaN3HZ9bfWDJgIjFemtYV5jE/edit?ts=5ca222f3#gid=0 Contribution Assignment Grid] we created to keep track of our progress and contribution. Please carbon copy all team members and our mentor in the email.

== References ==
* Expertiza Web [https://github.com/expertiza/expertiza#expertiza]
* Expertiza on GitHub [https://github.com/expertiza/expertiza]
* Expertiza Wiki [http://wikis.lib.ncsu.edu/index.php/Expertiza]
* Rspec Documentation [https://relishapp.com/rspec]
* Wiki with Previous Work [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Fall_2018/E1873_Specialized_rubics_for_different_topic_types]
* Pull Request of the Previous Work [https://github.com/expertiza/expertiza/pull/1327]

File:Use case diagram.png

2019-04-26T14:35:13Z

Ngtitov: uploaded a new version of "File:Use case diagram.png"

2019-04-08T07:05:38Z

Ngtitov: uploaded a new version of "File:CSC520 Project 4 Diagram.PNG"

File:CSC520 Project 4 Diagram.PNG

2019-04-08T07:01:45Z

Ngtitov:

E1936 Specialized Rubrics

2019-04-08T07:01:20Z