Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:46:18Z

Yzhan114: /* RSpec Testing */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between <code>MetricsController</code> and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method, which returns either the metric average for one reviewer or the metric average for the whole class, depending on whether the <code>reviewer</code> is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which strips html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (<code>tag_prompt_deployment</code> instead of <code>tag_prompts_deployment</code>)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could pass

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter as 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is clicked

app/assets/stylesheets/response.scss
*Added styles for the disabled button and the spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer-tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer-tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added a bar chart in each row of the review report for metrics

==Test Plan==

===RSpec Testing===

Since this project involves changes in many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false to strip that part of logic out of testing.

One new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results onto the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple web service queries.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags is gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:32:27Z

Yzhan114: /* RSpec Testing */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between <code>MetricsController</code> and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method, which returns either the metric average for one reviewer or the metric average for the whole class, depending on whether the <code>reviewer</code> is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which strips html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (<code>tag_prompt_deployment</code> instead of <code>tag_prompts_deployment</code>)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could pass

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter as 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is clicked

app/assets/stylesheets/response.scss
*Added styles for the disabled button and the spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer-tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer-tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added a bar chart in each row of the review report for metrics

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

One new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple web service queries.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags is gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:31:17Z

Yzhan114: /* UI Testing */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between <code>MetricsController</code> and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method, which returns either the metric average for one reviewer or the metric average for the whole class, depending on whether the <code>reviewer</code> is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which strips html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (<code>tag_prompt_deployment</code> instead of <code>tag_prompts_deployment</code>)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could pass

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter as 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is clicked

app/assets/stylesheets/response.scss
*Added styles for the disabled button and the spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer-tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer-tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added a bar chart in each row of the review report for metrics

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple web service queries.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags is gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:30:57Z

Yzhan114: /* UI Testing */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between <code>MetricsController</code> and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method, which returns either the metric average for one reviewer or the metric average for the whole class, depending on whether the <code>reviewer</code> is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which strips html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (<code>tag_prompt_deployment</code> instead of <code>tag_prompts_deployment</code>)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could pass

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter as 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is clicked

app/assets/stylesheets/response.scss
*Added styles for the disabled button and the spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer-tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer-tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added a bar chart in each row of the review report for metrics

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple web service queries.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:28:33Z

Yzhan114: /* Implementation */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between <code>MetricsController</code> and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method, which returns either the metric average for one reviewer or the metric average for the whole class, depending on whether the <code>reviewer</code> is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which strips html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (<code>tag_prompt_deployment</code> instead of <code>tag_prompts_deployment</code>)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could pass

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter as 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is clicked

app/assets/stylesheets/response.scss
*Added styles for the disabled button and the spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer-tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer-tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added a bar chart in each row of the review report for metrics

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:06:15Z

Yzhan114: /* Bug Fixes */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:05:59Z

Yzhan114: /* Answer-Tagging Report Page */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding a column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:04:37Z

Yzhan114: /* Review Report Page */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row representing a student, a metrics chart is added below the volume chart that is already there. Graders get useful information by looking at these two charts combined and can offer more accurate feedback to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor over each bar to view its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:03:02Z

Yzhan114: /* Review Tagging Page */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. The tag in this form is still editable, meaning students can override some of the inferred tags if they wish to.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T20:01:50Z

Yzhan114: /* Metrics Analysis Page */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review-giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting requests multiple times is that the same set of comments are sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add a loader and a message to the bottom of the 'submit' button, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. Students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with the confidence level under the predefined threshold are not rendered in the report, so students do not see uncertain or inaccurate predictions. When students confirm to submit, they return to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T19:50:27Z

Yzhan114: /* Database Design */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a <code>confidence_level</code> column to the existing <code>answer_tags</code> table, which is originally used to store tags assigned by students. One can imagine results from web service to be a stack of tags assigned by the outside tool, with a confidence level indicating how confident the outside tool is to each tag it assigns. Therefore, the <code>answer_tags</code> table will have two types of tags, one from the student, which has the <code>user_id</code> but not the <code>confidence_level</code>, and the other inferred from web service, which has the <code>confidence_level</code> but not the <code>user_id</code>. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T19:43:07Z

Yzhan114: /* Control Flow Diagram */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to train the models (active learning) further

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of the <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It first gets called when students finish and about to submit their reviews on other students' work. Before the system marks their reviews as submitted, we plan that the <code>ReviewMetricsQuery</code> class intercepts these reviews' content and sends them to the Peer Logic web service for predictions. After it receives the predicted results, it caches them to the local Expertiza database and then releases the intercept. Instead of being redirected to the list of reviews, students are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is about this prediction. Whenever a student visits the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to check whether the web service has previously determined the value of the tag and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident about its prediction, it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish between normal tags and gray-out tags and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the algorithm's knowledge.

These cached data would also be used in the instructor's report views, and that's why we need these data to be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes thousands of such reviews. Querying web service results in real-time is impractical concerning the time it consumes. We limit the number of contacts with the web service the least by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T19:35:30Z

Yzhan114: /* Database Design */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

{| class="wikitable"
|+ answer_tags table
|-
! id !! answer_id !! tag_prompt_deployment_id !! user_id !! value !! confidence_level !! created_at !! updated_at
|-
| 1 || 1066685 || 5 || 7513 || 0 || NULL || 2017-11-04 02:32:59 || 2017-11-04 02:32:59
|-
| 353916 || 1430431 || 149 || NULL || -1 || 0.99700 || 2020-11-16 01:47:29 || 2020-11-16 01:47:29
|-
|}

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T19:28:31Z

Yzhan114: /* Notes */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm.' Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-21T19:26:04Z

Yzhan114: /* Background */

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm pre-determine the confidence level of the asked characteristic's presence in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-11T04:59:06Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Core Changes===
app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

===Cache Inferred Tags===

app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/assets/stylesheets/response.scss
*Added styles for disabled button and spinning loader

config/routes.rb
*Added a <code>confirm_submit</code> route

===Show Inferred Tags===

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/three_state_toogle.scss
*Added styles for different forms of tags (gray-out and overridden)

===Show Summary of Inferred Tags===

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

===Bug Fixes===

app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/versions/search.html.erb
*Fixed syntax error

==Test Plan==

===RSpec Testing===

Since this project involves changes to many places of the system, some existing tests needed to be fixed. These include

spec/features/peer_review_spec.rb
*The button "Submit Review" will no longer redirect students to the list of reviews. We made the test to click "Save Review" instead of "Submit Review" so the expected behavior could still be tested.

spec/models/tag_prompt_spec.rb
*This spec file tests functionalities regarding <code>TagPrompt</code>. Some tests break because we incorporated the logic of calling the <code>confident?</code> method to determine the slider's style. We fixed these tests by always letting the <code>confident?</code> method to return false, so as to strip that part of logic out of testing.

Two new spec files are written for the new code:

spec/models/review_metrics_query_spec.rb
*Ensure the <code>cache_ws_results</code> method calls <code>MetricsController</code> with the right parameters and saves the results into the <code>answer_tags</code> table.
*Ensure the <code>inferred_value</code> method interprets the web service results correctly
*Ensure the <code>inferred_confidence</code> method flips the confidence value for predictions that have a negative meaning
*Ensure <code>confident?</code>, <code>confidence()</code>, and <code>has?</code> methods access the right column in the <code>answer_tags</code> table.

=== UI Testing===

Following UI tests were done to ensure the following:

# The 'Submit' button is disabled after the student clicks it to prevent multiple queries to the web service.
# The student gets redirected to the analysis page after the web service request completes.
# The student sees the analysis of their review comments on the analysis page.
# The slider for inferred tags are gray-out.
# The student can override the gray-out tag with a new value, and the slider changes to the overridden style.
# The instructor sees the new bar chart for review metrics.
# The instructor sees the column summarizing inferred tags in the answer-tagging report.

==Reference==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T05:05:33Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|450px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T05:05:11Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|400px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T04:53:13Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|500px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T04:50:21Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|400px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer_tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

File:Answer tagging report page.png

2020-11-02T04:49:29Z

Yzhan114: Yzhan114 uploaded a new version of File:Answer tagging report page.png

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T04:49:20Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|800px|center]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|400px|center]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|800px|center]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|800px|center]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|800px|center]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|800px|center]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer-tagging_report_page.png|800px|center]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

File:Control flow diagram 2.png

2020-11-02T04:45:42Z

Yzhan114: Yzhan114 uploaded a new version of File:Control flow diagram 2.png

File:Control flow diagram 1.png

2020-11-02T04:44:30Z

Yzhan114: Yzhan114 uploaded a new version of File:Control flow diagram 1.png

File:Control flow diagram 1.png

2020-11-02T04:44:23Z

Yzhan114: Yzhan114 uploaded a new version of File:Control flow diagram 1.png

File:Control flow diagram 1.png

2020-11-02T04:44:21Z

Yzhan114: Yzhan114 uploaded a new version of File:Control flow diagram 1.png

File:Answer tagging report page.png

2020-11-02T04:41:58Z

Yzhan114: Yzhan114 uploaded a new version of File:Answer tagging report page.png

File:Control flow diagram 2.png

2020-11-02T04:41:46Z

Yzhan114:

File:Control flow diagram 1.png

2020-11-02T04:41:33Z

Yzhan114:

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T04:40:12Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:Control_flow_diagram_1.png|200px]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:Control_flow_diagram_2.png|200px]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:Loader.png|1000px]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

[[File:Metrics_analysis_page.png|1000px]]

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:Review_tagging_page.png|1000px]]

From the above image, one can see that the slider has been changed into three forms:

[[File:Original_tag.png|100px]]

The original form, meaning it needs input from the user.

[[File:Gray_out_tag.png|100px]]

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:Overridden_tag.png|100px]]

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:Review_report_page.png|1000px]]

In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:Answer-tagging_report_page.png|1000px]]

Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

File:Overridden tag.png

2020-11-02T04:35:07Z

Yzhan114:

File:Review tagging page.png

2020-11-02T04:34:49Z

Yzhan114:

File:Review report page.png

2020-11-02T04:34:27Z

Yzhan114:

File:Loader.png

2020-11-02T04:34:14Z

Yzhan114:

File:Original tag.png

2020-11-02T04:34:03Z

Yzhan114:

File:Metrics analysis page.png

2020-11-02T04:33:50Z

Yzhan114:

File:Answer tagging report page.png

2020-11-02T04:33:24Z

Yzhan114:

File:Gray out tag.png

2020-11-02T04:33:10Z

Yzhan114:

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-02T04:32:27Z

Yzhan114:

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

[[File:control_flow_diagram_1.png]]

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

[[File:control_flow_diagram_2.png]]

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
[[File:metrics_analysis_page.png]]

When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
[[File:review_tagging_page.png]]
From the above image, one can see that the slider has been changed into three forms:

[[File:original_tag.png]]
The original form, meaning it needs input from the user.

[[File:gray_out_tag.png]]
The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

[[File:overridden_tag.png]]
The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
[[File:review_report_page.png]]
In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
[[File:answer-tagging_report_page.png]]
Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==Test Plan==

==References==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging

2020-11-01T20:54:14Z

Yzhan114: Active Learning for Review Tagging draft

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
From the above image, one can see that the slider has been changed into three forms:

The original form, meaning it needs input from the user.

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==References==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020

2020-11-01T20:54:00Z

Yzhan114:

* [[CSC/ECE 517 Summer 2020 - Active Learning for Review Tagging]]

CSC/ECE 517 Fall 2020 - Active Learning for Review Tagging

2020-11-01T20:52:13Z

Yzhan114: Active Learning for Review Tagging Draft

This page provides a description of the Expertiza based independent development project.

__TOC__
==Introduction==

===Background===
The web application Expertiza, used by students in CSC 517 and other courses, allows students to peer-review and give suggestive comments to each other's work. Students will later be asked to voluntarily participate in an extra-credit review-tagging assignment in which they tag comments they received for helpfulness, positive tone, and other characteristics interested by researchers. Currently, students have to tag hundreds of comments they received in order to get full participation credits. Researchers are concerned that this amount of work would cause inattentive participants to submit responses deviating from what they should be, thus corrupting the established model. Therefore, by having the machine-learning algorithm to pre-determine the confidence level of the presence of the asked characteristic in a comment, one can ask students to assign only tags that the algorithm is unsure of, so students can focus on fewer tags with more attention and accuracy.

===Problem Statement===
The goal of this project is to construct a workable infrastructure for active learning, by incorporating machine-learning algorithms in evaluating which tags, by having a manual input, can help the AI learn more effectively. In particular, the following requirements are fulfilled:

*Incorporate metrics analysis into the review-giving process
*Reduce the number of tags students have to assign
*Reveal gathered information to report pages
*Update the web service to include paths to the confidence level of each prediction
*Decide a proper tag certainty threshold that says how certain the ML algorithm must be of a tag value before it will ask the author to tag it manually

===Notes===
This project is simultaneously being held with another project named 'Integrate Suggestion Detection Algorithm'. Whereas that project focuses on forming a central outlet to external web services, this project focuses more on interpreting results gotten back from external web services.

==Design==

===Control Flow Diagram===

Peer Logic is an NSF-funded research project that provides services for educational peer-review systems. It has a set of mature machine learning algorithms and models that compute metrics on the reviews. It would be helpful for Expertiza to integrate these algorithms into the peer-review process. Specifically, we want to

*# Let students see the quality of their review before submission, and
*# Selectively query manual tagging that are used to further train the models (active learning)

In order to integrate these algorithms into the Expertiza system, we have to build a translator-like model, which we named <code>ReviewMetricsQuery</code>, that converts outputs from external sources into a form that our system can understand and use.

Below we show the control flow diagram to help illustrate the usage of <code>ReviewMetricsQuery</code> model.

The <code>ReviewMetricsQuery</code> class is closely tied to the peer-review process. It is first called when students finish and about to submit their reviews on other students' work. Our plan is that, before the system marks their reviews as submitted, the <code>ReviewMetricsQuery</code> class intercepts the content of these reviews and send them to the Peer Logic web service for predictions. After it receives the results, it caches them to the local Expertiza database and then releases the intercept. Students, instead of being redirected to the list of reviews, are presented with an analysis report on the quality of their reviews. They may go back and edit their review comments, or confirm to submit, depending on whether they are satisfied with the results displayed to them.

Every prediction from the web service comes with a confidence level, indicating how confident the algorithm is in this prediction. Whenever a student goes to the review tagging page, before rendering any tags, the system consults the <code>ReviewMetricsQuery</code> class to see whether the value of the tag has previously been determined by the web service and whether its confidence level exceeds the pre-set threshold. If yes, meaning the algorithm is confident of its prediction, then it applies a lightening effect onto the tag to make it less noticeable. Students who do the tagging can easily distinguish the difference between normal tags and gray-out tags, and focus their attention more on normal tags. This is what active learning is about, to query manual inputs only if it adds to the knowledge of the algorithm.

These cached data would also be used in the instructor's report views, and that's the reason why these data must be cached locally. One review consists of about 10 to 20 comments and takes about minutes to process, and a report composes of thousands of such reviews. Querying web service results in real-time is impractical with respect to the time it consumes. We limit the number of contacts with the web service the least, by sending requests only when students decide to submit their reviews. In this way, the predicted values of each tag are up to date with the stored reviews.

===Database Design===
The only change to the database is to add a "confidence_level" column to the existing answer_tags table which is originally used to store tags assigned by students. One can imagine results from web service being a stack of tags assigned by the outside tool, with confidence level indicating how confident the outside tool is to each tag is assigned. The answer_tags table will therefore have two types of tags, one from the student, which has user_id but not confidence_level, and the other inferred from web service, which has confidence_level but not user_id. The system can determine what type of tags they are by checking the presence of values in these two fields.

===UI Design===
Four pages are needed to be modified to reflect the addition of new functionality.

====Metrics Analysis Page====
When students click the 'submit' button on the review giving page, the button will be put on the disabled effect to prevent students from submitting requests multiple times. The consequence of submitting request multiple times is that the same set of comments are being sent to the external web service for processing, wasting resources on both sides. To reassure students that the request has been made, we add to the bottom of the 'submit' button a loader and message, asking them to wait patiently to avoid overload the system.

About half a minute after students click the 'submit' button, they are redirected to a page that shows the analysis of their submitted reviews. On that page, students can see every of their submitted comments along with the analyzed result on each metric. These metrics came from tag prompt deployments set by the instructor in a per questionnaire scope. Predictions with confidence levels under the predefined threshold are not rendered to the report so students do not see predictions that are uncertain or inaccurate. When students confirm to submit, they are returned to the list of reviews to perform other actions.

====Review Tagging Page====
From the above image, one can see that the slider has been changed into three forms:

The original form, meaning it needs input from the user.

The gray-out form, presenting tag inferred by the web service. Tag in this form is editable, meaning students can override some of the inferred tags if they wish.

The overridden form, which is used to represent a tag that originally has a value assigned by the web service but gets overrode by the user.

====Review Report Page====
In each row that represents each student, a metrics chart is added below the volume chart that's already there. Graders get useful information by looking at these two charts combined and are able to offer more accurate review grades to students. Due to the space limitation, each metric name cannot be fully expanded. Grader could hover the cursor above each bar to see its corresponding metric name.

====Answer-Tagging Report Page====
Changes made to this page include changing column names and adding an additional column for the number of inferred tags. Below we explained how each column is calculated.

*% tags applied by author = # tags applied by author / # appliable tags
* # tags applied by author = from # appliable tags, how many are tagged by the author
* # tags not applied by author = # appliable tags - # tags applied by the author
* # appliable tags = # tags whose comment is longer than the length threshold - # tags inferred by ML
* # tags inferred by ML = # tags whose comment is predicted by the machine learning algorithm with high confidence

==Implementation==

===Models===
app/models/answer.rb
*Added <code>de_tag_comments</code> method which remove html tags from the submitted review comment

app/models/answer_tag.rb
*Corrected typo (tag_prompt_deployment instead of tag_prompts_deployment)
*Added validation clause that checks the presence of either the <code>user_id</code> or the <code>confidence_level</code>

app/models/review_metrics_query.rb
*The only model class that is responsible for communications between `MetricsController` and the rest of the Expertiza system where tags are used
*Added <code>average_number_of_qualifying_comments</code> method which returns either the average for one reviewer or the average for the whole class, depending on whether the reviewer is supplied

app/models/tag_prompt.rb
*Added codes to set up the style of the slider (none, gray-out, or overridden) when it is about to be rendered

app/models/tag_prompt_deployment.rb
*Slightly changed how each column in the answer_tagging report is calculated

app/models/vm_user_answer_tagging.rb & app/helpers/report_formatter_helper.rb
*Added one variable that stores the number of tags inferred by ML

===Controllers===
app/controllers/metrics_controller.rb
*Created an empty <code>MetricsController</code> class so tests could be passed

app/controllers/response_controller.rb
*Alternated the redirection so students could be redirected to the analysis page after they click to submit their reviews
*Added <code>confirm_submit</code> method which marks the review in the parameter to 'submitted'

===Views===
app/views/popup/view_review_scores_popup.html.erb
*Fixed syntax error

app/views/reports/_answer_tagging_report.html.erb
*Renamed columns in the answer_tagging report
*Added a new column to the table named "# tags inferred by ML"

app/views/reports/_review_report.html.erb & app/helpers/review_mapping_helper.rb
*Added an additional bar chart in each row of the review report for metrics
*Fixed redirection bug

app/views/response/analysis.html.erb
*Drafted the analysis page, which shows the web service's prediction for each comment on each metric

app/views/response/response.html.erb
*Added codes that disable the "Submit" button after it is being clicked

app/views/versions/search.html.erb
*Fixed syntax error

===Peripheral Changes===
app/assets/javascripts/answer_tags.js
*Controlled the dynamic effect of overriding an inferred tag

app/assets/stylesheets/response.scss
*Styles for disabled button and spinning loader

app/assets/stylesheets/three_state_toogle.scss
*Styles for different forms of tags

config/routes.rb
*Added a <code>confirm_submit</code> route

db/migrate/20200825210644_add_confidence_level_to_answer_tags_table.rb
*Added <code>confidence_level</code> column to the <code>answer_tags</code> table

db/schema.rb
*Updated schema

==References==
Yulin Zhang (yzhan114@ncsu.edu)

CSC/ECE 517 Summer 2020

2020-11-01T20:35:45Z

Yzhan114: Created page with "* CSC/ECE 517 Fall 2020 - Active Learning for Review Tagging"

* [[CSC/ECE 517 Fall 2020 - Active Learning for Review Tagging]]

MainPage

2020-11-01T20:34:23Z

Yzhan114: /* Expertiza */

==Expertiza==
* [[Expertiza documentation]]

* [[CSC/ECE 517 Summer 2008]]
* [[CSC/ECE 517 Fall 2010]]
* [[CSC/ECE 517 Fall 2011]]
* [[CSC/ECE 517 Fall 2012]]
* [[CSC/ECE 517 Fall 2013]]
* [[CSC/ECE 517 Fall 2014]]
* [[CSC/ECE 517 Fall 2015]]
* [[CSC/ECE 517 Fall 2016]]
* [[CSC/ECE 517 Spring 2014]]
* [[CSC/ECE 517 Spring 2015]]
* [[CSC/ECE 517 Spring 2016]]
* [[CSC/ECE 517 Spring 2017]]
* [[CSC/ECE 517 Fall 2017]]
* [[CSC/Independent Study Spring 2018]]
* [[CSC/ECE 517 Spring 2018]]
* [[CSC/ECE 517 Fall 2018]]
* [[CSC/ECE 517 Spring 2019]]
* [[CSC/ECE 517 Fall 2019]]
* [[CSC/ECE 517 Spring 2020]]
* [[CSC/ECE 517 Summer 2020]]
* [[CSC/ECE 517 Fall 2020]]
* [[CSC 456 Spring 2011|CSC 456 Spring 2012]]
* [[ECE 633]]
* [[KCU]]
* [[Progress reports]]
* [[ReactJs Frontend]]
* [[Front-End/Back-End]]

==Application Behavior==
* [[Grading]]

==Metaprogramming==
* [[CSC/ECE_517_Spring_2013/ch1b_1k_hf|Lecture on Metaprogramming]]

==Development==

''Expertiza now has a Java dependency, so the machine you are using to develop Expertiza on should have the JVM installed.''

* [[Setting Up a Development Machine]]
* [[Creating a Linux Development Environment for Expertiza - Installation Guide]]
* [[Using git and github for projects]]
* [[Using heroku to deploy your projects]]
* [[How to Begin a Project from the Current Expertiza Repository]]
* [[Git]]
* [[How to Change a User's Password on a Development Machine]]
* [[Debugging Rails]]
* [http://rajanalwan.com/ui_guidelines/ Design Template]

==Production==
* [[Deploying to Production]]
* [[Downloading Production Data]]
* [[Accessing the Production Server]]

==Testing==
* [[Using Cucumber with Expertiza]]
* [[Rails Testing Overview]]
* [[Expertiza Continuous Integration]]

==References==
* [[Object-Oriented Design and Programming]]

CSC/ECE 517 Spring 2020 E2016 Revision planning tool

2020-04-14T03:14:11Z

Yzhan114: /* Revision planning page */

=Introduction=

==About our team==

Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

==What is Revision Planning?==

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

==Previous Implementation==

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

==Problem Statement==

For this project, we identified 4 major work items that together fulfill the stated requirements.

'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''

We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> will have either questions with no <code>team_id</code> or with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.

'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

'''Every new question must be linked to the second-round questionnaire.'''

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

'''Every new question must be linked to the author’s submission (this will probably involve a DB migration)'''

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

=Design=

==Control Flow Diagram==

The below image shows the control flow of the revision planning functionality.

[[File:E2016_control_flow_diagram.png]]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student(reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student(reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.

==UI Design==

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

===Enabling revision planning===

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

[[File:E2016_enabling_revision_planning.png]]

===Link to the revision planning page===

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

[[File:E2016_link_to_the_revision_planning_page.png]]

===Revision planning page===

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers during the review. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

[[File:E2016_revision_planning_page.png]]

===Review page===

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

[[File:E2016_review_page.png]]

[[File:E2016_review_page_2.png]]

===Feedback report===

Teaching staff and students have different windows to access the feedback report.
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

====TA/Instructor====

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

[[File:E2016_feedback_report_instructor.png]]

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

====Student====

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

[[File:E2016_feedback_report_student.png]]

==Database Design==

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

[[File:E2016_database_design.png]]

In the Assignment table, the column ''is_revision_planning_enabled?'' will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish where each question belongs to. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be the question created by the team associated with the team_id. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

==Code Modifications==

===Controllers===

app/controllers/questionnaires_controller.rb
*edit_revision_plan: a new method added to the <code>QuestionnaireController</code> that prepares view template and supplies revision planning questions that belong only to the current team
*update_revision_plan: a new method added to the <code>QuestionnaireController</code> that saves revision plan questions under the current round rubric
*Require some refactoring to share some existing functionalities with the two new methods we described above
app/controllers/response_controller.rb
*Replace all the occurrences of <code>@questionnaire.questions</code> with <code>@questionnaire.questions(@map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
app/controllers/grades_controller.rb
*Call the retrieve_questions method every time with an extra parameter “<code>@team_id</code>”.

===Models ===

app/models/question.rb
*Form association relationship with <code>Team</code> via <code>team_id</code>
**e.g. <code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
*questions: change the method signature to <code>questions(team_id=nil)</code> which uses nil as the parameter <code>team_id</code>’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.
app/models/assignment_team.rb
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**e.g. <code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
app/models/response.rb
*Replace all the occurrences of <code>questionnaire.questions</code> with <code>questionnaire.questions(self.response_map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.

===Views===

app/views/questionnaires/edit_revision_plan.html.erb
*It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
app/views/questionnaires/_questions.html.erb
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' view file described above.
app/views/student_task/view.html.erb
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
app/views/assignments/edit/_review_strategy.html.erb
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

===Helpers===

app/helpers/grades_helper.rb
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with this <code>team_id</code> and retrieve a proper question set for each team.

===Schema===

*Assignment table: add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
*Question table: add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

=Testing=

==RSpec Test Plan==

'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb
**Describe ‘#edit_revision_plan’
***Context ‘when params[:id] is valid’
***Context ‘when params[:id] is not valid'
**Describe ‘#update_revision_plan’
***Context 'when params[:add_new_questions] is not nil'
***Context 'when params[:view_advice] is not nil'
***Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
*spec/controllers/grades_controller_spec.rb
**Describe ‘#view’
***Context ‘When the revision plan is included in one round’
**Describe ‘#view_my_scores’
***Context ‘When the revision plan is included in one round’
*spec/controllers/response_controller_spec.rb
**Fix all failed tests to accommodate modified codes.
**Add more tests regarding revision planning.

'''Models'''
*spec/models/question_spec.rb
**Describe ‘#questions’
***Context ‘when team_id is supplied’
***Context ‘when team_id is not supplied’

==UI Testing Instructions (For Reviewers)==

'''Setup'''

Login information
*Visit xxx [expertiza deployment link]
User name: instructor6/student8030/student8031
Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.

'''Functionalities'''

Edit a Revision Plan
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “'''Save review questionnaire'''” button, and the revision plan should be saved.
Test retrieval of revision plan questions for a specific team
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool

2020-04-14T03:13:00Z

Yzhan114: /* Revision planning page */

=Introduction=

==About our team==

Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

==What is Revision Planning?==

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

==Previous Implementation==

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

==Problem Statement==

For this project, we identified 4 major work items that together fulfill the stated requirements.

'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''

We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> will have either questions with no <code>team_id</code> or with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.

'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

'''Every new question must be linked to the second-round questionnaire.'''

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

'''Every new question must be linked to the author’s submission (this will probably involve a DB migration)'''

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

=Design=

==Control Flow Diagram==

The below image shows the control flow of the revision planning functionality.

[[File:E2016_control_flow_diagram.png]]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student(reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student(reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.

==UI Design==

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

===Enabling revision planning===

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

[[File:E2016_enabling_revision_planning.png]]

===Link to the revision planning page===

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

[[File:E2016_link_to_the_revision_planning_page.png]]

===Revision planning page===

The revision plan is just like other questionnaires in that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far. Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

[[File:E2016_revision_planning_page.png]]

===Review page===

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

[[File:E2016_review_page.png]]

[[File:E2016_review_page_2.png]]

===Feedback report===

Teaching staff and students have different windows to access the feedback report.
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

====TA/Instructor====

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

[[File:E2016_feedback_report_instructor.png]]

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

====Student====

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

[[File:E2016_feedback_report_student.png]]

==Database Design==

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

[[File:E2016_database_design.png]]

In the Assignment table, the column ''is_revision_planning_enabled?'' will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish where each question belongs to. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be the question created by the team associated with the team_id. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

==Code Modifications==

===Controllers===

app/controllers/questionnaires_controller.rb
*edit_revision_plan: a new method added to the <code>QuestionnaireController</code> that prepares view template and supplies revision planning questions that belong only to the current team
*update_revision_plan: a new method added to the <code>QuestionnaireController</code> that saves revision plan questions under the current round rubric
*Require some refactoring to share some existing functionalities with the two new methods we described above
app/controllers/response_controller.rb
*Replace all the occurrences of <code>@questionnaire.questions</code> with <code>@questionnaire.questions(@map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
app/controllers/grades_controller.rb
*Call the retrieve_questions method every time with an extra parameter “<code>@team_id</code>”.

===Models ===

app/models/question.rb
*Form association relationship with <code>Team</code> via <code>team_id</code>
**e.g. <code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
*questions: change the method signature to <code>questions(team_id=nil)</code> which uses nil as the parameter <code>team_id</code>’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.
app/models/assignment_team.rb
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**e.g. <code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
app/models/response.rb
*Replace all the occurrences of <code>questionnaire.questions</code> with <code>questionnaire.questions(self.response_map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.

===Views===

app/views/questionnaires/edit_revision_plan.html.erb
*It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
app/views/questionnaires/_questions.html.erb
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' view file described above.
app/views/student_task/view.html.erb
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
app/views/assignments/edit/_review_strategy.html.erb
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

===Helpers===

app/helpers/grades_helper.rb
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with this <code>team_id</code> and retrieve a proper question set for each team.

===Schema===

*Assignment table: add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
*Question table: add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

=Testing=

==RSpec Test Plan==

'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb
**Describe ‘#edit_revision_plan’
***Context ‘when params[:id] is valid’
***Context ‘when params[:id] is not valid'
**Describe ‘#update_revision_plan’
***Context 'when params[:add_new_questions] is not nil'
***Context 'when params[:view_advice] is not nil'
***Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
*spec/controllers/grades_controller_spec.rb
**Describe ‘#view’
***Context ‘When the revision plan is included in one round’
**Describe ‘#view_my_scores’
***Context ‘When the revision plan is included in one round’
*spec/controllers/response_controller_spec.rb
**Fix all failed tests to accommodate modified codes.
**Add more tests regarding revision planning.

'''Models'''
*spec/models/question_spec.rb
**Describe ‘#questions’
***Context ‘when team_id is supplied’
***Context ‘when team_id is not supplied’

==UI Testing Instructions (For Reviewers)==

'''Setup'''

Login information
*Visit xxx [expertiza deployment link]
User name: instructor6/student8030/student8031
Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.

'''Functionalities'''

Edit a Revision Plan
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “'''Save review questionnaire'''” button, and the revision plan should be saved.
Test retrieval of revision plan questions for a specific team
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool

2020-04-14T03:10:58Z

Yzhan114: /* Database Design */

=Introduction=

==About our team==

Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

==What is Revision Planning?==

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

==Previous Implementation==

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

==Problem Statement==

For this project, we identified 4 major work items that together fulfill the stated requirements.

'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''

We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> will have either questions with no <code>team_id</code> or with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.

'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

'''Every new question must be linked to the second-round questionnaire.'''

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

'''Every new question must be linked to the author’s submission (this will probably involve a DB migration)'''

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

=Design=

==Control Flow Diagram==

The below image shows the control flow of the revision planning functionality.

[[File:E2016_control_flow_diagram.png]]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student(reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student(reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.

==UI Design==

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

===Enabling revision planning===

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

[[File:E2016_enabling_revision_planning.png]]

===Link to the revision planning page===

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

[[File:E2016_link_to_the_revision_planning_page.png]]

===Revision planning page===

The revision plan is just like other questionnaires that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far.

Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

[[File:E2016_revision_planning_page.png]]

===Review page===

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

[[File:E2016_review_page.png]]

[[File:E2016_review_page_2.png]]

===Feedback report===

Teaching staff and students have different windows to access the feedback report.
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

====TA/Instructor====

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

[[File:E2016_feedback_report_instructor.png]]

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

====Student====

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

[[File:E2016_feedback_report_student.png]]

==Database Design==

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

[[File:E2016_database_design.png]]

In the Assignment table, the column ''is_revision_planning_enabled?'' will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish where each question belongs to. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be the question created by the team associated with the team_id. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

==Code Modifications==

===Controllers===

app/controllers/questionnaires_controller.rb
*edit_revision_plan: a new method added to the <code>QuestionnaireController</code> that prepares view template and supplies revision planning questions that belong only to the current team
*update_revision_plan: a new method added to the <code>QuestionnaireController</code> that saves revision plan questions under the current round rubric
*Require some refactoring to share some existing functionalities with the two new methods we described above
app/controllers/response_controller.rb
*Replace all the occurrences of <code>@questionnaire.questions</code> with <code>@questionnaire.questions(@map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
app/controllers/grades_controller.rb
*Call the retrieve_questions method every time with an extra parameter “<code>@team_id</code>”.

===Models ===

app/models/question.rb
*Form association relationship with <code>Team</code> via <code>team_id</code>
**e.g. <code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
*questions: change the method signature to <code>questions(team_id=nil)</code> which uses nil as the parameter <code>team_id</code>’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.
app/models/assignment_team.rb
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**e.g. <code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
app/models/response.rb
*Replace all the occurrences of <code>questionnaire.questions</code> with <code>questionnaire.questions(self.response_map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.

===Views===

app/views/questionnaires/edit_revision_plan.html.erb
*It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
app/views/questionnaires/_questions.html.erb
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' view file described above.
app/views/student_task/view.html.erb
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
app/views/assignments/edit/_review_strategy.html.erb
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

===Helpers===

app/helpers/grades_helper.rb
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with this <code>team_id</code> and retrieve a proper question set for each team.

===Schema===

*Assignment table: add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
*Question table: add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

=Testing=

==RSpec Test Plan==

'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb
**Describe ‘#edit_revision_plan’
***Context ‘when params[:id] is valid’
***Context ‘when params[:id] is not valid'
**Describe ‘#update_revision_plan’
***Context 'when params[:add_new_questions] is not nil'
***Context 'when params[:view_advice] is not nil'
***Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
*spec/controllers/grades_controller_spec.rb
**Describe ‘#view’
***Context ‘When the revision plan is included in one round’
**Describe ‘#view_my_scores’
***Context ‘When the revision plan is included in one round’
*spec/controllers/response_controller_spec.rb
**Fix all failed tests to accommodate modified codes.
**Add more tests regarding revision planning.

'''Models'''
*spec/models/question_spec.rb
**Describe ‘#questions’
***Context ‘when team_id is supplied’
***Context ‘when team_id is not supplied’

==UI Testing Instructions (For Reviewers)==

'''Setup'''

Login information
*Visit xxx [expertiza deployment link]
User name: instructor6/student8030/student8031
Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.

'''Functionalities'''

Edit a Revision Plan
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “'''Save review questionnaire'''” button, and the revision plan should be saved.
Test retrieval of revision plan questions for a specific team
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool

2020-04-14T03:08:52Z

Yzhan114: /* RSpec Test Plan */

=Introduction=

==About our team==

Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

==What is Revision Planning?==

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

==Previous Implementation==

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

==Problem Statement==

For this project, we identified 4 major work items that together fulfill the stated requirements.

'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''

We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> will have either questions with no <code>team_id</code> or with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.

'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

'''Every new question must be linked to the second-round questionnaire.'''

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

'''Every new question must be linked to the author’s submission (this will probably involve a DB migration)'''

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

=Design=

==Control Flow Diagram==

The below image shows the control flow of the revision planning functionality.

[[File:E2016_control_flow_diagram.png]]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student(reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student(reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.

==UI Design==

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

===Enabling revision planning===

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

[[File:E2016_enabling_revision_planning.png]]

===Link to the revision planning page===

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

[[File:E2016_link_to_the_revision_planning_page.png]]

===Revision planning page===

The revision plan is just like other questionnaires that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far.

Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

[[File:E2016_revision_planning_page.png]]

===Review page===

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

[[File:E2016_review_page.png]]

[[File:E2016_review_page_2.png]]

===Feedback report===

Teaching staff and students have different windows to access the feedback report.
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

====TA/Instructor====

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

[[File:E2016_feedback_report_instructor.png]]

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

====Student====

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

[[File:E2016_feedback_report_student.png]]

==Database Design==

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

[[File:E2016_database_design.png]]

In the Assignment table, the column ''is_revision_planning_enabled?'' will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish whether the question is on the original rubric or is added by students as part of their revision plan. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be regarded as the question created by the team that the team_id refers to. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

==Code Modifications==

===Controllers===

app/controllers/questionnaires_controller.rb
*edit_revision_plan: a new method added to the <code>QuestionnaireController</code> that prepares view template and supplies revision planning questions that belong only to the current team
*update_revision_plan: a new method added to the <code>QuestionnaireController</code> that saves revision plan questions under the current round rubric
*Require some refactoring to share some existing functionalities with the two new methods we described above
app/controllers/response_controller.rb
*Replace all the occurrences of <code>@questionnaire.questions</code> with <code>@questionnaire.questions(@map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
app/controllers/grades_controller.rb
*Call the retrieve_questions method every time with an extra parameter “<code>@team_id</code>”.

===Models ===

app/models/question.rb
*Form association relationship with <code>Team</code> via <code>team_id</code>
**e.g. <code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
*questions: change the method signature to <code>questions(team_id=nil)</code> which uses nil as the parameter <code>team_id</code>’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.
app/models/assignment_team.rb
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**e.g. <code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
app/models/response.rb
*Replace all the occurrences of <code>questionnaire.questions</code> with <code>questionnaire.questions(self.response_map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.

===Views===

app/views/questionnaires/edit_revision_plan.html.erb
*It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
app/views/questionnaires/_questions.html.erb
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' view file described above.
app/views/student_task/view.html.erb
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
app/views/assignments/edit/_review_strategy.html.erb
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

===Helpers===

app/helpers/grades_helper.rb
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with this <code>team_id</code> and retrieve a proper question set for each team.

===Schema===

*Assignment table: add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
*Question table: add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

=Testing=

==RSpec Test Plan==

'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb
**Describe ‘#edit_revision_plan’
***Context ‘when params[:id] is valid’
***Context ‘when params[:id] is not valid'
**Describe ‘#update_revision_plan’
***Context 'when params[:add_new_questions] is not nil'
***Context 'when params[:view_advice] is not nil'
***Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
*spec/controllers/grades_controller_spec.rb
**Describe ‘#view’
***Context ‘When the revision plan is included in one round’
**Describe ‘#view_my_scores’
***Context ‘When the revision plan is included in one round’
*spec/controllers/response_controller_spec.rb
**Fix all failed tests to accommodate modified codes.
**Add more tests regarding revision planning.

'''Models'''
*spec/models/question_spec.rb
**Describe ‘#questions’
***Context ‘when team_id is supplied’
***Context ‘when team_id is not supplied’

==UI Testing Instructions (For Reviewers)==

'''Setup'''

Login information
*Visit xxx [expertiza deployment link]
User name: instructor6/student8030/student8031
Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.

'''Functionalities'''

Edit a Revision Plan
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “'''Save review questionnaire'''” button, and the revision plan should be saved.
Test retrieval of revision plan questions for a specific team
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool

2020-04-14T03:00:19Z

Yzhan114: /* UI Testing Instructions (For Reviewers) */

=Introduction=

==About our team==

Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

==What is Revision Planning?==

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

==Previous Implementation==

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

==Problem Statement==

For this project, we identified 4 major work items that together fulfill the stated requirements.

'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''

We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> will have either questions with no <code>team_id</code> or with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.

'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

'''Every new question must be linked to the second-round questionnaire.'''

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

'''Every new question must be linked to the author’s submission (this will probably involve a DB migration)'''

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

=Design=

==Control Flow Diagram==

The below image shows the control flow of the revision planning functionality.

[[File:E2016_control_flow_diagram.png]]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student(reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student(reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.

==UI Design==

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

===Enabling revision planning===

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

[[File:E2016_enabling_revision_planning.png]]

===Link to the revision planning page===

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

[[File:E2016_link_to_the_revision_planning_page.png]]

===Revision planning page===

The revision plan is just like other questionnaires that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far.

Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

[[File:E2016_revision_planning_page.png]]

===Review page===

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

[[File:E2016_review_page.png]]

[[File:E2016_review_page_2.png]]

===Feedback report===

Teaching staff and students have different windows to access the feedback report.
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

====TA/Instructor====

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

[[File:E2016_feedback_report_instructor.png]]

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

====Student====

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

[[File:E2016_feedback_report_student.png]]

==Database Design==

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

[[File:E2016_database_design.png]]

In the Assignment table, the column ''is_revision_planning_enabled?'' will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish whether the question is on the original rubric or is added by students as part of their revision plan. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be regarded as the question created by the team that the team_id refers to. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

==Code Modifications==

===Controllers===

app/controllers/questionnaires_controller.rb
*edit_revision_plan: a new method added to the <code>QuestionnaireController</code> that prepares view template and supplies revision planning questions that belong only to the current team
*update_revision_plan: a new method added to the <code>QuestionnaireController</code> that saves revision plan questions under the current round rubric
*Require some refactoring to share some existing functionalities with the two new methods we described above
app/controllers/response_controller.rb
*Replace all the occurrences of <code>@questionnaire.questions</code> with <code>@questionnaire.questions(@map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
app/controllers/grades_controller.rb
*Call the retrieve_questions method every time with an extra parameter “<code>@team_id</code>”.

===Models ===

app/models/question.rb
*Form association relationship with <code>Team</code> via <code>team_id</code>
**e.g. <code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
*questions: change the method signature to <code>questions(team_id=nil)</code> which uses nil as the parameter <code>team_id</code>’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.
app/models/assignment_team.rb
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**e.g. <code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
app/models/response.rb
*Replace all the occurrences of <code>questionnaire.questions</code> with <code>questionnaire.questions(self.response_map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.

===Views===

app/views/questionnaires/edit_revision_plan.html.erb
*It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
app/views/questionnaires/_questions.html.erb
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' view file described above.
app/views/student_task/view.html.erb
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
app/views/assignments/edit/_review_strategy.html.erb
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

===Helpers===

app/helpers/grades_helper.rb
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with this <code>team_id</code> and retrieve a proper question set for each team.

===Schema===

*Assignment table: add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
*Question table: add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

=Testing=

==RSpec Test Plan==

'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb
**Describe ‘#edit_revision_plan’
***Context ‘when params[:id] is valid’
***Context ‘when params[:id] is not valid'
**Describe ‘#update_revision_plan’
***Context 'when params[:add_new_questions] is not nil'
***Context 'when params[:view_advice] is not nil'
***Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
*spec/controllers/grades_controller_spec.rb
**Describe ‘#view’
***Context ‘When the revision plan is included in one round’
**Describe ‘#view_my_scores’
***Context ‘When the revision plan is included in one round’
*spec/controllers/response_controller_spec.rb
**Describe

'''Models'''
*spec/models/question_spec.rb
**Describe ‘#questions’
***Context ‘when team_id is supplied’
***Context ‘when team_id is not supplied’

==UI Testing Instructions (For Reviewers)==

'''Setup'''

Login information
*Visit xxx [expertiza deployment link]
User name: instructor6/student8030/student8031
Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.

'''Functionalities'''

Edit a Revision Plan
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “'''Save review questionnaire'''” button, and the revision plan should be saved.
Test retrieval of revision plan questions for a specific team
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.

CSC/ECE 517 Spring 2020 E2016 Revision planning tool

2020-04-14T02:59:46Z

Yzhan114: /* RSpec Test Plan */

=Introduction=

==About our team==

Team members:
*Tianji Gao (tgao5@ncsu.edu)
*Guoyi Wang (gwang25@ncsu.edu)
*Yulin Zhang (yzhan114@ncsu.edu)
*Boxuan Zhong (bzhong2@ncsu.edu)

Project mentor: Edward Gehringer (efg@ncsu.edu)

==What is Revision Planning?==

In the first round of the Expertiza reviews, reviewers are asked to give authors some guidance on how to improve their work. Then in the second round, reviewers rate how well authors have followed their suggestions. Revision planning is a mechanism used to carry the interaction one step further by having authors to supply a revision plan based on the previous round reviews. That is, the authors would derive their plan for code improvement from the previous round reviews and reviewers would later assess how well they did it.

Revision planning is helpful because it makes the author think about what's necessary to improve the work before putting forth the effort to improve it. This leads to a more reflective work process and is likely to produce a better-finished product. When reviewers have an opportunity to give feedback to the author, they too will learn what a good revision plan looks like.

According to the given instructions, a revision plan consists of a description of the plan, followed by any number of questions that would later be appended to the future review questionnaire. The revision plan is per AssignmentTeam-based, which means the authors’ questions would only be used to evaluate their submission and not anyone else. By adding the functionality of revision planning, it helps researchers study the effect of the reviewer’s suggestions on the code improvement.

==Previous Implementation==

This functionality has previously been done by a team of students from the Fall semester of 2018. Their implementation was merged into the master branch but was reverted due to the following design concerns:
*The relationship between `Questionnaire` and `SubmissionRecord` is unclear.
*Uses a lot of '''special-purpose''' code when existing codes may fulfill the same job.
*Revision planning cannot be enabled or disabled for an assignment.
*Numeric labelings for the revision plan questions begin from 1 again, instead of continuing after the original rubric questions.
*Codebase contains commented codes that are no longer wanted.
Check out the wiki page and the pull request on GitHub if you would like to learn more about the previous implementation of this project.
*http://wiki.expertiza.ncsu.edu/index.php/E1875_Revision_Planning_Tool
*https://github.com/expertiza/expertiza/pull/1302
Please note that unlike the other teams we have reviewed, this project is a complete redo rather than modifications built upon the previous team’s codes because our approach to this problem would be different than theirs. Therefore, we will not mention the previous implementation in the later content.

==Problem Statement==

For this project, we identified 4 major work items that together fulfill the stated requirements.

'''Sort out the relationship among classes and introduce the new abstraction of the revision plan to the system in a way that it doesn’t interfere with the majority of codes'''

We decided to relate each <code>Question</code> object with team_id. A <code>ReviewQuestionnaire</code> will have either questions with no <code>team_id</code> or with a <code>team_id</code>. A question with no team_id indicates that it does not belong to any assignment teams so it is a question set up by the instructor. A question with a team_id, in contrast, indicates that it belongs to a particular team so it is a revision plan question. Both types of questions will be saved under the same questionnaire used for a given round. In this way, we can maximize the usage of existing codes and the only major change should be contained within the <code>Question</code> class.

'''Modify the existing views and controllers to accommodate the new functionality which includes'''
*Allowing teaching staff to enable/disable revision planning for an assignment.
*Allowing team members to create/edit their revision plan during each submission period after the first round.
*Allowing both rubric questions and revision plan questions to appear on the same page and be serialized correctly.
*Allowing feedback on the revision plan only to be viewed by the team that creates the plan and that team's instructor.

This will involve some minor changes such as appending some method signatures with an optional trailing parameter, adding interactive elements to the views, and slightly adjusting the structure of certain view templates.

In addition, we planned to:
*Provide an adequate amount of tests to improve code coverage.
*Do necessary refactoring and resolve any CodeClimate issues.

After communicated with our mentor Dr. Gehringer, we have been clarified with the following two problem statements.

'''Every new question must be linked to the second-round questionnaire.'''

This means both questions from the team’s revision plan and questions from the review rubric should be displayed together in the frontend. Since we decided to add revision plan questions to the review rubric of the round, we automatically linked every new question to the questionnaire of that round.

'''Every new question must be linked to the author’s submission (this will probably involve a DB migration)'''

By saying every new question must be linked to the author’s submission, it means that there should be some relationships between the team and the team’s revision plan questions presented in the database. We addressed this problem by associating them with a team_id field. See Database Design section for more details.

=Design=

==Control Flow Diagram==

The below image shows the control flow of the revision planning functionality.

[[File:E2016_control_flow_diagram.png]]

The below image shows the control flow of the revision planning functionality. It involves 3 types of actors, student(reviewee), student(reviewer) and instructor/TA who manages the assignment and review processes. To understand each actor’s responsibility, trace each colored line that arose from each actor in the direction specified by the arrows. The diamond shape represents a decision or precondition, that is, only after the condition meets can the next action proceeds.

Summary of actions
*A TA/Instructor can
*#Enable revision planning
*#Impersonate students to perform their responsibility
*#View feedback report of all teams
*A student(reviewee) can
*#Make revision during the second round submission period, which includes reading first-round feedback and adding revision plan questions according to that feedback.
*#View feedback report of the team it belongs to
*A student(reviewer) can
*#Give feedback on the team’s revised work by answering each question (including the team's revision plan questions) appeared on the review page.
*#View the feedback it wrote to the team.

==UI Design==

A revision plan should be similar to other review questionnaires. Since functionalities on the review questionnaire have been maturely implemented, we expected to make the least amount of interface changes by utilizing the existing view templates whenever possible. The subsections listed the changes we planned to make.

===Enabling revision planning===

Implementation of enabling/disabling revision planning for each assignment can be rather straightforward. We looked to add an additional checkbox under the "Review strategy" tab of the assignment’s edit page. This checkbox is labeled as "Enable Revision Planning?" to indicate whether the instructor wants to include this functionality in the newly-created assignment. It is most reasonable to place the checkbox here because it is review related and other similar functionalities like Self Reviews are also implemented in this manner.

[[File:E2016_enabling_revision_planning.png]]

===Link to the revision planning page===

If the instructor decided to include revision planning in this assignment, then the link to “Revision Planning” would appear on the student’s assignment page but would stay disabled during the first round. After that, It would become clickable during every submission period and greyed again during every review period. By clicking it, students would be redirected to a whole new page explained under the ‘Revision planning page’ subsection.

[[File:E2016_link_to_the_revision_planning_page.png]]

===Revision planning page===

The revision plan is just like other questionnaires that it contains a set of questions for reviewers to answer. The only difference is that the revision plan comes with an additional description to help reviewers understand what changes have been made so far.

Therefore, it should make use of most existing view templates and controller codes with minimized changes. As the image is shown, the only modification made from the existing questionnaire creation template would be to include a link that redirects students to the submission page, where the uploading of the revision plan will be handled by the existing implementation. The advantage to upload an external link rather than typing everything to the textbox element is that the description can be well-formatted if it displays outside the form and not causing a distraction effect for reviewers. We also decided to leave out (or hide) the place where instructors set the configuration stuff like the range of scores and the questionnaire's visibility. These configurations should use default values defined in the system rather than having students come up with their own.

[[File:E2016_revision_planning_page.png]]

===Review page===

The format of the review page remains almost exactly the same. To distinguish between rubric questions set up by the instructor and the revision plan questions created by the team under review, all the revision plan questions are placed after the rubric questions, split by an enlarged “Revision Planning” subheader.

[[File:E2016_review_page.png]]

[[File:E2016_review_page_2.png]]

===Feedback report===

Teaching staff and students have different windows to access the feedback report.
*'''Teaching staff''': Manage->Assignments->Edit Assignment->Other stuff->View scores
*'''Students''': Assignments->View Assignment->Alternative View
In addition, either instructor and TA can impersonate students to access the feedback report from their views. We would like to consider both cases and illustrate each of them separately.

====TA/Instructor====

Scores for the second round review rubric and the author’s revision plan questions will be displayed on the same table and are serialized correctly. See the figure below for an example. Let say the second round rubric has only 5 questions, the remaining questions (6-10) will be revision planning questions written by a particular team.

[[File:E2016_feedback_report_instructor.png]]

For reviewer: if you reviewed our first draft design, you should notice that we originally chose to place revision plan scores on a distinct table. After our mentor clarified to us, we realized that there can be possibly more than 2 rounds of submission and review periods for a given assignment. Therefore, scores for revision planning questions can vary round by round. Therefore, our previous solution will not work since it confuses the user of which round the revision plan scores refer to.

====Student====

The revision planning section will be added to the students’ view as shown in the snapshot below. It displays in the same order as how the review page does. A “Revision Planning” subheader is also used here to indicate the starting of the revision planning section.

[[File:E2016_feedback_report_student.png]]

==Database Design==

Here we present the diagram of our database design. As the yellow borders show, we only plan to modify the structure of the Question table and the Assignment table.

[[File:E2016_database_design.png]]

In the Assignment table, the column ''is_revision_planning_enabled?'' will be needed to indicate whether the instructor would like to incorporate the revision planning feature.

Additionally, we add <code>team_id</code> to each <code>Question</code> object to distinguish whether the question is on the original rubric or is added by students as part of their revision plan. A <code>Question</code> object with an empty <code>team_id</code> value will be the question under the original rubric, while the object with a non-empty team_id field will be regarded as the question created by the team that the team_id refers to. That is, instead of creating a whole new <code>RevisionPlanQuestionnaire</code> class, we decided to dump all the revision plan questions that are created in a given round to the rubric that is used for that round. In this way, we minimize the change to the system to make the original rubric questions and the revision planning questions retrieved together more easily.

==Code Modifications==

===Controllers===

app/controllers/questionnaires_controller.rb
*edit_revision_plan: a new method added to the <code>QuestionnaireController</code> that prepares view template and supplies revision planning questions that belong only to the current team
*update_revision_plan: a new method added to the <code>QuestionnaireController</code> that saves revision plan questions under the current round rubric
*Require some refactoring to share some existing functionalities with the two new methods we described above
app/controllers/response_controller.rb
*Replace all the occurrences of <code>@questionnaire.questions</code> with <code>@questionnaire.questions(@map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.
app/controllers/grades_controller.rb
*Call the retrieve_questions method every time with an extra parameter “<code>@team_id</code>”.

===Models ===

app/models/question.rb
*Form association relationship with <code>Team</code> via <code>team_id</code>
**e.g. <code>belongs_to :team, class_name: ‘AssignmentTeam’, foreign_key: ‘team_id’</code>
*questions: change the method signature to <code>questions(team_id=nil)</code> which uses nil as the parameter <code>team_id</code>’s default value, so users can choose to supply a team_id argument or not. When team_id is supplied, it returns both questions with no team_id as well as questions that have this team_id. In addition, it will add to the return list an unsaved “Revision Planning” QuestionnaireHeader so the list can be displayed nicely on the browser with each section separated.
app/models/assignment_team.rb
*Form aggregation relationship with <code>Question</code> via <code>team_id</code>
**e.g. <code>has_many :revision_plan_questions, class_name: ‘Question’, foreign_key: ‘team_id’</code>
app/models/response.rb
*Replace all the occurrences of <code>questionnaire.questions</code> with <code>questionnaire.questions(self.response_map.reviewee_id)</code> so it not only gets questions from the original rubric but also from the revision plan proposed by the team with the corresponding <code>reviewee_id</code>.

===Views===

app/views/questionnaires/edit_revision_plan.html.erb
*It is a newly-added view file for students to create and edit their revision plan. It will utilize some existing codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to reduce code redundancy.
app/views/questionnaires/_questions.html.erb
*Extract codes from the ''app/views/questionnaires/_questionnaire.html.erb'' view file to make it a standalone partial template that will later be loaded by the ''app/views/questionnaires/edit_revision_plan.html.erb'' view file described above.
app/views/student_task/view.html.erb
*Add a “Revision Planning” link for students to edit their revision plan. The link will lead students to the “Edit Revision Plan” page. If the revision planning feature is enabled, this link will appear disabled at first and only become clickable during each submission period after round 1.
app/views/assignments/edit/_review_strategy.html.erb
*Add “Enable Revision Planning?” checkbox for each assignment because not every assignment needs this feature. TA/instructor have the option to include this feature more flexibly. The “Revision Planning” link will disappear from the assignment page if the checkbox is not checked.

===Helpers===

app/helpers/grades_helper.rb
*retrieve_questions: add an extra parameter <code>team_id</code> whose default value is set to be nil. It then invoke <code>Questionnaire</code> model’s <code>questions</code> method with this <code>team_id</code> and retrieve a proper question set for each team.

===Schema===

*Assignment table: add one column named <code>is_revision_planning_enabled?</code> to indicate whether this feature has been activated.
*Question table: add one column named <code>team_id</code> to distinguish whether the question is from the official review rubric or from a particular team's revision plan.

=Testing=

==RSpec Test Plan==

'''Controllers'''
*spec/controllers/questionnaires_controller_spec.rb
**Describe ‘#edit_revision_plan’
***Context ‘when params[:id] is valid’
***Context ‘when params[:id] is not valid'
**Describe ‘#update_revision_plan’
***Context 'when params[:add_new_questions] is not nil'
***Context 'when params[:view_advice] is not nil'
***Context 'when both params[:add_new_questions] and params[:view_advice] are nil'
*spec/controllers/grades_controller_spec.rb
**Describe ‘#view’
***Context ‘When the revision plan is included in one round’
**Describe ‘#view_my_scores’
***Context ‘When the revision plan is included in one round’
*spec/controllers/response_controller_spec.rb
**Describe

'''Models'''
*spec/models/question_spec.rb
**Describe ‘#questions’
***Context ‘when team_id is supplied’
***Context ‘when team_id is not supplied’

==UI Testing Instructions (For Reviewers)==

'''Setup'''

Login information
*Visit xxx [expertiza deployment link]
User name: instructor6/student8030/student8031
Password: password
Enable/Disable Revision Planning
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Review strategy'''” tab, check the checkbox labeled “'''Enable Revision Planning?'''” to enable the revision planning feature.
Enable/Disable the “Revision Planning” link
*After the instructor configures the assignment to include the revision planning feature, the “'''Revision Planning'''” link will appear on the student's assignment page but will remain disabled and only be enabled during each submission period after round 1. Therefore, to enable the link:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date in the future.
*To disable the link after round 2 submission period:
**Login as instructor6.
**Go to an assignment’s edit page. Under the “'''Due dates'''” tab, change the round 2 submission date to whenever date from the past and change the round 2 review date to whenever date in the future.
Create teams for the assignment
*Login as instructor6.
*Go to an assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Add participant'''”.
*In the new page, add two students, student8030 and student8031, so one student can create revision plan questions during the submission period while the other can respond to these questions during the review period.
*Go back to the assignment’s edit page. Under the “'''Other stuff'''” tab, click “'''Create teams'''”.
*In the new page, group the two added students to separate teams.

Edit a Revision Plan
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link. which redirects the user to a new page used to create a revision plan. Fill the blanks and click on the “'''Save review questionnaire'''” button, and the revision plan should be saved.
Test retrieval of revision plan questions for a specific team
*Login as student8030.
*In the assignment page, click on the “'''Revision Planning'''” link after steps under '''Edit a Revision Plan''' have been done. The “'''Revision Planning'''” link should redirect the user to the Revision Planning edit page that is populated with previously saved questions.
Check the Revision Plan questions in the questionnaire.
*Login as instructor6.
*Adjust the time frame to the second round review period.
*Log out and log in again as student8031.
*In the assignment page, click on the “'''Others’ work'''” link, which takes the user to the review page where one requests a new team’s submission to review. Go to the only other team’s review page and check if the questions are properly displayed under the “Revision Planning” subheader.
Check responses to the Revision Plan questions
*Login again as student8030.
*In the assignment page, click on the “'''Alternative View'''” link, and see if student8030 gets responses for both the original rubric questions as well as its revision plan questions.