Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2019 - Project E1905. Refactor questionnaires controller.rb

2019-04-04T17:55:50Z

Vshah: /* The export and import methods */

[[File:Example.jpg]]__TOC__

===Description of the project===
The questionnaires controller had several issues; this project aimed to address some of them, including:

* Remove or move logic that should reside elsewhere, e.g. in a model class.
* Remove logic and references to logic that is no longer being used.

===Files modified in the project===
# app/controllers/questionnaires_controller.rb
# app/models/questionnaire.rb
# app/views/questionnaires/_questionnaire.html.erb
# app/views/questionnaires/edit_questionnaire.html.erb (deleted)
# spec/controllers/questionnaires_controller_spec.rb
# spec/models/questionnaire_spec.rb

===Issues and Improvements===

====The ''assign_instructor_id'' method====

=====Problems=====
This method had two issues:
# Its name was misleading because it did not actually assign a value; it simply obtained a value from the user in the session.
# It was delving into implementation details of the ''User'' class.

=====Solution=====
The equivalent logic was already implemented in the ''User'' class's ''instructor_id'' method, so calls to the ''assign_instructor_id'' method were simply replaced with:
<pre>
session[:user].instructor_id
</pre>

A description in ''spec/controllers/questionnaires_controller_spec.rb'' was updated to reflect that ''assign_instructor_id'' was no longer in scope.

====The ''export'' and ''import'' methods and ''edit_questionnaires.html.erb''====

=====Problems=====
These methods were actually no longer being used, because the functionality they provided was moved to the ''export_file'' and ''import_file'' routes.

=====Solution=====
* The logic was triggered by the presence of the parameters ''import'' or ''export'' in the ''save_all_questions'' action. Code that caused these parameters to be present had been commented out of ''_questionnaire.html.erb'' and ''edit_questionnaire.html.erb'' in the questionnaire views. The latter file was no longer reachable, as there was no reference to an ''edit_questionnaire'' action in ''config/routes.rb'' or any direct references to it in the questionnaires controller itself. Thus the commented-out code was removed from ''_questionnaire.html.erb'' and the ''edit_questionnaire.html.erb'' file was deleted.
* The logic referencing the parameters and methods was removed from the ''save_all_questions'' method.

====The 'copy' and 'copy_questionnaire_details' methods====

=====Problems=====

#There were two methods ''copy'' and ''copy_questionnaire_details'' present in app/controllers/questionnaire_controller.rb which implemented just one functionality of making a copy of a particular questionnaire.
#There was no need of two methods in the controller itself. Also, some parts of what was happening needed to be in the model.

=====Solution=====
* Nothing special was happening in the ''copy_questionnaire_details'' method. Also, it needed to be placed into the model because of the nature of things happening inside it.
* So, now we have one method ''copy'' in the controller, which calls the ''copy_questionnaire_details'' method, which is present in the Questionnaire model. The instructor_id as well as the params are passed as arguments to the ''copy_questionnaire_details'' method.
* The other methods that the older ''copy_questionnaire_details'' methods used to call, like ''assign_instructor_id'' were removed in the changes above and they are now included in the method in the model itself.
* The Exception (if it occurs while saving the Questionnaire object) is being handled in the ''copy'' method of the controller so that the user can be redirected easily.

====Making code DRYer====

=====Problems=====

#There were parts of the code, specifically when QuizQuestionChoice was created, which was copied throughout the questionnaires_controller.rb.

=====Solution=====
* To make the code DRYer, the repeating code sequences were added into a function called ''create_quiz_question_choice''.
* Calls were made to this method wherever QuizQuestionChoice was made.

=== Test Plan ===
There is no means of testing the removal of dead code, except to run the existing suite of test cases:
<pre>
rspec spec
</pre>
Removal of the ''assign_instructor_id'' method can be tested by running test cases for the questionnaires controller, since existing test cases rely on the instructor ID being retrieved correctly:
<pre>
rspec spec/controllers/questionnaires_controller_spec.rb
</pre>
Testing the method ''create_quiz_question_choice'' can likewise be done with this command.
The logic associated with getting the instructor ID from a user is tested using
<pre>
rspec spec/models/user_spec.rb
</pre>
The additions to the questionnaires model can be tested using
<pre>
rspec spec/models/questionnaire_spec.rb
</pre>
Note: If you are having difficulties running the ''rspec'' command, try using
<pre>
bundle exec rspec
</pre>
instead. This will use the gems specified in ''Gemfile'' that were installed via ''bundle install''.

Also, tests were included for the code movement from the questionnaires controller to the questionnaire model. Checks were being made for database calls in the controller rspec method, the same format was followed while writing tests for the moved methods.

CSC/ECE 517 Fall 2007/wiki2 4 2q

2019-01-10T19:29:31Z

Vshah: /* Basic use case template */

'''A Guide to Use Cases on the WEB'''

''Topic :Use cases. There are even more pages on the Web on use cases than on MVC. If someone wants to learn about them, what should (s)he do? Look at the first few hits in Google? I expect we can do better than that. Write a review of the use-case sites on the Web. Which are best for learning about the concept? Which have the most instructive examples? Which teach advanced concepts that are not apparent in perusing most use-case sites? Read at least several dozen pages before deciding how to organize your overview.''

=Introduction=

This page tries to review the use-case sites on the Web which can be useful to best understand the concept. Also a few instructive examples are covered paying attention to the basic use case as well as advanced concepts that are not apparent in perusing most use-case sites.

=What is a Use Case?=

==Definition==
<blockquote> "'''''Use case''' is a methodology used in software and systems engineering to understand and identify the functional requirements of a system. Use cases describe the interaction between a primary actor, the initiator of the interaction and the system itself, represented as a sequence of simple steps.''"[1][2]</blockquote>

An actor who can more generally be described as the user of the system, is the external entity and can be a person or another system itself. A use case can be best described as what happens when the actor is using the system to achieve some goal. This goal is nothing but the end result which follows the sequence of events as described in the use cases.

Every use case gives a complete list of events which occur from the time the actor starts interacting with the system till the time the final goal is achieved. A use case also records all the possible scenarios that may occur while trying to reach the desired goal. We can think of it as existence of a main course of events that will occur in normal circumstances, to which the exceptional cases are added. Anything that is not relevant to the actor/user is not a part of the use case.

Listed below are a few links to external websites which further illustrate this concept.

* [https://www.ibm.com/developerworks/library/ws-using-oo/ OO design process: Use cases, an introduction]
* [http://www.cs.uno.edu/~jaime/Courses/4210/useCaseFundamentals.html Use Case Fundamentals]

==UML Diagrams and Use cases==
<blockquote> "''The Use case diagram is used to identify the primary elements and processes that form the system. The primary elements are termed as 'actors' and the processes are called "use cases." The Use case diagram shows which actors interact with each use case."''[3]</blockquote>

They are useful in providing a pictorial view of the interactions between the users (actors) and sets of events (use cases) associated with the system.They also evolve as the projects or system change with time. The standard elements of a use case diagram makes comprehension easy for anyone trying to understand the use case.

Listed below are useful links in understanding how to draw typical use case diagrams.

* [https://www.smartdraw.com/uml-diagram/ An introduction to UML diagrams & Tutorial]
* [http://www.developer.com/design/article.php/2109801 Elements of a Use Case Diagram]

=Examples using Use Cases=
In this section, a few instructive examples have been outlined along with their references which illustrate different ways in which use cases can be written. The first example is a basic use-case template.

===Basic use case template===
Described below are some the characteristics of a typical use case.
'''CHARACTERISTIC INFORMATION'''
* ''Goal in Context'': Buyer issues request directly to our company, expects goods shipped and to be billed.
* ''Scope'': Company
* ''Preconditions'': We know Buyer, their address, etc.
* ''Success End Condition'': Buyer has goods, we have money for the goods.
* ''Failed End Condition'': We have not sent the goods, Buyer has not spent the money.
* ''Primary Actor'': Buyer, any agent (or computer) acting for the customer
* ''Trigger'': purchase request comes in.

'''MAIN SUCCESS SCENARIO'''
1. Buyer calls in with a purchase request.
2. Company captures buyer’s name, address, requested goods, etc.
3. Company gives buyer information on goods, prices, delivery dates, etc.
4. Buyer signs for order.
5. Company creates order, ships order to buyer.
6. Company ships invoice to buyer.
7. Buyers pays invoice.

'''EXTENSIONS'''
3a. Company is out of one of the ordered items:
3a1. Renegotiate order.
4a. Buyer pays directly with credit card:
4a1. Take payment by credit card (use case 44)
7a. Buyer returns goods:
7a. Handle returned goods (use case 105)
The basic steps that can be followed so as to ensure writing an effective use case has been described well in [http://www.gatherspace.com/static/use_case_example.html Writing Effective Use Cases].[4]

===Restaurant===
This example provides one use case but distinctly provides a step by step analysis of the events that constitute a Use Case. It mentions the actors, pre-conditions, Exceptions and Alternative scenarios.

* [http://www.uml-diagrams.org/restaurant-uml-use-case-diagram-example.html Restaurant]

===Purchase order===
This example provides a typical use case specification by listing the Activity, Result, Constraints and Extensions involved with a purchase order from a Finance web page.

* [https://creately.com/diagram/example/ifq4uulr/Purchase%20Order Purchase order System]

===ATM system===
This example is a little more involved in that it provides a number of use cases associated with using an ATM system including cash withdrawal,
transfers, deposits, balance inquiry, etc.

* [http://www.cs.fsu.edu/~baker/swe1/restricted/templates/rr631gv1_stuwrk_withdraw_cash_use-case_spec.pdf ATM System]

===A shopping website order Process===
This is a very good example that describes the scenarios and extensions involved in writing a use case for a online shopping order. It also describes the UML diagram.

* [http://www.uml-diagrams.org/examples/online-shopping-use-case-diagram-example.html?context=uc-examples Shopping website order]

===Self Service Machine===
This example provides the typical Use Cases in a self service machine. Some of the aspects of using a self service machine are: buying a product, re-stocking the machine, collecting the money etc. The following link expands on how to do this and also provides the UML diagrams which depict the interaction between the users and system through the use case

* [http://www.cloudbus.org/~raj/254/Lectures/RajUML2.pdf Self-Service Machine (page-3)]

=How to Write a Use Case?=

==Writing Effective Use Cases==

One of the most difficult problem in software development is capturing precisely what you want to build. Inaccurate requirement will end-up with significant delay, rework or even abandonment of the project. Effective applying use case technique helps your team capturing requirements in user point of view which can be easily understood by both the end-user and your team. Use case driven development support subsequent development activities such as analysis and design and testing.

Listed below are useful links in understanding how to write effective use cases.
* [https://www.visual-paradigm.com/tutorials/writingeffectiveusecase.jsp Writing Effective Use Cases]

=References=

* [1] Ivar Jacobson (1992). Object-Oriented Software Engineering. Addison Wesley Professional. ISBN 0-201-54435.
* [2] [http://en.wikipedia.org/wiki/Use_case Wikipidea Page] Basic definition of Use Case
* [3] [http://www.developer.com/design/article.php/2109801 Use Case Diagrams]
* [4] [http://www.gatherspace.com/static/use_case_example.html#1 Writing Effective Use Cases]

=External Links=

* [http://www.cs.uno.edu/~jaime/Courses/4210/useCaseFundamentals.html Use Case Fundamentals]
* [http://www.developer.com/design/article.php/2109801 Elements of a Use Case Diagram]
* [https://www.visual-paradigm.com/tutorials/writingeffectiveusecase.jsp Writing effective Use Cases]

CSC/ECE 517 Fall 2007/wiki2 4 2q

2019-01-10T19:29:12Z

Vshah: /* References */

'''A Guide to Use Cases on the WEB'''

''Topic :Use cases. There are even more pages on the Web on use cases than on MVC. If someone wants to learn about them, what should (s)he do? Look at the first few hits in Google? I expect we can do better than that. Write a review of the use-case sites on the Web. Which are best for learning about the concept? Which have the most instructive examples? Which teach advanced concepts that are not apparent in perusing most use-case sites? Read at least several dozen pages before deciding how to organize your overview.''

=Introduction=

This page tries to review the use-case sites on the Web which can be useful to best understand the concept. Also a few instructive examples are covered paying attention to the basic use case as well as advanced concepts that are not apparent in perusing most use-case sites.

=What is a Use Case?=

==Definition==
<blockquote> "'''''Use case''' is a methodology used in software and systems engineering to understand and identify the functional requirements of a system. Use cases describe the interaction between a primary actor, the initiator of the interaction and the system itself, represented as a sequence of simple steps.''"[1][2]</blockquote>

An actor who can more generally be described as the user of the system, is the external entity and can be a person or another system itself. A use case can be best described as what happens when the actor is using the system to achieve some goal. This goal is nothing but the end result which follows the sequence of events as described in the use cases.

Every use case gives a complete list of events which occur from the time the actor starts interacting with the system till the time the final goal is achieved. A use case also records all the possible scenarios that may occur while trying to reach the desired goal. We can think of it as existence of a main course of events that will occur in normal circumstances, to which the exceptional cases are added. Anything that is not relevant to the actor/user is not a part of the use case.

Listed below are a few links to external websites which further illustrate this concept.

* [https://www.ibm.com/developerworks/library/ws-using-oo/ OO design process: Use cases, an introduction]
* [http://www.cs.uno.edu/~jaime/Courses/4210/useCaseFundamentals.html Use Case Fundamentals]

==UML Diagrams and Use cases==
<blockquote> "''The Use case diagram is used to identify the primary elements and processes that form the system. The primary elements are termed as 'actors' and the processes are called "use cases." The Use case diagram shows which actors interact with each use case."''[3]</blockquote>

They are useful in providing a pictorial view of the interactions between the users (actors) and sets of events (use cases) associated with the system.They also evolve as the projects or system change with time. The standard elements of a use case diagram makes comprehension easy for anyone trying to understand the use case.

Listed below are useful links in understanding how to draw typical use case diagrams.

* [https://www.smartdraw.com/uml-diagram/ An introduction to UML diagrams & Tutorial]
* [http://www.developer.com/design/article.php/2109801 Elements of a Use Case Diagram]

=Examples using Use Cases=
In this section, a few instructive examples have been outlined along with their references which illustrate different ways in which use cases can be written. The first example is a basic use-case template.

===Basic use case template===
Described below are some the characteristics of a typical use case.
'''CHARACTERISTIC INFORMATION'''
* ''Goal in Context'': Buyer issues request directly to our company, expects goods shipped and to be billed.
* ''Scope'': Company
* ''Preconditions'': We know Buyer, their address, etc.
* ''Success End Condition'': Buyer has goods, we have money for the goods.
* ''Failed End Condition'': We have not sent the goods, Buyer has not spent the money.
* ''Primary Actor'': Buyer, any agent (or computer) acting for the customer
* ''Trigger'': purchase request comes in.

'''MAIN SUCCESS SCENARIO'''
1. Buyer calls in with a purchase request.
2. Company captures buyer’s name, address, requested goods, etc.
3. Company gives buyer information on goods, prices, delivery dates, etc.
4. Buyer signs for order.
5. Company creates order, ships order to buyer.
6. Company ships invoice to buyer.
7. Buyers pays invoice.

'''EXTENSIONS'''
3a. Company is out of one of the ordered items:
3a1. Renegotiate order.
4a. Buyer pays directly with credit card:
4a1. Take payment by credit card (use case 44)
7a. Buyer returns goods:
7a. Handle returned goods (use case 105)
The basic steps that can be followed so as to ensure writing an effective use case has been described well in [http://www.gatherspace.com/static/use_case_example.html Writing Effective Use Cases].[5]

===Restaurant===
This example provides one use case but distinctly provides a step by step analysis of the events that constitute a Use Case. It mentions the actors, pre-conditions, Exceptions and Alternative scenarios.

* [http://www.uml-diagrams.org/restaurant-uml-use-case-diagram-example.html Restaurant]

===Purchase order===
This example provides a typical use case specification by listing the Activity, Result, Constraints and Extensions involved with a purchase order from a Finance web page.

* [https://creately.com/diagram/example/ifq4uulr/Purchase%20Order Purchase order System]

===ATM system===
This example is a little more involved in that it provides a number of use cases associated with using an ATM system including cash withdrawal,
transfers, deposits, balance inquiry, etc.

* [http://www.cs.fsu.edu/~baker/swe1/restricted/templates/rr631gv1_stuwrk_withdraw_cash_use-case_spec.pdf ATM System]

===A shopping website order Process===
This is a very good example that describes the scenarios and extensions involved in writing a use case for a online shopping order. It also describes the UML diagram.

* [http://www.uml-diagrams.org/examples/online-shopping-use-case-diagram-example.html?context=uc-examples Shopping website order]

===Self Service Machine===
This example provides the typical Use Cases in a self service machine. Some of the aspects of using a self service machine are: buying a product, re-stocking the machine, collecting the money etc. The following link expands on how to do this and also provides the UML diagrams which depict the interaction between the users and system through the use case

* [http://www.cloudbus.org/~raj/254/Lectures/RajUML2.pdf Self-Service Machine (page-3)]

=How to Write a Use Case?=

==Writing Effective Use Cases==

One of the most difficult problem in software development is capturing precisely what you want to build. Inaccurate requirement will end-up with significant delay, rework or even abandonment of the project. Effective applying use case technique helps your team capturing requirements in user point of view which can be easily understood by both the end-user and your team. Use case driven development support subsequent development activities such as analysis and design and testing.

Listed below are useful links in understanding how to write effective use cases.
* [https://www.visual-paradigm.com/tutorials/writingeffectiveusecase.jsp Writing Effective Use Cases]

=References=

* [1] Ivar Jacobson (1992). Object-Oriented Software Engineering. Addison Wesley Professional. ISBN 0-201-54435.
* [2] [http://en.wikipedia.org/wiki/Use_case Wikipidea Page] Basic definition of Use Case
* [3] [http://www.developer.com/design/article.php/2109801 Use Case Diagrams]
* [4] [http://www.gatherspace.com/static/use_case_example.html#1 Writing Effective Use Cases]

=External Links=

* [http://www.cs.uno.edu/~jaime/Courses/4210/useCaseFundamentals.html Use Case Fundamentals]
* [http://www.developer.com/design/article.php/2109801 Elements of a Use Case Diagram]
* [https://www.visual-paradigm.com/tutorials/writingeffectiveusecase.jsp Writing effective Use Cases]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T22:16:50Z

Vshah: /* References */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video of running output is available here: [https://youtu.be/R_lMT5bXHEk] -- As you can see in the later part of the video, HTML tags have been rendered correctly.

==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

==== Test Case 3 ====

1. Edit any text field and enter a supported UTF8 character.
2. Save.
3. Try to edit it again and try to save. Earlier, this step was failing as it tried entering UTF8 character to the versions table which did not support UTF8.

After the fix, editing it works as seen here: [https://youtu.be/PMe-jgLZw3w]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive. The video is available here: [https://youtu.be/cT9Tju-_fY4]

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Video Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Video Issue 1: UTF-8 character support : [https://youtu.be/PMe-jgLZw3w]
# Video Issue 2: Rendering HTML : [https://youtu.be/R_lMT5bXHEk]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T22:16:38Z

Vshah: /* Test Case 2 */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video of running output is available here: [https://youtu.be/R_lMT5bXHEk] -- As you can see in the later part of the video, HTML tags have been rendered correctly.

==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

==== Test Case 3 ====

1. Edit any text field and enter a supported UTF8 character.
2. Save.
3. Try to edit it again and try to save. Earlier, this step was failing as it tried entering UTF8 character to the versions table which did not support UTF8.

After the fix, editing it works as seen here: [https://youtu.be/PMe-jgLZw3w]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive. The video is available here: [https://youtu.be/cT9Tju-_fY4]

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Video Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Video Issue 1: UTF-8 character support : [https://youtu.be/fDBx1OcKYlA]
# Video Issue 2: Rendering HTML : [https://youtu.be/R_lMT5bXHEk]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T22:09:24Z

Vshah: /* Test Case 1 */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video of running output is available here: [https://youtu.be/R_lMT5bXHEk] -- As you can see in the later part of the video, HTML tags have been rendered correctly.

==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive. The video is available here: [https://youtu.be/cT9Tju-_fY4]

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Video Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Video Issue 1: UTF-8 character support : [https://youtu.be/fDBx1OcKYlA]
# Video Issue 2: Rendering HTML : [https://youtu.be/R_lMT5bXHEk]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T22:08:44Z

Vshah: /* Test Case 1 */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video of running output is available here: [https://youtu.be/R_lMT5bXHEk]

==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive. The video is available here: [https://youtu.be/cT9Tju-_fY4]

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Video Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Video Issue 1: UTF-8 character support : [https://youtu.be/fDBx1OcKYlA]
# Video Issue 2: Rendering HTML : [https://youtu.be/R_lMT5bXHEk]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T22:07:54Z

Vshah: /* Automated Tests */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video attached with the submission shows this case
==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive. The video is available here: [https://youtu.be/cT9Tju-_fY4]

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Video Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Video Issue 1: UTF-8 character support : [https://youtu.be/fDBx1OcKYlA]
# Video Issue 2: Rendering HTML : [https://youtu.be/R_lMT5bXHEk]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T22:07:03Z

Vshah: /* References */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video attached with the submission shows this case
==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Video Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Video Issue 1: UTF-8 character support : [https://youtu.be/fDBx1OcKYlA]
# Video Issue 2: Rendering HTML : [https://youtu.be/R_lMT5bXHEk]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-09T21:43:32Z

Vshah: /* References */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

As part of the project we were given task to fix following two issues.

1. [https://github.com/expertiza/expertiza/issues/927 Issue 1]: In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. [https://github.com/expertiza/expertiza/issues/962 Issue 2]: The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

These are important issues from usability point of view and need to be fixed.

== Problem as experienced by the user ==

===Issue 1===
While trying to save utf8 chars the application was throwing up error as following

[[File:Issue1_error.PNG]]

===Issue 2===
If a user looks at the review score table, he/she can see html tags along with the review comments. There are other instances such as the self review page where the problem is repeated.

[[File:Html_tags_error.png]]

== Solution : Approach and Fix ==

=== Issue 1 ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

Thus we see that after lot of exploration and careful debugging we were able to zero down on the root cause. This approach is the best possible outcome as it is not a point fix and solves the root cause. Fix also avoids any duplication of code / point fixes.

=== Issue 2 ===

The HTML template issue was caused due to a security feature of Ruby , which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize() function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.
[https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/sanitize sanitize]

We had to carefully test all flows where the html tags were getting rendered. We discovered about more than 2 places where this could be done.

However there were additional screens such as tooltip in grade scores which too were showing html tags. Therefore we added code to strip tags in html.

For tooltip changes, sanitize does not work, therefore we escaped all html form the text.

<td class="<%= score.color_code %>" align="center" data-toggle="tooltip" title="<%= strip_tags(score.comment).html_safe%>">

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb 
5. Changed app/views/grades/view_team.html.erb 
6. Changed db/schema.rb 

== Test Plan ==
=== Manual UI Tests ===
To check if our fix for rendering html tags works we relied on manual setup and verification. We reran the steps that were causing the issue in first place and checked if it is fixed by inspection. 

==== Test Case 1 ====
Steps for verification : 
1. Log in as instructor6 
2. Create a copy of “Final Project (and design doc)” 
3. Change all due dates to today’s dates. Enable “Use signup deadline” as well and set that date to day’s date 
4. Save 
5. Click on add participants, then import all participants from course (The following steps assume student7488 and student7489 were added to the course during this step. Double-check to make sure they were added, or use some other students that were added) 
6. Login as student7488 – sign up for the first topic. 
7. Login as student7489 – sign up for second topic. 
8. Login as instructor6 – set signup date to yesterday’s date (so we are now past the sign up deadline) 
9. Login as student7488, submit any example hyperlink for this assignment. Perform a self review under “Your scores” (Add bold, italics, lists etc.. in this review as well) 
10. Do the same as student7489 
11. Login as instructor6 – set submission date to yesterday’s date (so we are now past the submission deadline) 
12. Login as student7488 – go to “Other’s work” and click “Request new submission for review”. Perform review. 
13. Do the same as student7489. 
14. Login as instructor6 – set review1 date to yesterday’s date 
15. Navigate back to “Manage -> Assignments”. Click on “View review report”. On the top-left dropdown, select “Self-review report”. Click on “View”. Click on student7488 in the “Self Review” column. You will not see HTML tags if you had entered them in step 9 instead appropriate html will be rendered. 

Video attached with the submission shows this case
==== Test Case 2 ====

1. Repeat 1-8, create assignments and add reviewers 
2. Login as student2 and give review that contains html tags, logout 
3. Login as student1 and check reviews 

After the fix ( refer to section 3.2 ) 
Following is the output with html escaped.
[[File:Issue2 resolved.PNG]]

=== Automated Tests ===

We have added Rspec tests to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# GitHub pull request: [https://github.com/expertiza/expertiza/pull/1225]
# Issue 1: Non-UTF8 character removal test + Rspec runs: [https://youtu.be/cT9Tju-_fY4]
# Issue 1: UTF-8 character support : [https://youtu.be/fDBx1OcKYlA]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T02:03:24Z

Vshah: /* Solution */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Solution Approach ==

=== Files Created or Refactored ===

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb

=== Solution ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T02:03:12Z

Vshah: /* Files Created or Refactored */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Solution Approach ==

=== Files Created or Refactored ===

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb

== Solution ==

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T02:03:00Z

Vshah: /* Solution Approach */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Solution Approach ==

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb

== Solution ==

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T02:02:43Z

Vshah: /* Files Created or Refactored */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

=== Solution Approach ===

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb

== Solution ==

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T02:01:27Z

Vshah: /* Files Created or Refactored */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8 
4. Created Rspec file application_controller_spec.rb

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T02:01:12Z

Vshah: /* Files Refactored */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Created or Refactored ==

The following files were modified for this project namely: 
1. Refactored application_controller 
2. Refactored self_review_popup 
3. Created a new migration - VersionTableSupportUTF8
4. Created Rspec file application_controller_spec.rb

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T01:59:35Z

Vshah: /* Test Plan */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Refactored ==

The following files were modified for this project namely: 
1. application_controller 
2. self_review_popup 
3. Created a new migration - VersionTableSupportUTF8

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. 
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-08T01:59:13Z

Vshah: /* Test Plan */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Refactored ==

The following files were modified for this project namely: 
1. application_controller 
2. self_review_popup 
3. Created a new migration - VersionTableSupportUTF8

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate.
We have implemented RSPEC test to test that the given non-UTF8 character is removed and a valid UTF-8 character is not removed. This ensures that the functionality is exhaustive.

Both the Rspec tests pass:

[[File:vRspec.png]]

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

File:VRspec.png

2018-11-08T01:58:30Z

Vshah: Rspec testing works image.

Rspec testing works image.

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-06T02:33:59Z

Vshah: /* Solution Approach */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Refactored ==

The following files were modified for this project namely: 
1. application_controller 
2. self_review_popup 
3. Created a new migration - VersionTableSupportUTF8

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle. An alternative approach will be removing the problematic characters within individual functions, but this leads to repetitive code and a violation of DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not evaluate strings.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. We are yet to implement an RSPEC test to test that the given non-utf8 character is not saved in the database.

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-05T23:38:28Z

Vshah: /* Testing Details */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Refactored ==

The following files were modified for this project namely: 
1. application_controller 
2. self_review_popup 
3. Created a new migration - VersionTableSupportUTF8

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not render string text.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Test Plan ==

The HTML tags issue is tested by simply checking the output is properly formatted and appropriate. We are yet to implement an RSPEC test to test that the given non-utf8 character is not saved in the database.

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-05T02:28:07Z

Vshah: /* Solution Approach */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Refactored ==

The following files were modified for this project namely: 
1. application_controller 
2. self_review_popup 
3. Created a new migration - VersionTableSupportUTF8

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |
1 row in set (0.01 sec)

We changed the charset with the command: 

mysql> ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Output of show create table now is

mysql> show create table versions;
| Table | Create Table
| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=utf8 |
1 row in set (0.01 sec)

This solved the problem. Therefore, we created the migration VersionTableSupportUTF8 to change version's characterset.

rails g migration VersionTableSupportUTF8

def change
execute "ALTER TABLE versions CONVERT TO CHARACTER SET utf8"
end

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

The HTML template issue was caused due to a security feature of Ruby, which by default does not render string text.
To resolve the HTML template issue, we used the sanitize function. This strips all the tags that aren't whitelisted, thus ruby now renders the standard HTML tags. We have sanitized required pages.

== Testing Details==

We have manually tested the fix.

=== RSpec ===
We found no existing test cases for the above two problem statements. So we are to define test cases which generate the list of unwanted Chinese non-UTF8 characters and also HTML tags in the views and then refactor our code so that it automatically removes them without any failing of functionalities.

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-04T00:04:37Z

Vshah: /* Solution Approach */

E1846. OSS Project Navy: Character Issues Fall 2018, CSC/ECE 517.

== Problem Statement ==

1. In the existing Expertiza setup, the database supports only UTF-8 characters. Hence, if a user enters a non UTF-8 character, the database throws an error. This further leads to loss of data while refreshing or going back to the input page as data wasn't saved in database, effectively leading to loss of entire review if there's even a single non UTF-8 character. We need to solve the problem by removing such unsupported characters.

2. The existing expertiza stores the HTML formatting tags (Like <nowiki></nowiki> for bold) as a string. However, while rendering the string these tags are not escaped, resulting in no formatting. We need to solve the issue and display proper formatting.

== Files Refactored ==

The following files were modified for this project namely: 
1. application_controller 
2. self_review_popup 
3. Created a new migration - VersionTableSupportUTF8

=== Solution Approach ===

One of the solutions proposed was filtering out the non-UTF8 characters before saving the input in the database. Since the non-UTF8 input can come from any view, we implemented a ''filter_non_UTF8'' method in application controller to do just that and adhere to DRY principle.

However, while experimenting with the fix, we found out that not all tables support UTF8 formatting. For E.g., the versions table which has the latin charset.

mysql> show create table versions;

| Table | Create Table |

| versions | CREATE TABLE `versions` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_type` varchar(255) NOT NULL,
`item_id` int(11) NOT NULL,
`event` varchar(255) NOT NULL,
`whodunnit` varchar(255) DEFAULT NULL,
`object` mediumtext,
`created_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_versions_on_item_type_and_item_id` (`item_type`,`item_id`)
) ENGINE=InnoDB AUTO_INCREMENT=142423 DEFAULT CHARSET=latin1 |

1 row in set (0.01 sec)

We changed the charset with the command: 
ALTER TABLE versions CONVERT TO CHARACTER SET utf8;

Which solved the problem. Thus, we created the migration VersionTableSupportUTF8 to change version's characterset.

If we want to fix this in all tables, we can do following for each database via script or add migration for each table. 

SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') FROM information_schema.TABLES WHERE
TABLE_SCHEMA = 'expertiza_development';

== Testing Details==

=== RSpec ===
We found no existing test cases for the above two problem statements. So we are to define test cases which generate the list of unwanted Chinese non-UTF8 characters and also HTML tags in the views and then refactor our code so that it automatically removes them without any failing of functionalities.

== References ==
# Link to the videos to see the steps followed for testing and resolving the problems: [https://drive.google.com/drive/folders/1vA2xm1vo45hHANJ_cv9bwePt-TswgmTM?usp=sharing]

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-02T23:13:12Z

Vshah: /* Files Refactored */

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-02T22:53:06Z

Vshah: /* Problem Statement */

CSC/ECE 517 Fall 2018- Project E1846. OSS Project Navy: Character Issues

2018-11-02T22:51:55Z

Vshah: /* Problem Statement */