Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-14T18:09:33Z

Chuang23: /* Reference */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

1.Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a metric_data_value table for each data type. Then we can change the the value field in table metric_data_points to the value_id, which help us to find the value in the corresponding metric_data_value table.

When storing data in the database, we can create the new data using the factory pattern as shown in the figure below, and the string parameter “type” could be used for specifying which type of data is created. Each metric_data_point_type and metric combination could only have one metric_data_point, so if it already exists, the new data will replace the old one. Here each metric_data_point_type row actually represent one field of a metric. If two or more metric_data_point_type rows have the same value_type, their name should be different. So each row of metric_data_point_type is like a column header of a table, which make the database more extensible.

When querying data in the database, we can first utilize the has_many relationship between metric and metri_data_point to get all the metric_data_points. Then get the each metric_data_value table by the value_type field in the metric_data_point_type. Finally use the value_id in metric_data_points table to find the data.

[[File:Metric_data_factory.png]]

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

For example, even if the new data type is an array, we could use the dimension field in metric_data_point_type to specify the length of the array, then use the value_id field in metric_data_point to specify the start id the array, finally in the corresponding

metric_data_value table we can use the start id and the dimension to get all the elements in the array.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

2. We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

To implement all the parsers, we planned to use a factory pattern as shown in the diagram above. It allows further extension on new metric sources. For each parser class, we implement duck typing by defining the parse method with the same signature, so we don’t really need a super class or interface. The caller code could execute the parese method without knowing exactly which parser is created by the factory. Thus the caller code is close to modification.

[[File:CSC517_Design_Schema_database.png]]

Each parser will parse the given data source and use the metric data factory to create model for each data then use these data to create the metric model.

==Test Plan==
<ol>
<li>Given a valid url of GitHub, the GitHub parser could correctly get all the data.
<li>Given a valid url of Readability, the Readability parser could correctly get all the data.
<li>Given a valid url of Trello, the Trello parser could correctly get all the data.
<li>Test that the parser should correctly create metric_data_point_type if it doesn’t exist.
<li>Test that the parser should be able to create the correct type of metric_data_value model according to the type specified by the metric_data_point_type.
<li>Test that the parser should correctly create metric_data_value models if it doesn’t exist.
<li>Test that the parser should correctly update metric_data_value models if it already exist.
<li>Test that the metric_data_point_value could be correctly queried with the data from a given metric.
</ol>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-14T16:19:42Z

Chuang23: /* Model Improvement */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

1.Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a metric_data_value table for each data type. Then we can change the the value field in table metric_data_points to the value_id, which help us to find the value in the corresponding metric_data_value table.

When storing data in the database, we can create the new data using the factory pattern as shown in the figure below, and the string parameter “type” could be used for specifying which type of data is created. Each metric_data_point_type and metric combination could only have one metric_data_point, so if it already exists, the new data will replace the old one. Here each metric_data_point_type row actually represent one field of a metric. If two or more metric_data_point_type rows have the same value_type, their name should be different. So each row of metric_data_point_type is like a column header of a table, which make the database more extensible.

When querying data in the database, we can first utilize the has_many relationship between metric and metri_data_point to get all the metric_data_points. Then get the each metric_data_value table by the value_type field in the metric_data_point_type. Finally use the value_id in metric_data_points table to find the data.

[[File:Metric_data_factory.png]]

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

For example, even if the new data type is an array, we could use the dimension field in metric_data_point_type to specify the length of the array, then use the value_id field in metric_data_point to specify the start id the array, finally in the corresponding

metric_data_value table we can use the start id and the dimension to get all the elements in the array.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

2. We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

To implement all the parsers, we planned to use a factory pattern as shown in the diagram above. It allows further extension on new metric sources. For each parser class, we implement duck typing by defining the parse method with the same signature, so we don’t really need a super class or interface. The caller code could execute the parese method without knowing exactly which parser is created by the factory. Thus the caller code is close to modification.

[[File:CSC517_Design_Schema_database.png]]

Each parser will parse the given data source and use the metric data factory to create model for each data then use these data to create the metric model.

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-14T16:19:17Z

Chuang23: /* Model Improvement */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

==Model Improvement==

1.Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a metric_data_value table for each data type. Then we can change the the value field in table metric_data_points to the value_id, which help us to find the value in the corresponding metric_data_value table.

When storing data in the database, we can create the new data using the factory pattern as shown in the figure below, and the string parameter “type” could be used for specifying which type of data is created. Each metric_data_point_type and metric combination could only have one metric_data_point, so if it already exists, the new data will replace the old one. Here each metric_data_point_type row actually represent one field of a metric. If two or more metric_data_point_type rows have the same value_type, their name should be different. So each row of metric_data_point_type is like a column header of a table, which make the database more extensible.

When querying data in the database, we can first utilize the has_many relationship between metric and metri_data_point to get all the metric_data_points. Then get the each metric_data_value table by the value_type field in the metric_data_point_type. Finally use the value_id in metric_data_points table to find the data.

[[File:Metric_data_factory.png]]

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

For example, even if the new data type is an array, we could use the dimension field in metric_data_point_type to specify the length of the array, then use the value_id field in metric_data_point to specify the start id the array, finally in the corresponding

metric_data_value table we can use the start id and the dimension to get all the elements in the array.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

2. We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

To implement all the parsers, we planned to use a factory pattern as shown in the diagram above. It allows further extension on new metric sources. For each parser class, we implement duck typing by defining the parse method with the same signature, so we don’t really need a super class or interface. The caller code could execute the parese method without knowing exactly which parser is created by the factory. Thus the caller code is close to modification.

[[File:CSC517_Design_Schema_database.png]]

Each parser will parse the given data source and use the metric data factory to create model for each data then use these data to create the metric model.

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-14T16:18:23Z

Chuang23: /* Design Schema */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Model Improvement==

1.Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a metric_data_value table for each data type. Then we can change the the value field in table metric_data_points to the value_id, which help us to find the value in the corresponding metric_data_value table.

When storing data in the database, we can create the new data using the factory pattern as shown in the figure below, and the string parameter “type” could be used for specifying which type of data is created. Each metric_data_point_type and metric combination could only have one metric_data_point, so if it already exists, the new data will replace the old one. Here each metric_data_point_type row actually represent one field of a metric. If two or more metric_data_point_type rows have the same value_type, their name should be different. So each row of metric_data_point_type is like a column header of a table, which make the database more extensible.

When querying data in the database, we can first utilize the has_many relationship between metric and metri_data_point to get all the metric_data_points. Then get the each metric_data_value table by the value_type field in the metric_data_point_type. Finally use the value_id in metric_data_points table to find the data.

[[File:Metric_data_factory.png]]

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

For example, even if the new data type is an array, we could use the dimension field in metric_data_point_type to specify the length of the array, then use the value_id field in metric_data_point to specify the start id the array, finally in the corresponding

metric_data_value table we can use the start id and the dimension to get all the elements in the array.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

2. We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

To implement all the parsers, we planned to use a factory pattern as shown in the diagram above. It allows further extension on new metric sources. For each parser class, we implement duck typing by defining the parse method with the same signature, so we don’t really need a super class or interface. The caller code could execute the parese method without knowing exactly which parser is created by the factory. Thus the caller code is close to modification.

[[File:CSC517_Design_Schema_database.png]]

Each parser will parse the given data source and use the metric data factory to create model for each data then use these data to create the metric model.

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-14T16:17:11Z

Chuang23: /* Model Improvement */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Model Improvement==

1.Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a metric_data_value table for each data type. Then we can change the the value field in table metric_data_points to the value_id, which help us to find the value in the corresponding metric_data_value table.

When storing data in the database, we can create the new data using the factory pattern as shown in the figure below, and the string parameter “type” could be used for specifying which type of data is created. Each metric_data_point_type and metric combination could only have one metric_data_point, so if it already exists, the new data will replace the old one. Here each metric_data_point_type row actually represent one field of a metric. If two or more metric_data_point_type rows have the same value_type, their name should be different. So each row of metric_data_point_type is like a column header of a table, which make the database more extensible.

When querying data in the database, we can first utilize the has_many relationship between metric and metri_data_point to get all the metric_data_points. Then get the each metric_data_value table by the value_type field in the metric_data_point_type. Finally use the value_id in metric_data_points table to find the data.

[[File:Metric_data_factory.png]]

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

For example, even if the new data type is an array, we could use the dimension field in metric_data_point_type to specify the length of the array, then use the value_id field in metric_data_point to specify the start id the array, finally in the corresponding

metric_data_value table we can use the start id and the dimension to get all the elements in the array.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

2. We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

[[File:CSC517_Design_Schema_database.png]]

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

File:Metric data factory.png

2017-11-14T16:11:48Z

Chuang23: uploaded a new version of "File:Metric data factory.png"

File:Metric data factory.png

2017-11-14T16:11:10Z

Chuang23:

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-14T16:10:52Z

Chuang23: /* Design Schema */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Model Improvement==

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

[[File:CSC517_Design_Schema_database.png]]

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T23:36:44Z

Chuang23: /* Design Schema */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

[[File:CSC517_Design_Schema_database.png]]

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T23:36:10Z

Chuang23: /* Design Schema */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Design Schema==

[[File: CSC517_Design_Schema_factory.png]]

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

File:CSC517 Design Schema database.png

2017-11-13T23:35:05Z

Chuang23:

File:CSC517 Design Schema factory.png

2017-11-13T23:34:39Z

Chuang23:

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T23:16:46Z

Chuang23: /* Reference */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Design Schema==

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T23:16:02Z

Chuang23: /* Introduction */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T17:16:26Z

Chuang23: /* Model: */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg]]
 

====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T17:14:06Z

Chuang23: /* Metric_data_point_types */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg ]]
 
====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===Metric_data_point_types===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T17:13:38Z

Chuang23: /* Metric */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg ]]
 
====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===''Metric_data_point_types''===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T17:13:22Z

Chuang23: /* Metric_data_points */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg ]]
 
====Schema:====
[[File: metrics_schema.png]]

===Metric_data_points===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===''Metric_data_point_types''===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T17:12:34Z

Chuang23: /* Metric */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===Metric===
====Model:====
[[File:Metrics_models.jpg ]]
 
====Schema:====
[[File: metrics_schema.png]]

===''Metric_data_points''===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===''Metric_data_point_types''===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T11:13:38Z

Chuang23: /* 2. Trello[3][4] */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===''Metric''===
====Model:====
[[File:Metrics_models.jpg ]]
 
====Schema:====
[[File: metrics_schema.png]]

===''Metric_data_points''===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===''Metric_data_point_types''===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2.[https://developers.trello.com/reference#tokens Trello]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T11:13:01Z

Chuang23: /* Reference */

==Introduction==
In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will introduce the idea of "Readability." To get the level of readability, we will import the content of write-ups written by students, split the sentences to get the number of sentences, the number of words, etc., and then we calculate the indices by using these numbers and formulas.

Our primary task for the final project is to design tables which allow the Expertiza app to store data fetched from external sources, such as GitHub, Trello, and write-ups. For the next step, we would like to utilize this raw data for virtualized charts and grading metrics.

==Current Design==
Currently, there are three models created to store the raw data from metrics source. (Metrics, Metric_data_points, Metric_data_point_types)

===''Metric''===
====Model:====
[[File:Metrics_models.jpg ]]
 
====Schema:====
[[File: metrics_schema.png]]

===''Metric_data_points''===
====Model:====

[[File:metric_data_points_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

===''Metric_data_point_types''===
====Model:====
[[File:metric_data_point_types_model.jpg]]

====Schema:====
[[File:metric_data_points_schema.png]]

The current framework only defined the schema, but the models are still empty, and the methods of the data parser have not been implemented yet.

This schema is a clever design because it follows the "Open to Extension and Closed to Modification" principle. When new data is added to the database, developers don't have to change the metric_data_point_types and metric_data_points tables. The developers only need to add two methods to translate the data type to and from strings. By browsing the code, the most basic types already have those methods to meet our requirements. But it is not flawless, and we will talk about the problems in the next section.

Besides, we only have GitHub to be our data source currently. As a result, we also need to find other data sources to be one of the grading metrics.

==Analysis of the Problem==

<li>In model metric_data_points, it is defined that each metric_data_point belongs to a metric and a metric_data_point_type. However, in both model metric_data_point and metric, they haven’t defined has_many metric_data_points. So we can’t query all the metric_data_points of a metric or a metric_data_point_type.</li>

<li>In schema metric_data_points, value has been defined as a string to accommodate different data type. But this requires the program to translate data into strings when storing the data and translating the string back to data when accessing it. To make it worse, using strings to store data types, such as float type or Time class, would either lose the precision or incur more abundant storage space.</li>

<li>More external sources are needed.</li>

==Proposed Design==

==='''Current source: GitHub'''===

Our first and current source for grading metrics is GitHub, and Zach and Tyler have implemented the integration with GitHub API for fetching the commit data. However, the integration looks that it is still in the first stage; the app can fetch the data and store into the database. We don't have actual implementations of getting the valid data and the usage of this data to be one of the grading metrics yet.

From the API offered by GitHub, we can fetch the commit information that the number of additions and the number of deletions is made by each contributor in the repository. We can just use these numbers to calculate the contributions for the metrics, or we can put these numbers into some equations to get the impact factor to represent as contributions for each group member.

==='''Proposed source for metrics:'''===

===='''1. [https://www.webpagefx.com/tools/read-able/ Readability]'''====

Sometimes, students only have project write-ups to submit (for example, this stage of the final project). As a result, there might be no GitHub commits to check the number of additions or deletions of the working repository. Here, we introduce some formulas for calculating "Readability" of those write-ups to be one of grading metrics.

The readability indices contain:
====='''Flesch Kincaid Reading Ease'''=====

Based on a 0-100 scale. A high score means the text is easier to read. Low scores suggest the text is complicated to understand.

206.835 - 1.015 x (words/sentences) - 84.6 x (syllables/words)
A value between 60 and 80 should be easy for a 12 to 15 year old to understand.

====='''Flesch Kincaid Grade Level'''=====

0.39 x (words/sentences) + 11.8 x (syllables/words) - 15.59
====='''Gunning Fog Score'''=====

0.4 x ( (words/sentences) + 100 x (complexWords/words) )
====='''SMOG Index'''=====

1.0430 x sqrt( 30 x complexWords/sentences ) + 3.1291
====='''Coleman Liau Index'''=====

5.89 x (characters/words) - 0.3 x (sentences/words) - 15.8
====='''Automated Readability Index (ARI)'''=====
4.71 x (characters/words) + 0.5 x (words/sentences) - 21.43

To calculate these indices, we need to fetch the article to get the number of sentences, the number of words and the number of complex words, and then we can use these numbers to calculate the indices mentioned above to get the readability level.

===='''2. Trello[3][4]'''====
Trello is a web-based project management application. It helps students to understand what tasks have been accomplished, what works are in progress and what jobs are waiting for being started by adding cards and writing down to-do lists inside, and it also helps instructors to keep track of how students work by looking into these cards and lists.

It is useful for both coding projects and writing projects because we can fetch the information from the activities to calculate the percentages of the workloads for each group member, and this result can be one of the grading metrics.

For example, a group of students has a to-do list, which contains eight tasks. Student A finishes two tasks, Student B finishes one, and Student finishes one as well. We can use the information fetched from the activity of project from Trello RESTful API to calculate the percentage of finished jobs (which is 50%) and the proportions of contributions for each student (which are 50%, 25%, 25%, respectively).

Using this data from Trello might be a good idea for being one of the grading metrics because we cannot merely conclude contributions by observing the number of additions and the number of deletions. What if a student just adjusts the indentation for all files in the project for an hour and the other student thinks about a complicated algorithm to get a correct answer for days or even weeks?

However, the data from Trello offers a different aspect of the grading rubric; it concludes the contributions by calculating how many tasks are done by each student. Also, it can be used as a grading metric for both coding projects and writing projects because we only care about the todo-lists.

====='''Model Improvement'''=====

<li>
Since each metric and metric_data_point_type could have many metric_data_points, we need to add has_many metric_data_points in those models.

To make the database store different data types as they are, we can create a table for each data type. Then we can change the value field in table metric_data_points to data_id, which is a foreign key of the corresponding data type.

When storing data in the database, we can create the new data using the factory pattern, and the name of datatype could be used for specifying which type of data is created.

When querying data in the database, we can first get the model class by the type name, then use the data_id from table metric_data_points to data_id to find the data.

This design could solve the problem discussed above. Each data would be stored as their original data type. Thus, we do not need to convert data type to/from strings which lead to precision loss or space problem.

This design also follows "Open to Extension and Closed to Modification" principle. Each time when we need to add a new datatype, we don’t need to modify the previous schema, we only need to add a new model and schema.

But this design has drawbacks too. First of all, the logic becomes more complex, it would be more difficult for a programmer to understand. Secondly, the table of each data type would need to store the id, which costs extra space. However, this overhead should be less than the overhead caused by string storage or frequent type casting.

</li>

<li>
We also need to implement the data parser methods. The data parser methods will be implemented in the data source model class. We can utilize Ruby’s duck typing character to implement polymorphism, so when the application gets a data source object, it could invoke the parser method without knowing which type of data source it is. This ensures this program is closed to modification and the data source is open to extension.
</li>

==Reference==
#[https://www.webpagefx.com/tools/read-able/ Readability1]
#[https://raventools.com/blog/ultimate-list-of-online-content-readability-tests/ Readability2]
#[https://trello.com/ Trello1]
#[https://developers.trello.com/reference#tokens Trello2]

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T11:10:29Z

Chuang23: /* 1. Readability[1][2] */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T11:04:34Z

Chuang23: /* 2. Trello[3][4] */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T11:01:58Z

Chuang23: /* 1. Readability[1][2] */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:59:47Z

Chuang23: /* Flesch Kincaid Reading Ease */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:59:27Z

Chuang23: /* Proposed source for metrics: */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:58:57Z

Chuang23: /* Current source: GitHub */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:58:05Z

Chuang23: /* Reference */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:57:21Z

Chuang23: /* Proposed Design */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:55:11Z

Chuang23: /* Proposed Design */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:50:34Z

Chuang23: /* Analysis of the Problem */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:48:48Z

Chuang23: /* Current Design */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:47:47Z

Chuang23: /* Schema: */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:47:21Z

Chuang23: /* Model: */

File:Metric data point types model.jpg

2017-11-13T10:46:53Z

Chuang23:

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:45:28Z

Chuang23: /* Schema: */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:44:44Z

Chuang23: /* Model: */

File:Metric data points model.jpg

2017-11-13T10:44:10Z

Chuang23: uploaded a new version of "File:Metric data points model.jpg"

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:41:57Z

Chuang23: /* Schema: */

File:Metric data point types schema.png

2017-11-13T10:40:45Z

Chuang23:

File:Metric data points model.jpg

2017-11-13T10:40:33Z

Chuang23:

File:Metric data points schema.png

2017-11-13T10:40:20Z

Chuang23:

File:Metrics schema.png

2017-11-13T10:40:04Z

Chuang23:

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:39:30Z

Chuang23: /* Current Design */

CSC/ECE 517 Fall 2017/E1790 Text metrics

2017-11-13T10:15:27Z

Chuang23: Created page with "==Introduction== In this final project “Text Metric”, first, we will integrate a couple of external sources such as Github, Trello to fetch information. Second, we will intro..."

File:Metrics models.jpg

2017-11-13T10:04:26Z

Chuang23:

File:Metrics models.jpg

2017-11-13T09:58:47Z

Chuang23:

File:Metrics models.jpg

2017-11-13T09:55:35Z

Chuang23:

CSC/ECE 517 Fall 2017

2017-11-12T18:03:43Z

Chuang23:

'''Design Project Documentation'''
*[[CSC/ECE 517 Fall 2017/E17A2 Lightweight Badging System]]
*[[CSC/ECE 517 Fall 2017/E17A8. Use a profiler to identify the problems / pages that take some time to load & fix them]]
*[[CSC/ECE 517 Fall 2017/E1793. Help students find teams to join]]
*[[CSC/ECE 517 Fall 2017/E17A4 Allow calibration to be part of an assignment]]
*[[CSC/ECE 517 Fall 2017/E17A0 Team-based reviewing]]
*[[CSC/ECE 517 Fall 2017/E1793. Help students find teams to join_Team1964]]
*[[CSC/ECE 517 Fall 2017/E1791. Track the time that students look at the other submissions - logging improvement]]
*[[CSC/ECE 517 Fall 2017/E1799 Improve self-review Link peer review & self-review to derive grades]]
*[[CSC/ECE_517_Fall_2017/E1797_Timestamps_for_students_submissions]]
*[[CSC/ECE_517_Fall_2017/E17A1|CSC/ECE_517_Fall_2017/E17A1 - Let experts as well as students do reviews]]
*[[CSC/ECE_517_Fall_2017/M1753_OffscreenCanvas]]
*[[CSC/ECE 517 Fall 2017/M1754_Mutation_Testing]]
*[[CSC/ECE 517 Fall 2017/E1794. Student-generated questions added to rubric]]
*[[CSC/ECE 517 Fall 2017/E17A6. Fix account creation over web to work reasonably]]
*[[CSC/ECE 517 Fall 2017/E1792 OSS Visualizations for instructors]]
*[[CSC/ECE 517 Fall 2017/E17A4 Allow calibration to be part of an assignment_Team34]]
*[[CSC/ECE 517 Fall 2017/E17A7_Allow_Reviewers_to_bid_on_what_to_review]]
*[[CSC/ECE 517 Fall 2017/E17A9 Lazy loading (infinite scroll) for assignments courses questionnaires and user lists with Jscroll]]
*[[CSC/ECE 517 Fall 2017/E1796. Unify Create Assignment and Edit Assignment pages]]
*[[CSC/ECE 517 Fall 2017/E17A5 Allow users to create an account and submit work to an "assignment" (e.g., for conference reviewing)]]
*[[CSC/ECE 517 Fall 2017/E17A3 Upgrade review input UI and sanitize text input]]
*[[CSC/ECE 517 Fall 2017/E17AA Nomination for Badges]]
*[[CSC/ECE 517 Fall 2017/E1790 Text metrics]]

'''Writing Assignment 2'''
*[[CSC/ECE 517 Fall 2017/E1779. Fix teammate advertisements and requests to join a team]]
*[[CSC/ECE_517_Fall_2017/E1773 Investigate and Fix Expertiza Production Version Runtime Exceptions.rb]]
*[[CSC/ECE_517_Fall_2017/E1774 Metareview fixes and improvements.rb]]
*[[CSC/ECE_517_Fall_2017/E1788 OSS project Maroon Heatmap fixes]]
*[[CSC/ECE 517 Fall 2017/E1781 Topic Management]]
*[[CSC/ECE_517_Fall_2017/E1745_Refactor_response_controller.rb]]
*[[CSC/ECE_517_Fall_2017/E1752 Refactor assignments controller]]
*[[CSC/ECE_517_Fall_2017/E1776_Enhance_Imports]]
*[[CSC/ECE_517_Fall_2017/E1756 TLD Refactor response.rb]]
*[[CSC/ECE_517_Fall_2017/E1772_Refactor reputation_web_service_controller.rb]]
*[[CSC/ECE_517_Fall_2017/M1753_OffscreenCanvas API]]
*[[CSC/ECE_517_Fall_2017/M1754_Mutation Testing on Servo]]
*[[CSC/ECE_517_Fall_2017/E1753 OSS project bidding tests]]
*[[CSC/ECE_517_Fall_2017/E1766_Test team functionality]]
*[[CSC/ECE 517 Fall 2017/E1767 Improve imports]]
*[[CSC/ECE_517_Fall_2017/E1786_OSS project Juniper Bookmark Enhancements]]
*[[CSC/ECE_517_Fall_2017/E1787_OSS project Bronze Score calculations]]
*[[CSC/ECE_517_Fall_2017/E1763 Fix Staggered-Deadline Assignments]]
*[[CSC/ECE 517 Fall 2017/E1757 Introduce a Student View for instructors]]
*[[CSC/ECE 517 Fall 2017/E1769_Refactor assignment_form.rb]]
*[[CSC/ECE 517 Fall 2017/E1754_Feature_test_of_rubric_advice]]
*[[CSC/ECE 517 Fall 2017/E1782 OSS Project Red Assignment Directories]]
*[[CSC/ECE 517 Fall 2017/E1748 Add past-due assignments to task list]]
*[[CSC/ECE 517 Fall 2017/E1761 Test First Development Refactor assignment.rb]]
*[[CSC/ECE_517_Fall_2017/E1780_OSS_Project_Teal_Email_Notification_Enhancements]]
*[[CSC/ECE_517_Fall_2017/E1762 Test various kinds of response-map hierarchies]]
*[[CSC/ECE_517_Fall_2017/E1759 ]]
*[[CSC/ECE_517_Fall_2017/E1749_Test First Development Refactor questionnaire_controller.rb]]
*[[CSC/ECE 517 Fall 2017/E1777 Coherent specification of review requirements.rb]]
*[[CSC/ECE 517 Fall 2017/E1771_Refactor_team.rb]]
*[[CSC/ECE 517 Fall 2017/E1784 Fix mass assignments reported by Brakeman.rb]]

CSC/ECE 517 Fall 2017/E1773 Investigate and Fix Expertiza Production Version Runtime Exceptions.rb

2017-10-27T00:39:27Z

Chuang23: /* Exception 2. AbstractController::ActionNotFound */

==Introduction==
===Background===
[http://expertiza.ncsu.edu/ Expertiza]is a open-source ruby-on-rails project on [https://github.com/expertiza/expertiza github]. It constructs a peer-review system by enabling interactions among users and instructors. Students can sign up for a class, view assignments, submit assignments and give peer reviews using expertiza. Instructors can publish assignments, surveys, and reviews, view statistical results and make announcements. The website is created and currently maintained mainly by the students and faculties from NCSU.

===Motivation===
[https://airbrake.io/ Airbrake] is a online debugging tool for rails projects that is currently incorporated in Expertiza. Thus, Expertiza production errors at run time are tracked and reported statistically to Airbrake for reviewing and debugging.
Our goal for this OSS project is to fix [https://upload.wikimedia.org/wikipedia/commons/9/9c/10errors.png the top 10 run time exceptions] ranked by occurrences reported to Airbrake. You can check the 10 most ranked exceptions by click on the previous hyperlink.

==Airbrake Run-time Exceptions and Fixes==
By investigating into the top 10 errors from airbrake, we can divide them into different categories by the root of their causes. 

'''1. NoMethodError: undefined method for nil:NilClass (4 out of 10) '''
'''2. AbstractController::ActionNotFound (4 out of 10) '''
'''3. ActionController::InvalidAuthenticityToken (1 out of 10) '''
'''4. ActiveRecord::RecordNotFound: Couldn't find Participant without an ID (1 out of 10) '''

As we have discovered, the same fix can be used to fix each and every different category, which is how we categorized these exceptions initially.

===Files Modified===

/config/routes.rb

===Exception 1. NoMethodError: undefined method 'call' for nil:NilClass ===
From the Airbrake issue page, we could find out that the problem is actually caused by a gem called passenger. 
Although we couldn't find any sign of passenger gem in the recent master branch, but there is a clear sign we
found during our review of other errors that some developers had used passenger. 
The gem might be under the production path `/home/rails/.rvm/gems/ruby-2.1.5/gems/passenger-5.0.16/'
So the solution might be just to remove the gem passenger from production environment. 
For more information, please check [https://github.com/phusion/passenger/issues/1730 here]

''' NoMethodError: undefined method 'role' for nil:NilClass is one similar error that occurs during run-time '''

[[File:Error1.png|thumb|left|600px|pic1]][[File:Error11.png|600px|pic2]]
 

After investigating this error by going to the error causing url,as shown in pic1, given by airbrake under development environment,
http://localhost:3000/response/view?id=91251, the error is verified does not appear again. The last time this error occurred was 5
months ago.
Note, the link is only accessible to super admin account. As you may have noticed, the picture on the right shows the passenger gem
is called and traced, which we couldn't find such a gem installed in the current gemfile.

''' NoMethodError: undefined method 'uniq' for nil:NilClass is one similar error that occurs during run-time '''

Similarly, by going to the error cause url(https://expertiza.ncsu.edu/sign_up_sheet/set_priority) again under development environment, there is no such page exist.

===Exception 2. AbstractController::ActionNotFound===
In most of the cases, the error is caused by an abusive use of RESTful routing provided in rails.
In the middle of somewhere of the app, an index or show action is considered "called" when the redirect url matches such an action which is undefined in the controller.

By thoroughly investigating the controller in trouble and its routes, we came up with a solution that to either reroute the url to an effective url or abort the RESTful route.

For one example, in '''app/controllers/auth_controller''', we found no show/index action defined or needed, but in '''/config/routes.rb''', we have the following:

'''=begin'''
resources :auth , :except=>[:show,:index,:create,:destroy,:update,:edit,:new] do
collection do
post :login
post :logout
end
end
'''=end'''
By using resources:auth, rails actually creates index and show actions by default. Somewhere in the app, a redirect is made available to the user to an index or show view page which does not exist. Thus, in order to save the trouble finding the redirect, we simply redirect anything points to those invalid pages to a valid page, which is the root.The modified code is below:

'''post '/auth/login' ,to:"auth#login"'''
'''post '/auth/logout' ,to:"auth#logout"'''
'''get '/auth/*path', to: redirect('/')'''

Following the same principle, we have modified '''/config/routes.rb''' for routes that have caused the most exceptions due to exposure of invalid url redirection to the user.
resources :response''', :except =>[:index]'''do
collection do
get :new_feedback
get :view
post :delete
get :remove_hyperlink
get :saving
get :redirection
get :show_calibration_results_for_student
post :custom_create
get :pending_surveys
'''get '/*path', to: redirect('/response/pending_surveys')'''
end
end

There are 10 similar errors in Top 40 errors involved 5 controllers, which are ResponseController, AuthController, AdminController, ImpersonateController and PublishingController. If a controller mentioned above never uses a method declared by resources, we delete the relevant resources and add the routes of method separately, otherwise we use except expression to exclude methods we don't use.

===Exception 3. ActionController::InvalidAuthenticityToken===

Error Message: You submitted a form with an outdated or missing authenticity token. Try reloading the page you just submitted and submit the form again.

The error was raised in '''config/initializers/request_forgery_protection_mod.rb'''

And the error actually happened in '''response#create'''

The background of this problem is that The authenticity token is a random value
generated in your view to prove a request is submitted from a form on your site,
not somewhere else. This protects against CSRF attacks.

The expertiza problem could be the user session expired and the solution can be
delete the code in the '''config/initializers/request_forgery_protection_mod.rb''',
and redirect or just refresh the page, which would not raise error.

===Exception 4. RuntimeError: This topic has too many reviews; please select another one===

The error was raised in '''app/models/review_assignment.rb:171'''

And the error actually happened in '''review_mapping#assign_reviewer_dynamically'''

The intuition is that the review to a topic should be assigned dynamically and
"too many reviews" problem should be handled by action '''revew_mapping#assign_reviewer_dynamically'''.
So we could add the below condition in the above controller action to handle this problem.

Instead of raising an error, we could simply handle this problem, which can make
"flash danger message" unnecessary.