Expertiza_Wiki - User contributions [en]

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:32:17Z

Zhu6: /* Comparison */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [https://en.wikipedia.org/wiki/Supervised_learning supervised learning] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [https://en.wikipedia.org/wiki/Logistic_regression logistic regression model] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[https://en.wikipedia.org/wiki/AWS_S3 AWS S3] table

|Text file uploaded into
Google storage

Google Spreadsheets

[https://en.wikipedia.org/wiki/AWS_S3 HTTPS] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [https://docs.treasuredata.com/articles/hive HiveQL]
|-
|Data formats
|[https://en.wikipedia.org/wiki/CSV_file CSV file]
[https://en.wikipedia.org/wiki/Amazon_S3 S3] and [https://en.wikipedia.org/wiki/Amazon_Redshift Redshift] database
|Text file
Spreadsheet

[https://en.wikipedia.org/wiki/JSON JSON]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[https://en.wikipedia.org/wiki/Zip_(file_format) zip]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[https://en.wikipedia.org/wiki/HTTP_request HTTP request]: 2 MB
|10 GB
|-
|[https://en.wikipedia.org/wiki/Data_type Data types]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[https://en.wikipedia.org/wiki/Data_visualization Data visualization]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[https://en.wikipedia.org/wiki/Feature_extraction Feature extraction|Feature normalization]
|yes
|no
|yes using [https://en.wikipedia.org/wiki/Principal_component_analysis PCA]
|-
|[https://en.wikipedia.org/wiki/Orthonormalization Orthonormalization]
|no
|no
|yes
|-
|[https://en.wikipedia.org/wiki/Imputation_(statistics) Missing value imputation]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[https://en.wikipedia.org/wiki/Amazon_Web_ServicesAWS 3]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:31:50Z

Zhu6: /* Comparison */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [https://en.wikipedia.org/wiki/Supervised_learning supervised learning] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [https://en.wikipedia.org/wiki/Logistic_regression logistic regression model] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[https://en.wikipedia.org/wiki/AWS_S3 AWS S3] table

|Text file uploaded into
Google storage

Google Spreadsheets

[https://en.wikipedia.org/wiki/AWS_S3 HTTPS] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [https://docs.treasuredata.com/articles/hive HiveQL]
|-
|Data formats
|[https://en.wikipedia.org/wiki/CSV_file CSV file]
[https://en.wikipedia.org/wiki/Amazon_S3 S3] and [https://en.wikipedia.org/wiki/Amazon_Redshift Redshift] database
|Text file
Spreadsheet

[https://en.wikipedia.org/wiki/JSON JSON]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[https://en.wikipedia.org/wiki/Zip_(file_format) zip]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[https://en.wikipedia.org/wiki/HTTP_request HTTP request]: 2 MB
|10 GB
|-
|[https://en.wikipedia.org/wiki/Data_type Data types]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[https://en.wikipedia.org/wiki/Data_visualization Data visualization]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[https://en.wikipedia.org/wiki/Feature_extraction Feature extraction|Feature normalization]
|yes
|no
|yes using [https://en.wikipedia.org/wiki/Principal_component_analysis PCA]
|-
|[https://en.wikipedia.org/wiki/Orthonormalization Orthonormalization]
|no
|no
|yes
|-
|[https://en.wikipedia.org/wiki/Imputation_(statistics) Missing value imputation]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:28:30Z

Zhu6: /* Comparison */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [https://en.wikipedia.org/wiki/Supervised_learning supervised learning] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [https://en.wikipedia.org/wiki/Logistic_regression logistic regression model] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[https://en.wikipedia.org/wiki/AWS_S3 AWS S3] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:27:50Z

Zhu6: /* Azure Marketplace */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [https://en.wikipedia.org/wiki/Supervised_learning supervised learning] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [https://en.wikipedia.org/wiki/Logistic_regression logistic regression model] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:27:16Z

Zhu6: /* Azure Machine Learning Studio */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [https://en.wikipedia.org/wiki/Supervised_learning supervised learning] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:26:51Z

Zhu6: /* Build and deploy machine learning models in Azure Machine Learning */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [https://en.wikipedia.org/wiki/Supervised_learning supervised learning] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:26:23Z

Zhu6: /* Data Collection & Management */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [https://en.wikipedia.org/wiki/Relational_database relational data] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [https://en.wikipedia.org/wiki/Apache_Hadoop Hadoop] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:25:38Z

Zhu6: /* Introduction */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [https://en.wikipedia.org/wiki/Representational_state_transfer REST] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:25:17Z

Zhu6: /* Introduction */

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [https://en.wikipedia.org/wiki/Machine_learning_algorithms machine learning algorithms], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [https://en.wikipedia.org/wiki/R_(programming_language) R] and [https://en.wikipedia.org/wiki/Python_(programming_language) Python] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [https://en.wikipedia.org/wiki/Microsoft_Azure#Cloud_services Azure Cloud] as a scalable and [https://en.wikipedia.org/wiki/Fault_tolerance fault-tolerant] [https://en.wikipedia.org/wiki/Web_service web service] using Azure's Machine Learning [https://en.wikipedia.org/wiki/Application_programming_interface API]. The web service created by Machine Learning API are nothing but [[Representational state transfer|REST]] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:22:53Z

Zhu6:

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [[machine learning algorithms]], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [[R (programming language)|R]] and [[Python (programming language)|Python]] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [[Microsoft Azure#Cloud services|Azure Cloud]] as a scalable and [[Fault tolerance|fault-tolerant]] [[web service]] using Azure's Machine Learning [[Application programming interface|API]]. The web service created by Machine Learning API are nothing but [[Representational state transfer|REST]] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">"Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:21:58Z

Zhu6:

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [[machine learning algorithms]], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [[R (programming language)|R]] and [[Python (programming language)|Python]] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [[Microsoft Azure#Cloud services|Azure Cloud]] as a scalable and [[Fault tolerance|fault-tolerant]] [[web service]] using Azure's Machine Learning [[Application programming interface|API]]. The web service created by Machine Learning API are nothing but [[Representational state transfer|REST]] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">["Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"]</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:21:26Z

Zhu6:

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [[machine learning algorithms]], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [[R (programming language)|R]] and [[Python (programming language)|Python]] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [[Microsoft Azure#Cloud services|Azure Cloud]] as a scalable and [[Fault tolerance|fault-tolerant]] [[web service]] using Azure's Machine Learning [[Application programming interface|API]]. The web service created by Machine Learning API are nothing but [[Representational state transfer|REST]] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">[isbn = 978-1-4842-0446-7 "Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"]</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
<references/>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:20:59Z

Zhu6:

'''Azure Machine Learning''' (also known as Azure ML) is a fully [https://en.wikipedia.org/wiki/Cloud_computing cloud]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [https://en.wikipedia.org/wiki/Big_data big data] processing including creating, testing, operationalizing and managing [https://en.wikipedia.org/wiki/Predictive_analytics predictive analytics] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name="Introduction to machine learning on Microsoft Azure">[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/ "Introduction to machine learning on Microsoft Azure"]</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [[machine learning algorithms]], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [[R (programming language)|R]] and [[Python (programming language)|Python]] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [[Microsoft Azure#Cloud services|Azure Cloud]] as a scalable and [[Fault tolerance|fault-tolerant]] [[web service]] using Azure's Machine Learning [[Application programming interface|API]]. The web service created by Machine Learning API are nothing but [[Representational state transfer|REST]] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name="Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes">[isbn = 978-1-4842-0446-7 "Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes"]</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref name="Cortana Analytics Gallery">[https://gallery.cortanaanalytics.com/ "Cortana Analytics Gallery"]</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref name="Microsoft Azure Machine Learning Review - RobustTechHouse">[http://robusttechhouse.com/microsoft-azure-machine-learning-review/ "Microsoft Azure Machine Learning Review - RobustTechHouse"]</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref name="Machine Learning as a Service - Benchmark">[https://blog.onliquid.com/machine-learning-service-benchmark/ "Machine Learning as a Service - Benchmark"]</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref name="Amazon Simple Storage Service (S3) - Object Storage">[https://aws.amazon.com/s3/ "Amazon Simple Storage Service (S3) - Object Storage"]</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
</references>

Calibration Assignment Submission (Azure Machine Learning)

2016-09-09T04:13:01Z

Zhu6: Created page with "'''Azure Machine Learning''' (also known as Azure ML) is a fully cloud-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ e..."

'''Azure Machine Learning''' (also known as Azure ML) is a fully [[Cloud computing|cloud]]-based [http://www.sitscommunity360.com/what-does-end-to-end-it-support-actually-mean/ end-to-end service] by Microsoft for [[big data]] processing including creating, testing, operationalizing and managing [[predictive analytics]] models in the cloud. Azure ML is now a part of Microsoft's big data and advanced analytics offering '[https://www.microsoft.com/en-us/server-cloud/cortana-analytics-suite/overview.aspx Cortana Analytics Suite]'.<ref name=":0">{{Cite web|title = Introduction to machine learning on Microsoft Azure|url = https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-machine-learning/|website = azure.microsoft.com|access-date = 2016-02-06}}</ref>

==Introduction==

Azure ML offers features such as visual composition, large pallet of modules, an extensive library of starting templates and powerful [[machine learning algorithms]], a large collection of built-in transformation tasks, and support for frequently used data science programming languages like [[R (programming language)|R]] and [[Python (programming language)|Python]] which makes building common predictive analytics models quick and easy. After developing the model, one can deploy it on [[Microsoft Azure#Cloud services|Azure Cloud]] as a scalable and [[Fault tolerance|fault-tolerant]] [[web service]] using Azure's Machine Learning [[Application programming interface|API]]. The web service created by Machine Learning API are nothing but [[Representational state transfer|REST]] APIs that allows accessing the model from almost anywhere including web sites and customer applications, As patterns in data change over time, or if the user wants to add some new source of data, the deployed model can be easily retrained and updated programmatically through Azure ML API. Azure ML provides all such facilities through an interactive visual workspace called [https://studio.azureml.net/ Azure Machine Learning Studio.]<ref name=":0" />

==Azure Machine Learning workflow==

Azure ML has mainly three components in its workflow to build model and operationalize machine learning solution.

===Data Collection & Management===

It allows to use data through blobs and tables ([https://azure.microsoft.com/en-us/services/storage/ Azure Storage]), [[Relational database|relational data]] ([https://azure.microsoft.com/en-us/services/sql-database/ Azure SQL Database]), [[Apache Hadoop|Hadoop]] ([https://azure.microsoft.com/en-us/services/hdinsight/ Azure HDInsight]) and massive data stores ([https://azure.microsoft.com/en-us/solutions/data-lake/ Azure Data Lake]).

===Machine Learning Service===

It allows to create Machine learning models through ML Studio web apps and output a web service that can run on a scheduled basis.

===Embedded ML Model===

It allows to embed the developed machine learning models into other applications such as apps, websites or business intelligence tools.

===Build and deploy machine learning models in Azure Machine Learning===

Azure Machine Learning has many different modules to help you build and deploy machine learning models in production. Following are the key steps to build and deploy a machine learning model in Azure<ref name=":1" />
# Import raw data
# Preprocess the data
# Do feature engineering and data labeling (for [[supervised learning]] such as classification)
# Train, score, and evaluate the model
# Model comparison and selection
# Save the trained model
# Create a predictive experiment
# Publish the web service in Azure Machine Learning

==Azure Machine Learning Studio==

[https://azure.microsoft.com/en-us/documentation/articles/machine-learning-what-is-ml-studio/ Azure Machine Learning Studio] is an interactive, visual workspace to build, test, and iterate on a [[predictive analytics]] model. Following are the components of the Machine Learning Studio:<ref name=":1" />

===Experiments===
It provides a list of experiments that have been created, run, and saved as drafts. It also includes a set of sample experiments to help jumpstart the projects.

===Web Services===
It provides a list of experiments that are already published by the user. If no experiments have been published, this list will be empty.

===Datasets===
It provides a list of sample datasets that ship with the product along with datasets uploaded by the user.

===Trained Models===
It provides a list of any trained models that the user has saved from experiments.

===Settings===
It includes a collection of settings so as to configure resources and account.

In Machine Learning Studio, the user can construct a predictive model by dragging and dropping datasets and analysis modules onto the interactive canvas, connecting them together with lines to show the flow of data and parameters through the workflow to form an experiment, which can be run in ML Studio. Thus, each experiment is complete workflow with all components required to build, test, and evaluate a predictive analytics model. One can iteratively update the model design by editing the experiment, saving the copy and running it again.

==Cortana Analytics Gallery==

Cortana Analytics [https://gallery.cortanaanalytics.com/ (formerly called as Machine Learning Gallery)] provides a gallery of sample machine learning models that one can use to quickly get started. Each model contains an experiment added by Microsoft, its partners or individual data scientists. <ref name=":1">{{Cite book|title = Predictive Analytics with Microsoft Azure Machine Learning: Build and Deploy Actionable Solutions in Minutes|last = |first = |publisher = |year = |isbn = 978-1-4842-0446-7|location = |pages = }}</ref> It also serves as a platform for users looking to learn from others, to get started developing their own solutions, or to contribute their own work to the advanced analytics community<ref>{{Cite web|title = Cortana Analytics Gallery|url = https://gallery.cortanaanalytics.com/|website = gallery.cortanaanalytics.com|access-date = 2016-02-16}}</ref>.

==Azure Marketplace==

Machine Learning Studio enables a user to configure Machine Learning models using graphical user interfaces. [https://azure.microsoft.com/en-us/marketplace/ Azure Marketplace], on the other hand, offers pre-packaged models that are ready to use. Some of the services that the Marketplace has to offer are listed below:<ref>{{Cite web|title = Microsoft Azure Machine Learning Review - RobustTechHouse|url = http://robusttechhouse.com/microsoft-azure-machine-learning-review/|website = RobustTechHouse|access-date = 2016-02-05|language = en-US}}</ref>

[https://datamarket.azure.com/dataset/amla/text-analytics Text Analytics]

The Text Analytics API is a suite of text analytics services built using Azure Machine Learning. It can be used to detect sentiments, extract keyphrases and detect the language of the text provided.

[https://datamarket.azure.com/dataset/amla/mba Frequently brought together]

This API can be used by website owners to help its customers discover items in its catalog that are bought together. It uses customer purchase history to provide "Frequently bought together" recommendations.

[https://datamarket.azure.com/dataset/aml_labs/log_regression Binary classifier API]

Binary Classifier API is an example built with Microsoft Azure Machine Learning that fits a [[Logistic regression|logistic regression model]] to data input by the user and then outputs the predicted value for each of the observations in the data.

==Comparison==
The summary comparison between various machine learning platforms has been summarized below<ref>{{Cite web|title = Machine Learning as a Service - Benchmark|url = https://blog.onliquid.com/machine-learning-service-benchmark/|website = Liquid|access-date = 2016-02-09}}</ref>
{| class="wikitable"
!
!AWS machine learning
!Google Prediction API
!MS Azure Machine Learning
|-
|Data sources
|Text file uploaded into S3
[https://aws.amazon.com/rds/ AWS RDS]

[https://aws.amazon.com/redshift/ AWS Redshift]

[[AWS S3]] table

|Text file uploaded into
Google storage

Google Spreadsheets

[[HTTPS]] requests

API update calls
|Uploaded text files
Azure storage

SQL database

Web URL

Hadoop [[HiveQL]]
|-
|Data formats
|[[CSV file]]
[[Amazon S3|S3]] and [[Amazon Redshift|Redshift]] database
|Text file
Spreadsheet

[[JSON]]
|CSV and text files
[https://docs.treasuredata.com/articles/hive Hive SQL] tables

[https://msdn.microsoft.com/en-us/data/hh237663.aspx Odata values]

[http://svmlight.joachims.org/ svmlight]

[http://www.cs.waikato.ac.nz/ml/weka/arff.html arff]

[[Zip (file format)|zip]]

[http://fileinfo.com/extension/rdata RData]
|-
|Dataset maximum size
|100GB
|text file: 2.5 GB
[[HTTP request]]: 2 MB
|10 GB
|-
|[[Data type|Data types]]
|boolean
categorical

numeric

string

|numeric
string

|boolean
categorical

datetime

numeric

timespan

string
|-
|[[Data visualization]]
|yes
table view
|no
|Table, histogram,
statistical summary
|-
|Mathematical transformations
|no
|no
|yes
|-
|[[Feature extraction|Feature normalization]]
|yes
|no
|yes using [[Principal component analysis|PCA]]
|-
|[[Orthonormalization]]
|no
|no
|yes
|-
|[[Imputation (statistics)|Missing value imputation]]
|yes indirectly chain feature
imputation model with main

prediction model
|yes automatic replaces missing
strings with ""

replaces missing numbers

with 0
|yes can replace with custom
values mean, median or mode.
|}
Terms used in the above comparison

[[Amazon Web Services|AWS 3]]: Amazon Simple Storage Service (Amazon S3), provides developers and IT teams with secure, durable, highly-scalable object storage<ref>{{Cite web|title = Amazon Simple Storage Service (S3) - Object Storage|url = https://aws.amazon.com/s3/|website = Amazon Web Services, Inc.|access-date = 2016-02-16}}</ref>.

Azure Machine Learning provides more flexibility in terms of the data formats that can be imported into its environment. It also supports more data visualization methods as compared to Amazon Machine Learning. The missing value imputaion can also be customized.

== Further reading ==
* [https://azure.microsoft.com/en-us/documentation/articles/machine-learning-create-experiment/ Tutorial on generating a linear regression model for predicting automobile prices]
* [http://cloudacademy.com/blog/azure-machine-learning/ CloudAcademy blog]
* [http://www.kdnuggets.com/2014/11/microsoft-azure-machine-learning.html Why Azure ML is the next big thing in Machine Learning (kdnuggets blog)]

== Videos ==
* [https://www.youtube.com/watch?v=kZ04LnSjWek Introducing Microsoft Azure Machine Learning(YouTube video)]

* [https://www.youtube.com/watch?v=GPXvtoIAbOg How to train a neural network using Azure Machine Learning]

== References ==
</references>

CSC/ECE 517 Spring 2016

2016-09-09T04:12:41Z

Zhu6: /* Calibration Assignment Submissions */

==Writing Assignment 1==
*[[CSC/ECE 517 Spring 2016/Active Job]]

==Calibration Assignment Submissions==
*[[Calibration Assignment Submission (OmniAuth)]]
*[[Calibration Assignment Submission (Patch_verb)]]
*[[Calibration Assignment Submission (Azure_Machine_Learning)]]

==Writing Assignment 2==
*[[CSC/ECE 517 Spring 2016/Mozilla Implement HTML5 form validation]]

*[[CSC/ECE_517_Spring_2016/Mozilla Conforming Image Loading to HTML 5 Specs]]

*[[CSC/ECE 517 Spring 2016/Functional Tests for Questionnaire Controller]]

*[[CSC/ECE 517 Spring 2016 / Expertiza Self-Review Feature]]

*[[CSC/ECE 517 Spring 2016/Refactor different question types from quiz feature]]

*[[CSC/ECE 517 Spring 2016/Implement private browsing]]

*[[CSC/ECE 517 Spring 2016/Write automated tests for WebDriver]]

*[[CSC/ECE 517 Spring 2016/Refactor sign_up_sheet_controller.rb and sign_up_topic.rb]]

*[[CSC/ECE 517 Spring 2016/Implement Common Parts of the CSSOM API]]

*[[CSC/ECE 517 Spring 2016/Functional tests for assignment creation function]]

*[[CSC/ECE 517 Spring 2016/E1604. Functional tests for Calibration function]]

*[[CSC/ECE 517 Spring 2016/Refactor and write unit tests for question type.rb]]

*[[CSC/ECE 517 Spring 2016/Refactor response controller]]

*[[CSC/ECE_517_Spring_2016_OSS_M1606]]

*[[CSC/ECE_517_Spring_2016/OSS_E1601]]

==Final Project Design Document==
*[[CSC/ECE 517 Spring 2016 M1601 Implement HTML5 form validation]]
*[[CSC/ECE 517 Spring 2016 M1602 Make image loads conform with the spec]]
*[[CSC/ECE 517 Spring 2016 E1624 Compose functional tests for Suggest topics of student functionality ]]
*[[CSC/ECE 517 Spring 2016 E1631 Team-based reviewing]]
*[[CSC/ECE 517 Spring 2016 E1628 Visualization]]
*[[CSC/ECE 517 Spring 2016 E1625 Show confidence ratings for grade based on reputations of reviewers]]
*[[CSC/ECE 517 Spring 2016 E1629 Performance improvement for Course & Assignment Listing]]
*[[CSC/ECE 517 Spring 2016 M1603 Implement Common Parts of the CSSOM API]]
*[[M1606: Implementing HTTP authorization UI and persistent sessions]]
*[[CSC/ECE 517 Spring 2016 E1620 Improve Survey Functionality]]
*[[CSC/ECE 517 Spring 2016 E1621 Refactor and Test the Quizzing Feature]]

Calibration Assignment Submission (Active Job)

2016-09-08T19:32:07Z

Zhu6: Created page with "== '''Introduction''' == Active Job is a framework that helps developers to write codes and run them on the background automatically under different scenarios. It’s an interfac..."

== '''Introduction''' ==
Active Job is a framework that helps developers to write codes and run them on the background automatically under different scenarios. It’s an interface that adapts various queueing backends like [https://github.com/nesquena/backburner Backburner], [https://github.com/collectiveidea/delayed_job Delayed Job], [https://github.com/bkeepers/qu Qu] and so on. Jobs can vary from schedule newsletters, follow-up emails to database housekeeping. <ref name= "Active-Job-Basics">Rails Guides [http://guides.rubyonrails.org/active_job_basics.html “Active Job Basics” ]</ref> Overall, Active Job is an interface which you can work with common queueing backends.
The reason to have Active Job in place is to make sure Rails apps can have a job infrastructure. Other existing gems can build on this framework without the limitation of API difference between different job runners such as Delayed Job and Resque. With this feature choosing queueing backends can become an operational concern. Also, switching between those queueing backends without rewrite jobs become possible.<ref name= "Active-Job-Basics">Rails Guides [http://guides.rubyonrails.org/active_job_basics.html “Active Job Basics” ]</ref>

__TOC__

====Ruby On Rails versions support Active Job====
{| class="wikitable sortable" style="float:left;"
|+ Version history
|-
! Version !! Date
|-
| 4.2 || 2014/12/19
|-
| 4.2.5 || 2015/11/13
|-
|}
 
====Active Job adapters<ref name="Active-Job-Adapter">Active Job Adapter[http://edgeapi.rubyonrails.org/classes/ActiveJob/QueueAdapters.html"Active Job Adapter" ]</ref>====
List of queueing backends Active Job support:
*Backburner[https://github.com/nesquena/backburner]
*Delayed Job[https://github.com/collectiveidea/delayed_job]
*Qu[https://github.com/bkeepers/qu]
*Que[https://github.com/chanks/que]
*queue_classic[https://github.com/QueueClassic/queue_classic]
*Resque 1.x[https://github.com/resque/resque]
*Sidekiq[http://sidekiq.org/]
*Sneakers[https://github.com/jondot/sneakers]
*Sucker Punch[https://github.com/brandonhilkert/sucker_punch]
*Active Job Async Job
*Active Job Inline[http://edgeapi.rubyonrails.org/classes/ActiveJob/QueueAdapters/InlineAdapter.html]

=='''How to use <ref>http://guides.rubyonrails.org/active_job_basics.html</ref>'''==
This chapter is an introduction on how to create a job and and how to add the job into a queue. <ref name= "Active-Job-Basics">Rails Guides [http://guides.rubyonrails.org/active_job_basics.html “Active Job Basics” 2014]</ref> 
=== Download ===
With RubyGems[https://en.wikipedia.org/wiki/RubyGems] you can install Active Job:
<pre>
$ gem install activejob
</pre>

[https://github.com/rails/rails/tree/master/activejob Souce code of Active Job] available on GitHub[https://en.wikipedia.org/wiki/GitHub], as part of Rails.<ref> GitHub [https://github.com/rails/rails/tree/master/activejob “Active Job -- Make work happen later” 2015] </ref>

=== Create a Job ===
In Active Job, a process inserted in a queue and waiting to carry out is called “Job”. It’s possible to generate a Job using the Generator provided by Rails. You can create a Job in app/jobs. By doing the following, we created a Job called “update_wiki”. 
<pre>
$ bin/rails generate job update_wiki
invoke test_unit
create test/jobs/update_wiki_job_test.rb
create app/jobs/update_wiki_job.rb
</pre>
Active Job provides the ability to run your Job on a specific queue by creating a job: 
<pre>
$ bin/rails generate job update_wiki --queue urgent
</pre>
Files inside of app/jobs can be created manually, instead of using a generator. In Rails 4.2 an ActiveJob class inherits from ActiveJob::Base. In Rails 5.0, it has changed to now inherit from ApplicationJob. When upgrading from Rails 4.2 to Rails 5.0, an application_job.rb file is needed to be created in app/jobs/ and add the following content:<ref> Rails Guides [http://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html#upgrading-from-rails-4-1-to-rails-4-2 “A Guide for Upgrading Ruby on Rails” 2015] </ref>
<pre>
class ApplicationJob < ActiveJob::Base
end
</pre>
In Rails 4.2 a Job class defined a perform method and set a “queue_as” value:
<pre>
class UpdateWikiJob < ActiveJob::Base
queue_as :default

def perform(*wiki)
# Do something later
wiki.update_contents
end
end
</pre>
There is a perform method to be called when the Job was first enqueued. 

=== Adding a Job to the queue ===
Enqueue the Job so it can be processed as soon as the queuing system is free:
<pre>
UpdateWikiJob.perform_later wiki
</pre>
Or add a Job be performed tomorrow at noon:
<pre>
UpdateWikiJob.set(wait_until: Date.tomorrow.noon).perform_later(wiki)
</pre>
Most of the queueing backends ( Sidekiq
, Delayed Job, etc. ) allow you to set a delay time.
<pre>
UpdateWikiJob.set(wait: 1.week).perform_later(wiki)
</pre>
=== Execution of Job ===
Active Job provides adapters for multiple queueing backends ([https://github.com/mperham/sidekiq/wiki Sidekiq], [https://github.com/resque/resque Resque], [https://github.com/collectiveidea/delayed_job Delayed Job] and [http://edgeapi.rubyonrails.org/classes/ActiveJob/QueueAdapters.html others]).<ref name= "Active-Job-Basics">Rails Guides [http://guides.rubyonrails.org/active_job_basics.html “Active Job Basics” 2014]</ref> Without setting any adapter, the job would be performed immediately. 
Queueing backend can be set at: /config/application.rb, this example uses the Sidekiq.
<pre>
module YourApp
class Application < Rails::Application
# Be sure to have the adapter's gem in your Gemfile and follow
# the adapter's specific installation and deployment instructions.
config.active_job.queue_adapter = :sidekiq
end
end
</pre>
=== Play with Queues ===
Active Job allows to schedule the job to be processed on a specific queue, this became helpful as common adapters support multiple queues.
<pre>
class UpdateWikiJob < ActiveJob::Base
queue_as :low_priority
#....
end
</pre>
Queue name can be prefixed for all jobs using config.active_job.queue_name_prefixin application.rb:
<pre>
# config/application.rb
module YourApp
class Application < Rails::Application
config.active_job.queue_name_prefix = Rails.env
end
end

# app/jobs/update_wiki_job.rb
class UpdateWikiJob < ActiveJob::Base
queue_as :low_priority
#....
end

# Now your job will run on queue production_low_priority on your
# production environment and on staging_low_priority on your staging
# environment
</pre>

Control a job to run on a queue by passing a :queue option to #set :
<pre>
MyJob.set(queue: :another_queue).perform_later(record)
</pre>

All the code snippets are referred from the source<ref name= "Active-Job-Basics">Rails Guides [http://guides.rubyonrails.org/active_job_basics.html “Active Job Basics” 2014]</ref>

== Examples ==
In this section, a example is provided to demonstrate how to use ActiveJob in practice.
=== Background Mail Sender ===
Send emails asynchronously with Action Mailer which Active Job is already integrated in.<ref name= "Action-Mailer-Basics">Rails Guides [http://guides.rubyonrails.org/action_mailer_basics.html “Action Mailer Basics” 2014]</ref> Here in this example, we try to use Active Job with Action Mailer. 
==== 1. Getting started with Rails<ref name= "Welcome_to_Rails">GitHub [https://github.com/rails/rails “Welcome to Rails” 2015]</ref> ====
<pre>
# Install Rails if you haven’t done it yet:
$ gem install rails

# Create a new Rails application, “myapp_activejob”
# is the name of the application:
$ rails new myapp_activejob

# Change your directory to myapp_activejob
$ cd myapp_activejob
</pre>
Edit the Gemfile in the folder, add following into it:
<pre>
gem 'responders'
</pre>
Run “ $ bundle update “ and make sure “$ rails server” works.

==== 2. Resque setup<ref name= "Download_Redis">Redis [http://redis.io/download “Download Redis” 2016]</ref>====
This example uses [https://github.com/resque/resque Resque] as the enqueuing backend. It'll need [http://redis.io/ Redis] before running Resque. Redis can be installed using [http://brew.sh/ Homebrew]:
<pre>
$ brew install redis
</pre>
Or download, extract and compile Redis using:<ref name= "Download_Redis">Redis [http://redis.io/download “Download Redis” 2016]</ref>
<pre>
$ wget http://download.redis.io/releases/redis-3.0.7.tar.gz
$ tar xzf redis-3.0.7.tar.gz
$ cd redis-3.0.7
$ make
</pre>
Redis does not officially support Microsoft [https://en.wikipedia.org/wiki/Microsoft_Windows Windows] platform, but it can be found [https://github.com/MSOpenTech/redis Redis on Windows] by Microsoft Open Tech group. 
Next get Resque be installed. To use resque with Active Job, we also need resque-scheduler.
Add the following into Gemfile, and run “$ bundle install”.
<pre>
gem 'resque'
gem 'resque-scheduler'
</pre>
After the installation, create a Resque configuration file resque.rb in config/initializers/
<pre>
#config/initializers/resque.rb

Resque.redis = Redis.new(host: 'localhost', post: 6379)
Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }
</pre>

As the Resque tasks and Resque Scheduler rake tasks are needed in this example, we need to create a resque.rake file in lib/tasks/:

<pre>
#lib/tasks/resque.rake

require 'resque/tasks'
require 'resque/scheduler/tasks'

namespace :resque do
task setup: :environment do
ENV['TERM_CHILD'] ||= '1'
ENV['QUEUE'] ||= '*'
require 'resque'
require 'resque-scheduler'
end
end
</pre>

==== 3. Creating a Mailer<ref name= "Action-Mailer-Basics">Rails Guides [http://guides.rubyonrails.org/action_mailer_basics.html “Action Mailer Basics” 2014]</ref> ====
By doing the things above, we already have our Rails project and the Resque for queuing get set. Next we will create a Mailer.

<pre>
$ bin/rails generate mailer user_mailer
</pre>

Create a method and a view for sending the email. 
In app/mailer/user_mailer.rb:

<pre>
#app/mailers/user_mailer.rb

class UserMailer < ActionMailer::Base
default from: 'from@example.com'

def test_email(email)
mail(
to: email,
subject: 'We are testing Active Job!'
)
end
end
</pre>
In app/views/user_mailer/test_email.text
<pre>
#app/views/user_mailer/test_email.text

Hey, we are testing Active Job!
</pre>

==== 4. Active Job<ref name= "Active-Job-Basics">Rails Guides [http://guides.rubyonrails.org/active_job_basics.html “Active Job Basics” 2014]</ref>====
Create a configuration file active_job.rb in config/initializers/ , to set Resque as queue_adapter.
<pre>
#config/initializers/active_job.rb

ActiveJob::Base.queue_adapter = :resque
</pre>

Create a Job with generator.
<pre>
$ bin/rails generate job test_email
</pre>

In the generated file app/jobs/test_email_job.rb, define the perform method and set it’s queue_as.

<pre>
#app/jobs/test_email_job.rb

class TestEmailJob < ActiveJob::Base
queue_as :email

def perform(email)
UserMailer.test_email(email).deliver_now
end
end
</pre>

Now a UserController is in place to put the job into the queue for later execution. Set the email to be sent one minute later for testing.
<pre>
$ bin/rails generate job test_email
</pre>

<pre>
#app/controllers/users_controller.rb

class UsersController < ApplicationController
def new
@user = User.new
end

def create
@user = User.create(user_params)
TestEmailJob.new(@user.email).deliver_later!(wait: 1.minute)
# redirect somewhere
end
end
</pre>

Routes and views need to be set in order to make it work.
<pre>
#config/routes.rb

Rails.application.routes.draw do
resources :users, only: [:new, :create]
end
</pre>

<pre>
#app/views/users/new.html.erb

<%= form_for @user do |f| %>
<%= f.email_field :email %>
<%= f.submit %>
<% end %>
</pre>

==== 5. Try execution ====
Before trying the Mailer, MailCatcher[https://rubygems.org/gems/mailcatcher/versions/0.6.1] is needed for the test. Add following into Gemfile and “$ bundle install”
<pre>
gem 'mailcatcher'
</pre>

To set the environment, one need to add the following into config/environments/development.rb.
<pre>
#config/environments/development.rb

Rails.application.configure do
...
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = { address: "localhost", port: 1025 }
end
</pre>

Run it!
<pre>
# Start mailcatcher
$ mailcatcher

# Start redis-server
$ redis-server

# Start resque:work
$ bundle exec rake resque:work

# Start resque:scheduler
$ rake environment resque:scheduler

# Finally start rails server
$ rails server
</pre>

Open the browser and visit localhost:3000/users/new. Then sign up as a new user. One minute later Resque scheduler has following output:

<pre>
resque-scheduler: [INFO] 2016-02-05T22:53:15+09:00: Processing Delayed Items
</pre>
And in MailCatcher:
<pre>
==> SMTP: Received message from '<from@example.com>' (315 bytes)
</pre>

=='''External links'''==

*[http://www.rubyonrails.org/ Ruby on Rails official website] – rubyonrails.org
*[http://railscasts.com/ Railscasts.com] – (defunct) Short Screencasts that focus on Ruby on Rails technique
*[http://rubycasts.io/ Rubycasts.io] – Weekly 15 minute screencasts on the whole Ruby/Rails development stack
*[http://www.railstutorial.org/ The Rails Tutorial] – The Ruby on Rails Tutorial book and screencast series teach you how to develop and deploy Ruby on Rails apps
*[https://github.com/rails/rails Rails source code]

=='''References'''==
<references/>

CSC/ECE 517 Fall 2016

2016-09-08T19:29:27Z

Zhu6: Created page with "==Calibration Assignment Submissions== *Calibration Assignment Submission (Firebrick JS) *Calibration Assignment Submission (Active Job)"

==Calibration Assignment Submissions==
*[[Calibration Assignment Submission (Firebrick JS)]]
*[[Calibration Assignment Submission (Active Job)]]

MainPage

2016-09-08T19:28:31Z

Zhu6: /* Course-Specific Topics */

==Expertiza==
* [[Expertiza documentation]]

==Course-Specific Topics==

* [[CSC/ECE 517 Summer 2008]]
* [[CSC/ECE 517 Fall 2010]]
* [[CSC/ECE 517 Fall 2011]]
* [[CSC/ECE 517 Fall 2012]]
* [[CSC/ECE 517 Fall 2013]]
* [[CSC/ECE 517 Fall 2014]]
* [[CSC/ECE 517 Fall 2015]]
* [[CSC/ECE 517 Fall 2016]]
* [[CSC/ECE 517 Spring 2014]]
* [[CSC/ECE 517 Spring 2015]]
* [[CSC/ECE 517 Spring 2016]]
* [[CSC 456 Spring 2011|CSC 456 Spring 2012]]
* [[ECE 633]]
* [[KCU]]
* [[Progress reports]]

==Application Behavior==
* [[Grading]]

==Metaprogramming==
* [[CSC/ECE_517_Spring_2013/ch1b_1k_hf|Lecture on Metaprogramming]]

==Development==

''Expertiza now has a Java dependency, so the machine you are using to develop Expertiza on should have the JVM installed.''

* [[Setting Up a Development Machine]]
* [[Creating a Linux Development Environment for Expertiza - Installation Guide]]
* [[Using git and github for projects]]
* [[Using heroku to deploy your projects]]
* [[How to Begin a Project from the Current Expertiza Repository]]
* [[Git]]
* [[How to Change a User's Password on a Development Machine]]
* [[Debugging Rails]]

==Production==
* [[Deploying to Production]]
* [[Downloading Production Data]]
* [[Accessing the Production Server]]

==Testing==
* [[Using Cucumber with Expertiza]]
* [[Rails Testing Overview]]
* [[Expertiza Continuous Integration]]

==References==
* [[Object-Oriented Design and Programming]]

Calibration Assignment Submission (Patch verb)

2016-01-26T20:38:22Z

Zhu6: Created page with "The '''PATCH''' method is a request method supported by the [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] protocol for making partial changes to an existing re..."

The '''PATCH''' method is a request method supported by the [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] protocol for making partial changes to an existing resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref> The PATCH method provides an entity containing a list of changes to be applied to the resource requested using the [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] [https://en.wikipedia.org/wiki/Uniform_Resource_Identifier URI].<ref name="RFC5789"/> The list of changes are supplied in the form of a PATCH document.<ref name="RFC5789"/> If the requested resource does not exist then the [https://en.wikipedia.org/wiki/Server_(computing) Server (computing)] may create the resource depending on the PATCH document media type and permissions.<ref name="RFC5789"/> The changes described in the PATCH document must be semantically well defined but can have a different media type than the resource being patched.<ref>Don't Patch Like An Idiot http://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/</ref> Frameworks such as [https://en.wikipedia.org/wiki/XML XML], [https://en.wikipedia.org/wiki/JSON JSON] can be used in describing the changes in the PATCH document.

==PUT vs PATCH vs POST==

[https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] is the foundation of data communication for the [https://en.wikipedia.org/wiki/World_Wide_Web World Wide Web]. [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] is a request-response protocol which helps users to communicate with the server to perform [https://en.wikipedia.org/wiki/Create,_read,_update_and_delete CRUD] operations. [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] supports a number of request methods such as PUT, [https://en.wikipedia.org/wiki/POST_(HTTP) POST] and PATCH to create or update resources.<ref name=RFC2616>Hypertext Transfer Protocol -- HTTP/1.1 https://www.ietf.org/rfc/rfc2616.txt</ref>

The main difference between the PUT method and PATCH method is that the PUT method uses the request [https://en.wikipedia.org/wiki/Uniform_Resource_Identifier URI] to supply a modified version of the requested resource which replaces the original version of the requested resource whereas the PATCH method supplies a set of instructions to modify the requested resource. If the PATCH document is larger than the size of the new version of the resource sent by the PUT method then the PUT method is preferred.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789|accessdate=2015-09-12}}</ref>

The [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method can be used for sending partial updates to a resource. The main difference between the [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method and the PATCH method is that the [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method can only be used when it is written to support the applications or the applications support its semantics whereas the PATCH method can be used in a generic way and does not require application support. If the outcome of using the PATCH method is not known then the [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method is preferred.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref> <ref>Why PATCH is Good for Your HTTP API https://www.mnot.net/blog/2012/09/05/patch</ref>

==Patching resources==

The PATCH method is [https://en.wikipedia.org/wiki/Atomicity_(database_systems) atomic].<ref name="RFC5789"/> Either all the changes specified by the PATCH method are applied or none of the changes are applied by the server.<ref name="RFC5789"/> There are many ways of checking whether a patch was applied successfully. For example, the 'diff' utility can be applied to the older version and newer version of a file to find the differences between them.<ref name="RFC5789"/>

A cached PATCH response is considered stale. It can only be used for the GET and HEAD requests that may follow the PATCH request.<ref name="RFC5789"/>

The entity headers in the PATCH document are only applicable to the PATCH document and cannot be applied to the requested resource.<ref name="RFC5789"/>

There is no standard format for the PATCH document and it is different for different types of resources. The server has to check whether the PATCH document received is appropriate for the requested resource.<ref name="RFC5789"/>

A '''JSON''' PATCH document would look like

<code lang="json">
{ "op": "add", "variable": "count", "value": 1 }
</code>

"op" represents the operation performed on the resource. "count" represents the resource being modified. "value" represents the amount being added to the existing resource. <ref name=RFCJSONPATCH>JSON Patch - draft-ietf-appsawg-json-patch-08 https://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08</ref> Before applying the changes in the PATCH document, the server has to check whether the PATCH document received is appropriate for the requested resource. If the PATCH request succeeds then it returns a '''200''' response.

==Caution==

The PATCH method is not safe. It may create new resources or change resources not mentioned in the [https://en.wikipedia.org/wiki/Uniform_Resource_Identifier URI].<ref name="RFC5789"/>

The PATCH method is not idempotent. It can be made idempotent by using a conditional request.<ref name=RFC5789 />When a client makes a conditional request to a resource, the request succeeds only if the resource has not been updated since the client last accessed that resource. This also helps in preventing corruption of the resource since some updates to a resource can only be performed starting from a certain base point.<ref name="RFC5789"/>

==Error handling==

A PATCH request can fail if any of the following errors occur:
===Malformed patch document===
The server returns a 400 (Bad request) response if the PATCH document is not formatted as required.<ref name="RFC5789">PATCH Method for HTTP|url=http://tools.ietf.org/html/rfc5789</ref>

===Unsupported patch document===
The server returns a 415 (Unsupported Media Type) response with an Accept-Patch response header containing supported media types when the client sends an unsupported patch document. This informs the client that the PATCH document sent by the client cannot be applied to the requested resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Unprocessable request===
The server returns a 422 (Unprocessable Entity) response when the server is able to understand the PATCH document but is unable to modify the requested resource either because it causes the resource to become invalid or it results in some other error state.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Resource not found===
The server returns a 404 (Not Found) response when the PATCH document cannot be applied to a non-existent resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Conflicting state===
The server returns a 409 (Conflict) response when the server cannot apply a patch for the current state of the resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Conflicting modification===
The server returns a 412 (Precondition Failed) response when the precondition supplied by the client using the If-Match or If-Unmodified-Since header fails. If no precondition is supplied and there is a conflicting modification then the server returns a 409 (Conflict) response.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Concurrent modification===
The server returns a 409 (Conflict) response if the PATCH requests to a certain resource need to be applied in a certain order and the server is not able to handle concurrent PATCH requests.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

==Trade-offs between PUT and PATCH==

When only a few changes need to be applied to a resource, using the PUT method consumes more bandwidth as compared to the PATCH method which can be used for sending only the changes. But when the PATCH method is used, it usually involves fetching the resource from the server, comparing the original and new files, creating and sending a diff file. On the server side,the server has to read the diff file and make the modifications. This involves a lot of work overhead compared to PUT method.<ref name=51elliot>REST API Best Practices 3: Partial Updates - PATCH vs PUT http://51elliot.blogspot.com/2014/05/rest-api-best-practices-3-partial.html</ref>
On the other hand, the PUT method requires a GET to be performed before the PUT and it is difficult to ensure that the resource was not modified between the GET and PUT requests.

==Security Considerations==

The PATCH request needs to use mechanisms such as conditional requests using Etags and the If-Match request header to ensure that data is not corrupted while patching.<ref name=RFC5789 /> In case of a failure of a PATCH request or failure of the channel or a timeout, the client can use a GET request to check the state of the resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref> The server has to ensure that malicious clients do not use the PATCH method for consuming excessive server resources.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

==References==

<references/>

Calibration Assignment Submission (OmniAuth)

2016-01-26T20:37:51Z

Zhu6: Created page with "OmniAuth is an authentication library written in [https://www.ruby-lang.org/en/ Ruby] under MIT License<ref name="home">Home Page https://opensource.org/licenses/MIT</ref>. OmniA..."

OmniAuth is an authentication library written in [https://www.ruby-lang.org/en/ Ruby] under MIT License<ref name="home">Home Page https://opensource.org/licenses/MIT</ref>. OmniAuth is used to enable third party authentication in [https://en.wikipedia.org/wiki/Ruby_on_Rails Ruby on Rails] applications. As of September 2015, OpenAuth supported authentication via 218 specific service providers<ref name="los">List of Strategies https://github.com/intridea/omniauth/wiki/List-of-Strategies</ref>. In addition to the specific service providers, 13 generic authentication plugins are also provided.<ref name="los"/>

==History==
While working a Intridea, Micheal Bleigh started working on OmniAuth in 2008<ref name="home"/><ref>Micheal Bleigh http://www.intridea.com/about/team/michael-bleigh</ref>. OmniAuth version 1.0 was released in November of 2011.<ref>Bleigh, Michael (2 November 2011) http://www.intridea.com/blog/2011/11/2/omniauth-1-0</ref> The latest release of OmniAuth, version 1.2.2, was tagged on July 9, 2014<ref>omniauth Releases https://github.com/intridea/omniauth/releases</ref>.

==Strategies==
OmniAuth referes to the plugins that provide authentication via authentication providers, ''strategies''. Strategies are further categorized into [[#Provider Strategies|provider strategies]] and [[#Developer Strategies|developer strategies]].

===Provider Strategies ===
Provider strategies enable authentication to a specific authentication provider. Some example provider strategies are:
* Facebook
* GitHub
* Twitter
* Google
* Windows Live ID

===Developer Strategies===
Developer strategies enable authentication to providers that conform to open or well known standards. Most of the specific provider strategies are customized versions of one of the developer strategies.
* CAS
* Http Basic
* Kerberos
* LDAP
* OAuth
* OAuth2
* OpenID
* OpenAM
* SAML
* Shibboleth
* WS-Federation
* XAuth
* ADFS

==Ruby on Rails Integration==
Integrating OmniAuth into a Rails application that follows the standard authentication pattern for a Rails application is simply a matter of: (1) initialize an OmniAuth::Builder with your authentication provider details, (2) route login requests to <code>/auth/:provider</code> instead of your own login page, and (3) route <code>/auth/:provider/callback</code> to a controller action that initializes your user session (e.g. <code>sessions#create</code>)<ref>OmniAuth Wiki https://github.com/intridea/omniauth/wiki</ref><ref>OmniAuth README.md https://github.com/intridea/omniauth</ref>.

==References==
<references/>

CSC/ECE 517 Spring 2016

2016-01-26T20:37:29Z

Zhu6: /* Calibration Assignment Submissions */

==Calibration Assignment Submissions==
*[[Calibration Assignment Submission (OmniAuth)]]
*[[Calibration Assignment Submission (Patch_verb)]]

CSC/ECE 517 Spring 2016

2016-01-26T20:33:11Z

Zhu6:

==Calibration Assignment Submissions==
*[[Calibration Assignment Submission 1]]
*[[Calibration Assignment Submission 2]]

CSC/ECE 517 Spring 2016

2016-01-26T20:32:51Z

Zhu6:

==Calibration Assignment Submissions==
*[[Calibration Assignment Submission (OmniAuth)]]
*[[Calibration Assignment Submission (Patch_verb)]]

Calibration Assignment Submission 2

2016-01-26T20:31:52Z

Zhu6: /* Patching resources */

Calibration Assignment Submission 2

2016-01-26T20:28:31Z

The '''PATCH''' method is a request method supported by the [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] protocol for making partial changes to an existing resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref> The PATCH method provides an entity containing a list of changes to be applied to the resource requested using the [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] [https://en.wikipedia.org/wiki/Uniform_Resource_Identifier URI].<ref name="RFC5789"/> The list of changes are supplied in the form of a PATCH document.<ref name="RFC5789"/> If the requested resource does not exist then the [https://en.wikipedia.org/wiki/Server_(computing) Server (computing)] may create the resource depending on the PATCH document media type and permissions.<ref name="RFC5789"/> The changes described in the PATCH document must be semantically well defined but can have a different media type than the resource being patched.<ref>Don't Patch Like An Idiot http://williamdurand.fr/2014/02/14/please-do-not-patch-like-an-idiot/</ref> Frameworks such as [https://en.wikipedia.org/wiki/XML XML], [https://en.wikipedia.org/wiki/JSON JSON] can be used in describing the changes in the PATCH document.

==PUT vs PATCH vs POST==

[https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] is the foundation of data communication for the [https://en.wikipedia.org/wiki/World_Wide_Web World Wide Web]. [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] is a request-response protocol which helps users to communicate with the server to perform [https://en.wikipedia.org/wiki/Create,_read,_update_and_delete CRUD] operations. [https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol HTTP] supports a number of request methods such as PUT, [https://en.wikipedia.org/wiki/POST_(HTTP) POST] and PATCH to create or update resources.<ref name=RFC2616>Hypertext Transfer Protocol -- HTTP/1.1 https://www.ietf.org/rfc/rfc2616.txt</ref>

The main difference between the PUT method and PATCH method is that the PUT method uses the request [https://en.wikipedia.org/wiki/Uniform_Resource_Identifier URI] to supply a modified version of the requested resource which replaces the original version of the requested resource whereas the PATCH method supplies a set of instructions to modify the requested resource. If the PATCH document is larger than the size of the new version of the resource sent by the PUT method then the PUT method is preferred.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789|accessdate=2015-09-12}}</ref>

The [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method can be used for sending partial updates to a resource. The main difference between the [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method and the PATCH method is that the [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method can only be used when it is written to support the applications or the applications support its semantics whereas the PATCH method can be used in a generic way and does not require application support. If the outcome of using the PATCH method is not known then the [https://en.wikipedia.org/wiki/POST_(HTTP) POST] method is preferred.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref> <ref>Why PATCH is Good for Your HTTP API https://www.mnot.net/blog/2012/09/05/patch</ref>

==Patching resources==

The PATCH method is [https://en.wikipedia.org/wiki/Atomicity_(database_systems) atomic].<ref name="RFC5789"/> Either all the changes specified by the PATCH method are applied or none of the changes are applied by the server.<ref name="RFC5789"/> There are many ways of checking whether a patch was applied successfully. For example, the 'diff' utility can be applied to the older version and newer version of a file to find the differences between them.<ref name="RFC5789"/>

A cached PATCH response is considered stale. It can only be used for the GET and HEAD requests that may follow the PATCH request.<ref name="RFC5789"/>

The entity headers in the PATCH document are only applicable to the PATCH document and cannot be applied to the requested resource.<ref name="RFC5789"/>

There is no standard format for the PATCH document and it is different for different types of resources. The server has to check whether the PATCH document received is appropriate for the requested resource.<ref name="RFC5789"/>

A '''JSON''' PATCH document would look like
<source lang="json">
{ "op": "add", "variable": "count", "value": 1 }
</source>
"op" represents the operation performed on the resource. "count" represents the resource being modified. "value" represents the amount being added to the existing resource. <ref name=RFCJSONPATCH>JSON Patch - draft-ietf-appsawg-json-patch-08 https://tools.ietf.org/html/draft-ietf-appsawg-json-patch-08</ref> Before applying the changes in the PATCH document, the server has to check whether the PATCH document received is appropriate for the requested resource. If the PATCH request succeeds then it returns a '''200''' response.

==Caution==

The PATCH method is not safe. It may create new resources or change resources not mentioned in the [https://en.wikipedia.org/wiki/Uniform_Resource_Identifier URI].<ref name="RFC5789"/>

The PATCH method is not idempotent. It can be made idempotent by using a conditional request.<ref name=RFC5789 />When a client makes a conditional request to a resource, the request succeeds only if the resource has not been updated since the client last accessed that resource. This also helps in preventing corruption of the resource since some updates to a resource can only be performed starting from a certain base point.<ref name="RFC5789"/>

==Error handling==

A PATCH request can fail if any of the following errors occur:
===Malformed patch document===
The server returns a 400 (Bad request) response if the PATCH document is not formatted as required.<ref name="RFC5789">PATCH Method for HTTP|url=http://tools.ietf.org/html/rfc5789</ref>

===Unsupported patch document===
The server returns a 415 (Unsupported Media Type) response with an Accept-Patch response header containing supported media types when the client sends an unsupported patch document. This informs the client that the PATCH document sent by the client cannot be applied to the requested resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Unprocessable request===
The server returns a 422 (Unprocessable Entity) response when the server is able to understand the PATCH document but is unable to modify the requested resource either because it causes the resource to become invalid or it results in some other error state.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Resource not found===
The server returns a 404 (Not Found) response when the PATCH document cannot be applied to a non-existent resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Conflicting state===
The server returns a 409 (Conflict) response when the server cannot apply a patch for the current state of the resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Conflicting modification===
The server returns a 412 (Precondition Failed) response when the precondition supplied by the client using the If-Match or If-Unmodified-Since header fails. If no precondition is supplied and there is a conflicting modification then the server returns a 409 (Conflict) response.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

===Concurrent modification===
The server returns a 409 (Conflict) response if the PATCH requests to a certain resource need to be applied in a certain order and the server is not able to handle concurrent PATCH requests.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

==Trade-offs between PUT and PATCH==

When only a few changes need to be applied to a resource, using the PUT method consumes more bandwidth as compared to the PATCH method which can be used for sending only the changes. But when the PATCH method is used, it usually involves fetching the resource from the server, comparing the original and new files, creating and sending a diff file. On the server side,the server has to read the diff file and make the modifications. This involves a lot of work overhead compared to PUT method.<ref name=51elliot>REST API Best Practices 3: Partial Updates - PATCH vs PUT http://51elliot.blogspot.com/2014/05/rest-api-best-practices-3-partial.html</ref>
On the other hand, the PUT method requires a GET to be performed before the PUT and it is difficult to ensure that the resource was not modified between the GET and PUT requests.

==Security Considerations==

The PATCH request needs to use mechanisms such as conditional requests using Etags and the If-Match request header to ensure that data is not corrupted while patching.<ref name=RFC5789 /> In case of a failure of a PATCH request or failure of the channel or a timeout, the client can use a GET request to check the state of the resource.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref> The server has to ensure that malicious clients do not use the PATCH method for consuming excessive server resources.<ref name="RFC5789">PATCH Method for HTTP http://tools.ietf.org/html/rfc5789</ref>

==References==

<references/>

Calibration Assignment Submission 1

2016-01-26T20:06:21Z

CSC/ECE 517 Spring 2016

2016-01-26T19:45:43Z

Zhu6: Created page with "==Calibration Assignment Submissions== *Calibration Assignment Submission 1 *Calibration Assignment Submission 2"

==Calibration Assignment Submissions==
*[[Calibration Assignment Submission 1]]
*[[Calibration Assignment Submission 2]]

MainPage

2016-01-26T19:43:18Z

Zhu6: /* Course-Specific Topics */

==Course-Specific Topics==
* [[CSC 216]] learning exercise
* [[CSC 379]]
* [[CSC/ECE 506 Fall 2007]]
* [[ECE506_Main_Page | CSC/ECE 506 Main Portal]]
* [[CSC/ECE 517 Fall 2007]]
* [[CSC/ECE 517 Summer 2008]]
* [[CSC/ECE 517 Fall 2010]]
* [[CSC/ECE 517 Fall 2011]]
* [[CSC/ECE 517 Fall 2012]]
* [[CSC/ECE 517 Fall 2013]]
* [[CSC/ECE 517 Fall 2014]]
* [[CSC/ECE 517 Fall 2015]]
* [[CSC/ECE 517 Spring 2014]]
* [[CSC/ECE 517 Spring 2015]]
* [[CSC/ECE 517 Spring 2016]]
* [[CSC 456 Spring 2011|CSC 456 Spring 2012]]
* [[ECE 633]]
* [[KCU]]
* [[Progress reports]]

==Application Behavior==
* [[Grading]]

==Metaprogramming==
* [[CSC/ECE_517_Spring_2013/ch1b_1k_hf|Lecture on Metaprogramming]]

==Development==

''Expertiza now has a Java dependency, so the machine you are using to develop Expertiza on should have the JVM installed.''

* [[Setting Up a Development Machine]]
* [[Creating a Linux Development Environment for Expertiza - Installation Guide]]
* [[Using git and github for projects]]
* [[Using heroku to deploy your projects]]
* [[How to Begin a Project from the Current Expertiza Repository]]
* [[Git]]
* [[How to Change a User's Password on a Development Machine]]
* [[Debugging Rails]]

==Production==
* [[Deploying to Production]]
* [[Downloading Production Data]]
* [[Accessing the Production Server]]

==Testing==
* [[Using Cucumber with Expertiza]]
* [[Rails Testing Overview]]
* [[Expertiza Continuous Integration]]

==Design==
* [[Object-Oriented Design and Programming]]

MainPage

2015-10-26T18:53:05Z

Zhu6: /* Course-Specific Topics */

==Course-Specific Topics==
* [[CSC 216]] learning exercise
* [[CSC 379]]
* [[CSC/ECE 506 Fall 2007]]
* [[ECE506_Main_Page | CSC/ECE 506 Main Portal]]
* [[CSC/ECE 517 Fall 2007]]
* [[CSC/ECE 517 Summer 2008]]
* [[CSC/ECE 517 Fall 2010]]
* [[CSC/ECE 517 Fall 2011]]
* [[CSC/ECE 517 Fall 2012]]
* [[CSC/ECE 517 Fall 2013]]
* [[CSC/ECE 517 Fall 2014]]
* [[CSC/ECE 517 Spring 2014]]
* [[CSC/ECE 517 Spring 2015]]
* [[CSC/ECE 517 Fall 2015]]
* [[CSC 456 Spring 2011|CSC 456 Spring 2012]]
* [[ECE 633]]
* [[KCU]]
* [[Progress reports]]

==Application Behavior==
* [[Grading]]

==Metaprogramming==
* [[CSC/ECE_517_Spring_2013/ch1b_1k_hf|Lecture on Metaprogramming]]

==Development==

''Expertiza now has a Java dependency, so the machine you are using to develop Expertiza on should have the JVM installed.''

* [[Setting Up a Development Machine]]
* [[Creating a Linux Development Environment for Expertiza - Installation Guide]]
* [[Using git and github for projects]]
* [[Using heroku to deploy your projects]]
* [[How to Begin a Project from the Current Expertiza Repository]]
* [[Git]]
* [[How to Change a User's Password on a Development Machine]]
* [[Debugging Rails]]

==Production==
* [[Deploying to Production]]
* [[Downloading Production Data]]
* [[Accessing the Production Server]]

==Testing==
* [[Using Cucumber with Expertiza]]
* [[Rails Testing Overview]]
* [[Expertiza Continuous Integration]]

==Design==
* [[Object-Oriented Design and Programming]]

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:36:45Z

Zhu6: /* Set topics staggered deadline */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of <code>due_date</code> variable <code>SignUpTopics</code>. However, we found that many attributes used not belonging to <code>SignUpTopics</code> class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in <code>due_date</code> table. And staggered-deadlines are stored in <code>TopicDeadline</code> table. This table also stores different deadline types.

Below is <code>save_topic_deadlines</code> method after refactoring.
def save_topic_deadlines
#session[:duedates] stores all original duedates info
#due_dates stores staggered duedates
due_dates = params[:due_date]
topics = SignUpTopic.where(assignment_id: params[:assignment_id])
review_rounds = Assignment.find(params[:assignment_id]).get_review_rounds
# j represents the review rounds
j = 0
topics.each { |topic|
for i in 1..review_rounds
topic_deadline_type_subm = DeadlineType.find_by_name('submission').id
topic_deadline_subm = TopicDeadline.where(topic_id: session[:duedates][j]['id'].to_i, deadline_type_id: topic_deadline_type_subm, round: i).first
topic_deadline_subm.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_submission_' + i.to_s + '_due_date']})
flash[:error] = "Please enter a valid " + (i > 1 ? "Resubmission deadline " + (i-1).to_s : "Submission deadline") if topic_deadline_subm.errors.length > 0
topic_deadline_type_rev = DeadlineType.find_by_name('review').id
topic_deadline_rev = TopicDeadline.where(topic_id: session[:duedates][j]['id'].to_i, deadline_type_id: topic_deadline_type_rev, round:i).first
topic_deadline_rev.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_review_' + i.to_s + '_due_date']})
flash[:error] = "Please enter a valid Review deadline " + (i > 1 ? (i-1).to_s : "") if topic_deadline_rev.errors.length > 0
end
topic_deadline_subm = TopicDeadline.where(topic_id: session[:duedates][j]['id'], deadline_type_id: DeadlineType.find_by_name('metareview').id).first
topic_deadline_subm.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_submission_' + (review_rounds+1).to_s + '_due_date']})
flash[:error] = "Please enter a valid Meta review deadline" if topic_deadline_subm.errors.length > 0
j = j + 1
}
redirect_to_assignment_edit(params[:assignment_id])
end

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
- redirect_to_sign_up(params[:assignment_id])
+ redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 2]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:35:03Z

Zhu6: /* Set topics staggered deadline */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:34:32Z

Zhu6: /* Save topics dependency and show dependency graph */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:33:50Z

Zhu6: /* Set topics staggered deadline */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of <code>due_date</code> variable <code>SignUpTopics</code>. However, we found that many attributes used not belonging to <code>SignUpTopics</code> class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in <code>due_date</code> table. And staggered-deadlines are stored in <code>TopicDeadline</code> table. This table also stores different deadline types.

Below is <code>save_topic_deadlines</code> method after refactoring.
def save_topic_deadlines
#session[:duedates] stores all original duedates info
#due_dates stores staggered duedates
due_dates = params[:due_date]
topics = SignUpTopic.where(assignment_id: params[:assignment_id])
review_rounds = Assignment.find(params[:assignment_id]).get_review_rounds
# j represents the review rounds
j = 0
topics.each { |topic|
for i in 1..review_rounds
topic_deadline_type_subm = DeadlineType.find_by_name('submission').id
topic_deadline_subm = TopicDeadline.where(topic_id: session[:duedates][j]['id'].to_i, deadline_type_id: topic_deadline_type_subm, round: i).first
topic_deadline_subm.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_submission_' + i.to_s + '_due_date']})
flash[:error] = "Please enter a valid " + (i > 1 ? "Resubmission deadline " + (i-1).to_s : "Submission deadline") if topic_deadline_subm.errors.length > 0
topic_deadline_type_rev = DeadlineType.find_by_name('review').id
topic_deadline_rev = TopicDeadline.where(topic_id: session[:duedates][j]['id'].to_i, deadline_type_id: topic_deadline_type_rev, round:i).first
topic_deadline_rev.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_review_' + i.to_s + '_due_date']})
flash[:error] = "Please enter a valid Review deadline " + (i > 1 ? (i-1).to_s : "") if topic_deadline_rev.errors.length > 0
end
topic_deadline_subm = TopicDeadline.where(topic_id: session[:duedates][j]['id'], deadline_type_id: DeadlineType.find_by_name('metareview').id).first
topic_deadline_subm.update_attributes({'due_at' => due_dates[session[:duedates][j]['id'].to_s + '_submission_' + (review_rounds+1).to_s + '_due_date']})
flash[:error] = "Please enter a valid Meta review deadline" if topic_deadline_subm.errors.length > 0
j = j + 1
}
redirect_to_assignment_edit(params[:assignment_id])
end

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
492 - redirect_to_sign_up(params[:assignment_id])
493 + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 2]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:32:50Z

Zhu6: /* Save topics dependency and show dependency graph */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of <code>due_date</code> variable <code>SignUpTopics</code>. However, we found that many attributes used not belonging to <code>SignUpTopics</code> class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in <code>due_date</code> table. And staggered-deadlines are stored in <code>TopicDeadline</code> table. This table also stores different deadline types.

Below is <code>save_topic_deadlines</code> method after refactoring.

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
492 - redirect_to_sign_up(params[:assignment_id])
493 + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 2]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:32:28Z

Zhu6: /* Set topics staggered deadline */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of <code>due_date</code> variable <code>SignUpTopics</code>. However, we found that many attributes used not belonging to <code>SignUpTopics</code> class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in <code>due_date</code> table. And staggered-deadlines are stored in <code>TopicDeadline</code> table. This table also stores different deadline types.

Below is <code>save_topic_deadlines</code> method after refactoring.

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
492 - redirect_to_sign_up(params[:assignment_id])
493 + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 1]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:31:37Z

Zhu6: /* Workflow */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of <code>due_date</code> variable <code>SignUpTopics</code>. However, we found that many attributes used not belonging to <code>SignUpTopics</code> class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in <code>due_date</code> table. And staggered-deadlines are stored in <code>TopicDeadline</code> table. This table also stores different deadline types.

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
492 - redirect_to_sign_up(params[:assignment_id])
493 + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 1]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:30:09Z

Zhu6: /* Set topics staggered deadline */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of ```due_date``` variable ```SignUpTopics```. However, we found that many attributes used not belonging to ```SignUpTopics``` class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session. We also comprehended the relationship of corresponding tables. The original due dates are stored in ```due_date``` table. And staggered-deadlines are stored in ```TopicDeadline``` table. This table also stores different deadline types.

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
492 - redirect_to_sign_up(params[:assignment_id])
493 + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 1]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-25T19:25:46Z

Zhu6: /* Set topics staggered deadline */

E1524. Refactor staggered-deadline assignments

=='''Overview'''==
===Introduction to Expertiza===
[http://expertiza.ncsu.edu/ Expertiza] is a web application where students can submit and peer-review learning objects (articles, code, web sites, etc). It is used in select courses at NC State and by professors at several other colleges and universities<ref>[https://github.com/expertiza/expertiza Expertiza. Github]</ref>.

===Staggered Deadlines===
Staggered deadlines involve planning alternate submission dates for papers, projects, or exams when a student has conflicting due dates for these. The key part of staggered deadlines is the planning. Staggered deadlines are always established well in advance of the scheduled due date. It is the advanced planning of these deadlines that makes them "staggered deadlines" rather than extensions<ref>[http://pages.vassar.edu/accessingvassar/2013/01/15/difficulties-with-staggered-deadlines/ Elmichaels. ''Difficulties with Staggered Deadlines''. Jan 15, 2013]</ref>.

__TOC__

=='''Documentation'''==
'''Controller'''
* [https://github.com/expertiza/expertiza/blob/5c7c8579b6fe44e19f96183af43fab26cd26d1d1/app/controllers/sign_up_sheet_controller.rb sign_up_sheet_controller.rb]
* [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb]

'''Model'''
* [https://github.com/expertiza/expertiza/blob/142a9134a18bd7cc552c6937fe5c93a1e0f4947d/app/models/assignment_form.rb assignment_form.rb]
* [https://github.com/expertiza/expertiza/blob/72b61129c70a88d1c1ae814a53cb83b6a15a3fa8/app/models/assignment.rb assignment.rb]

'''View'''
* [https://github.com/expertiza/expertiza/blob/3443fe119ee4ae02c676131b07cd42c2771f88e5/app/views/sign_up_sheet/add_signup_topics_staggered.html.erb add_signup_topics_staggered.html.erb]
* [https://github.com/expertiza/expertiza/blob/651176622bff84b09c7de9f745a246279501deb8/app/views/tree_display/actions/_assignments_actions.html.erb _assignments_actions.html.erb]
* [https://github.com/expertiza/expertiza/blob/f2495041c0d66ee5d35c3da23d45e043072f7a1c/app/views/sign_up_sheet/_table_header.html.erb _table_header.html.erb]

=='''Problem Statement'''==
==='''Background'''===
In this semester's 517 class. Wiki 1a and Wiki 1b are structured as separate assignments, with separate signup sheets, teams, and reviews. But really, since only one of the two was done by any student, it would've been better to have a single assignment. Still, some topics could be done soon after the course started, whereas others were better done after we had studied related topics in class.

==='''Staggered-deadline Assignment'''===
This raises the idea of a ''staggered-deadline assignment'', where different topics have different submission and review deadlines, rather than all topics having the same deadline.

==='''Benefit'''===
* '''Easy to manage.''' Because in the past, instructor has to build two separate assignments for Wiki 1a and 1b, now one assignment for Wiki is enough.

* '''Increase flexibility.''' For instance, when an OSS team facing a much more difficult problem than other teams, instructor may postpone their deadline several days. It will make Expertiza more humanize.

* '''Take time to do calibration.''' We find that two Wiki assignments started at Jan. 28th and ended at Feb. 25th, which takes too much time. And with staggered-deadline assignment, we can distribute less time on them. And we can take redundant time to do calibrating. Because we find that many students actually do not know how to evaluate other ones' work, even there are some rubrics offered. So we recommend to let students grade sample assignments first, then calibrate their grading behaviors. Although it will take some time, we consider it is important to help students distinguish good assignments from common ones.

=='''Product Verion Functionality'''==
* In production version, instructor can set ''Submission deadline'', ''Review deadline'' and ''Metareview deadline'' for each topic.

[[File:Staggered-deadline_assignments_M.png‎ |frame|center|Set different deadline for each topic]]

* Also, instructor can set the dependencies of different topics. And there is a dependency graph generated automatically.

[[File:Staggered-deadline_assignments1_M.png‎ |frame|center|Dependency graph for all topics]]

=='''Design Pattern'''==
* '''MVC'''
**Model–view–controller (MVC) is a software architectural pattern for implementing user interfaces. It divides a given software application into three interconnected parts, so as to separate internal representations of information from the ways that information is presented to or accepted from the user.<ref>[http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller. ''Model–view–controller''. Wikipedia]</ref>
**We need to modify model, view and controller in order to fix this functionality.

* '''Publish-subscribe'''
**In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, published messages are characterized into classes, without knowledge of what, if any, subscribers there may be. Similarly, subscribers express interest in one or more classes, and only receive messages that are of interest, without knowledge of what, if any, publishers there are.<ref>[http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern. ''Publish–subscribe pattern''. Wikipedia]</ref>
** When instructor set an assignment as staggered-dealine assignment, in theory, all the students will know.

=='''Use Case'''==

* '''Use Case #1: Create staggered-deadline assignment'''
** '''Actor:''' Instructor
** '''Actions:'''
*** Instructor logs in Expertiza.
*** Instructor creates a new staggered-deadline assignment.

* '''Use Case #2: Peer review only'''
** '''Actor:''' Student finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Hand in assignment.
*** Peer review.
*** Write author feedback to reviewer.

* '''Use Case #3: Peer review and still work on assignments'''
** '''Actor:''' Student not finish work for first round
** '''Actions:'''
*** Sign up topic.
*** Work on assignment.
*** Peer review.

[[File:UseCaseDiagram1_M.png‎ |frame|center|Use Case Diagram]]

=='''Error Message Present'''==
* First, login expertiza, in the "Manage Content" page, create a new assignment. Then, as shown below, choose the "pencil" icon to edit the assignment.

[[File:assignment1.jpg |frame|center|Assignment panel 1]]

* Check"Has topics?" and "Staggered deadline assignment?" two checkbox, and create new topics in "Topics" panel, click "save".

[[File:assignment2.jpg |frame|center|Create Topics]]

* Back to the "Manage Content" page, choose the "Edit signup sheet" icon.

[[File:assignment3.jpg |frame|center|Assignment panel 2]]

* Error: can't write unknown attribute 't_id'.

[[File:assignment4.jpg |frame|center|Error]]

=='''Problem Analysis'''==
* After discussing with professor, the "sign up sheet" icon in assignment pop up panel should be moved. Because that icon has the same functionality as "topic" panel in assignment edit window. These work has already done by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2015/oss_E1509_lds OSS1509]. So we have to move all the staggered-deadline realted code to the [https://github.com/expertiza/expertiza/blob/rails4/app/controllers/assignments_controller.rb assignments_controller.rb] and [https://github.com/expertiza/expertiza/blob/rails4/app/views/assignments/edit.html.erb assignments/edit.html.erb].

* In "sign_up_sheet_controller", there are several errors and confusions in function "add_sign_up_topic". The function is used to display a page that lists all the available topics for a staggered-deadline assignment. 
See the function code below:
<pre>
def add_signup_topic
load_add_signup_topics(params[:id])

@review_rounds = Assignment.find(params[:id]).get_review_rounds
@topics = SignUpTopic.where(assignment_id: params[:id])

#Use this until you figure out how to initialize this array
@duedates = SignUpTopic.find_by_sql("SELECT s.id as topic_id FROM sign_up_topics s WHERE s.assignment_id = " + params[:id].to_s)

unless @topics.nil?
i=0
@topics.each { |topic|

@duedates[i]['t_id'] = topic.id
@duedates[i]['topic_identifier'] = topic.topic_identifier
@duedates[i]['topic_name'] = topic.topic_name

for j in 1..@review_rounds
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('submission').id).first
duedate_rev = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('review').id).first
if !duedate_subm.nil? && !duedate_rev.nil?
@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
else
#the topic is new. so copy deadlines from assignment
set_of_due_dates = DueDate.where(assignment_id: params[:id])
set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, 0, topic.id)
}

@duedates[i]['submission_'+ j.to_s] = DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
@duedates[i]['review_'+ j.to_s] = DateTime.parse(duedate_rev['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")
end

end
duedate_subm = TopicDeadline.where(topic_id: topic.id, deadline_type_id: DeadlineType.find_by_name('metareview').id).first
@duedates[i]['submission_'+ (@review_rounds+1).to_s] = !(duedate_subm.nil?)?(DateTime.parse(duedate_subm['due_at'].to_s).strftime("%Y-%m-%d %H:%M:%S")):nil
i = i + 1
}
end

end
</pre>
The main confusions are:
* In line 8, the variable "duedates" are declared as type "SignUpTopic". The naming is quite confusing.
* In line 14, the SignUpTopic class doesn't have a column "t_id", it raises an error. We think it should be "topic_id".
* From the code we can guess the variable "duedates" is used to store the deadline for each topic. The class "DueData" does exist, but it does not have column "topic_identifier" or "topic_name". It only stores the deadline for a single assignment.

=='''Puzzle'''==
* If there is more than one round, which means students can submit their assignments more than once, so the staggered-deadline should refer to which deadline?
'''[Professor]''' In a staggered-deadline assignment, there is one submission deadline and one review deadline per round. These are always set on a per-topic basis.
If the view currently works (and I think it might), as soon as you make an assignment a staggered-deadline assignment, the Signup Sheet (Topics) page will have a link at the bottom to show deadlines for each topic. Click it, and you will see a separate text box for each deadline for each topic.

* If staggered-deadline of one assignment is even later than the second-round review, is it means that that assignment do not need peer review and go directly into meta-review stage?
'''[Professor]''' No, there are also separate meta-review deadlines for a staggered-deadline assignment.
I think in the current implementation, only submission deadlines, review deadlines, and meta-review deadlines vary by topic. In a more complete implementation, there would be a way for ANY deadline to vary by topic. This would include signup deadlines, drop-topic deadlines, team-formation deadlines, and any other kind of deadline that is defined later.

* When to set staggered-deadline assignments? Create assignment or Before submission?
'''[Professor]''' Basically, the staggered-deadline is set at the same time when an assignment is created. But if one team cannot hand in their work with sufficient reasons, instructor may extend the deadline of their topic.

* What is the meaning of "dependencies" between topics?
'''[Professor]''' The ''dependencies'' of topics means one topic cannot start until another topic finishes. For instance, ''Ruby'' topic must be finished before ''Rails'' topic.

=='''Workflow'''==

==='''Set topics staggered deadline'''===
At very beginning, we find that the type of ```due_date``` variable ```SignUpTopics```. However, we found that many attributes used not belonging to ```SignUpTopics``` class. So we deceide to use has tables instead. In order to keep update the staggered deadline, we put this variable in session.

==='''Save topics dependency and show dependency graph'''===
In edit signup sheet for assignment, we want to save topic dependency and show the depandency graph, as show below:
[[File:save_denpendency.jpg |frame|center|Assignment panel 1]]
In order to show the dependency graph, we must first save topics dependency and save the topic dependency graph under <code> public/assets/staggered_deadline_assignment_graph </code> path. We first add new method in 'due_date' model:
<pre>
def self.assign_start_due_date(assignment_id, set_of_topics)

#Remember, in create_common_start_time_topics function we reversed the graph so reverse it back
set_of_topics = set_of_topics.reverse

set_of_topics_due_dates = Array.new
i=0
days_between_submissions = Assignment.find(assignment_id)['days_between_submissions'].to_i
set_of_topics.each { |set_of_topic|
set_of_due_dates = nil
if i==0
#take the first set from the table which user stores
set_of_due_dates = DueDate.where(assignment_id)
offset = 0
else
set_of_due_dates = TopicDeadline.where(set_of_topics[i-1][0])
set_of_due_dates.sort_by { |a, b| a.due_at <=> b.due_at }
offset = days_between_submissions
end

set_of_topic.each { |topic_id|
#if the due dates have already been created and the save dependency is being clicked,
#then delete existing n create again
prev_saved_due_dates = TopicDeadline.where(topic_id)

#Only if there is a dependency for the topic
if !prev_saved_due_dates.nil?
num_due_dates = prev_saved_due_dates.length
#for each due date in the current topic he want to compare it to the previous due date
for x in 0..num_due_dates - 1
#we don't want the old date to move earlier in time so we save it as the new due date and destroy the old one
if DateTime.parse(set_of_due_dates[x].due_at.to_s) + offset.to_i < DateTime.parse(prev_saved_due_dates[x].due_at.to_s)
set_of_due_dates[x] = prev_saved_due_dates[x]
offset = 0
end
prev_saved_due_dates[x].destroy
end
end

set_of_due_dates.each { |due_date|
create_topic_deadline(due_date, offset, topic_id)
}
}
i = i+1
}
end
</pre>

Then, change the <code>save_topic_dependencies</code> method in <code> sign_up_sheet_controller.rb </code>:
<pre>
492 - redirect_to_sign_up(params[:assignment_id])
493 + redirect_to :action => 'add_signup_topics_staggered', :id => params[:assignment_id]
</pre>

Since we use <code>RGL:Graph#write_to_graphic_file</code> method to turn the dependency graph into <code>jpg</code> file, we need to install graphviz in our gems. Otherwise, the RGL will uses <code>dot</code> files as an intermediary format to produce image formats. If we download the <code>GraphViz</code> packege, RGL will invoke it to produce a <code>jpg</code> file.
<pre>
gem 'graphviz'
</pre>
Then, after we click the <code>Save dependency</code> button, and click <code>show dependency graph</code> button, the dependency graph is shown below:
[[File:show_denpendency.jpg |frame|center|Assignment panel 1]]

==='''Test'''===

==='''Deploy on VCL'''===

* Visit http://vcl.ncsu.edu/ > Make a Reservation
* Select Ruby on Rails / Expertiza from the "Please select the environment ..." dropdown, select a duration, and then click Create Reservation
* Once the reservation is available, click Connect
* Follow the instructions on the connect screen (connect via SSH and then VNC). Connecting to SSH starts up VNC, the graphical environment to which you will connect.
* When connecting for the first time, it will prompt you for a password to access your (VNC) desktop. You can use any password, and it will be saved for later sessions.
* Follow the instructions given to connect to VNC. It is recommended to use [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0CB0QFjAA&url=https%3A%2F%2Fchrome.google.com%2Fwebstore%2Fdetail%2Fvnc%25C2%25AE-viewer-for-google-ch%2Fiabmpiboiopbgfabjmgeedhcmjenhbla&ei=LEY7Veq8C8GNNrvhgegI&usg=AFQjCNGxB1W4GwZrwQWHyBfik1cbjvXMkw VNC Viewer for Google Chrome].
* Once connected to VNC, you should see Rubymine running.
* Open a terminal (Applications > Accessories > Terminal), try hitting <ctrl>-C to kill pending task if your terminal does not work properly.
* Ensure you have at least 20M disk quota available: fs lq . You can allocate more space at https://sysnews.ncsu.edu/tools-bin/usmdb-quota
* Clone the git repository: git clone (expertiza repository URL)
* Change to the directory (cd expertiza). You may create a "database.yml" file based on "database.yml.example".
* Use rvm use (ruby version) to switch to proper ruby version. Then run bundle install (you may need to downgrade the bundler to avoid bundle install failure) and rake db:migrate to set up development environment.
* Use "rails s -b 0.0.0.0" to start server which allows the access outside VCL by using url: VCL_ip:3000.

=='''Related materials'''==
*An [http://research.csc.ncsu.edu/efg/expertiza/reports/Instructor_documentation.doc instructor manual], explaining how to create an deploy an assignment in Expertiza.
*An [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/instructor.swf instructor video], slightly dated, showing how to create and deploy an assignment
* A guide for [[Creating_Custom_Rubric]]
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_documentation.pdf PDF] presentation explaining how to submit and review an assignment with Expertiza.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/videos/student.swf video] showing how to use the system to submit and review an assignment.
*For students, a [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.ppt Powerpoint] or [http://research.csc.ncsu.edu/efg/expertiza/presentations/student_wiki_documentation.pdf PDF] presentation explaining how to submit and review wiki pages with Expertiza.
*For students, a [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.ppt Powerpoint] or [http://courses.ncsu.edu/csc517/common/homework/topic-signup-team-formation.pdf PDF] presentation explaining how to form teams and sign up for topics<ref>[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza. Wolfwikis]</ref>.

=='''References'''==
<references/>

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-24T02:58:03Z

Zhu6:

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T15:16:23Z

Zhu6: /* Design Pattern */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T15:15:45Z

Zhu6: /* Design Pattern */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T15:13:56Z

Zhu6: /* Design Pattern */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T15:13:20Z

Zhu6:

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T15:05:20Z

Zhu6: /* Documentation */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T15:05:07Z

Zhu6: /* Documentation */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T14:57:26Z

Zhu6: /* Problem Analyse */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T14:56:33Z

Zhu6:

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-11T14:54:23Z

Zhu6: /* Initial Analyse */

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-01T01:12:41Z

Zhu6:

CSC/ECE 517 Spring 2015 S1524 FSZZ

2015-04-01T01:12:10Z

Zhu6: /* Puzzle */