Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T18:16:39Z

Aagarwa6: /* Summary */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

<<SCREENSHOT COMES HERE>>>>

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search After Save ==

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Further Work (wrt to OSS Final Project) ==
The further work in the OSS project is listed below:
*Integration with Parsers
*rewire the 'parse_twitter' Parser
*Run Parsers on the list
*Extract Documents Parser
*Parser should run on any Message
*Parser should read attachments
*Parser should add all URLs as documents
*Parser should parse link types to see if there are Images
*optionally parser should be able to download linked images/documents into local store
*Train these parsers

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==
The Sahana Eden Foundation developed for Disaster Management provides a web platform to provide solution for the relief effort. The project provides a solution to manage organizations, people, projects, inventory and assets as well as collecting information through assessments and providing situational awareness through maps.
Hence, enhancing the searches and reports will help more people in Disaster Management.

== References ==

<references/>

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T18:11:52Z

Aagarwa6: /* Further Work (wrt to OSS Final Project) */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

<<SCREENSHOT COMES HERE>>>>

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search After Save ==

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Further Work (wrt to OSS Final Project) ==
The further work in the OSS project is listed below:
*Integration with Parsers
*rewire the 'parse_twitter' Parser
*Run Parsers on the list
*Extract Documents Parser
*Parser should run on any Message
*Parser should read attachments
*Parser should add all URLs as documents
*Parser should parse link types to see if there are Images
*optionally parser should be able to download linked images/documents into local store
*Train these parsers

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== References ==

<references/>

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T18:01:19Z

Aagarwa6: /* Further Work (wrt to OSS Final Project) */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

<<SCREENSHOT COMES HERE>>>>

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search After Save ==

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Further Work (wrt to OSS Final Project) ==
The further work in the project include things like
*Integration with Parsers
*rewire the 'parse_twitter' Parser
*Run Parsers on the list
*Extract Documents Parser
*Runs on any Message
*read attachments
*Adds all URLs as documents
*parses link types to see if there are Images (ideally following links as some links aren't direct to the jpg/png)
*optionally downloads linked images/documents into local store
*Train these parsers

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== References ==

<references/>

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T18:00:41Z

Aagarwa6: /* Further Work (wrt to OSS Final Project) */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

<<SCREENSHOT COMES HERE>>>>

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search After Save ==

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Further Work (wrt to OSS Final Project) ==
The further work in the project include things like
*Integration with Parsers
*rewire the 'parse_twitter' Parser
*Run Parsers on the list
*Extract Documents Parser
*Runs on any Message
*read attachments
*Adds all URLs as documents
*msg_message a doc_entity? or use link table
*parses link types to see if there are Images (ideally following links as some links aren't direct to the jpg/png)
*optionally downloads linked images/documents into local store
*Train these parsers

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== References ==

<references/>

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T17:56:25Z

Aagarwa6: /* Future Work */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

<<SCREENSHOT COMES HERE>>>>

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search After Save ==

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Further Work (wrt to OSS Final Project) ==
*Run Parsers on the list
*Extract Documents Parser
*Runs on any Message
*Can read attachments?
*Adds all URLs as documents
*msg_message a doc_entity? or use link table
*parses link types to see if there are Images (ideally following links as some links aren't direct to the jpg/png)
*optionally downloads linked images/documents into local store
*Train these parsers

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== References ==

<references/>

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T17:54:26Z

Aagarwa6: /* Further Work */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.<ref>Tressel, Pat, Fran Boon, Shikhar Kohli, Dominic Koenig, Belinda Lopez, Eli Lev, Michael Howden, and Anne Goldenberg(2011, October) ''Sahana Eden'' Lulu.com, Retrieved from <http://en.flossmanuals.net/sahana-eden/index/> </ref>
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==
The Sahana Eden’s social media feeds requires the user to configure the twitter settings. This is done by providing the twitter login ID, and the OAuth credentials in the Twitter Settings page

<<SCREENSHOT COMES HERE>>>>

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

== Twitter Search Results Filter ==
The twitter search results for the search queries can be found in the Twitter Search Results page. The search filters on the Results page allow searching of feeds by "Tweeted date" and "Tweeted by".

: [[File:New-tweeted-by.png]]

:[[File:New-tweetedby.png]]

== Twitter Search After Save ==

== Twitter Search Results Report ==

The Reports page generates Reports based on date filters and "tweeted by". The report contains the total number of records, search queries, Tweeted on and language that can further be grouped by search query, tweeted on or language.

:[[ File:Report1.png]]

:[[ File:Report2.png]]

== Future Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== References ==

<references/>

== Notes ==
* All references and further reading links are in the APA6 style
* All the content on this page is from websites that have a Attribution-NonCommercial-NoDerivs 3.0 or similar license that allows us to use their content.

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T15:16:52Z

Aagarwa6: /* Confgiuration - Twitter */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Configuration - Twitter ==

== Twitter Search Results Filter ==
The search filters implemented allows searching of feeds by "Tweeted date" and "Tweeted by"

== Twitter Search After Save ==

== Twitter Search Results Report ==

== Further Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== Notes ==
{{reflist|1}}

== References ==

<references/>
* http://en.flossmanuals.net/

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T15:02:35Z

Aagarwa6: /* S3 */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.). S3CRUD has filter forms integrated in the standard list view (=multi-record request w/o URL method). Filter forms can be rendered by any method handler. Since filter forms will never be submitted, the method handler does not need to handle any subsequent POSTs related to the filter form, but just insert the filter form HTML into the page.

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Confgiuration - Twitter ==

== Twitter Search Results Filter ==
The search filters implemented allows searching of feeds by "Tweeted date" and "Tweeted by"

== Twitter Search After Save ==

== Twitter Search Results Report ==

== Further Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== Notes ==
{{reflist|1}}

== References ==

<references/>
* http://en.flossmanuals.net/

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T15:00:55Z

Aagarwa6: /* S3 */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.).

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The functions like list,count,sun and avg have been tested so far.

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Confgiuration - Twitter ==

== Twitter Search Results Filter ==
The search filters implemented allows searching of feeds by "Tweeted date" and "Tweeted by"

== Twitter Search After Save ==

== Twitter Search Results Report ==

== Further Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== Notes ==
{{reflist|1}}

== References ==

<references/>
* http://en.flossmanuals.net/

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:59:38Z

Aagarwa6: /* S3 */

== Introduction to Eden ==

Sahana Eden is an Open Source Humanitarian Platform which has been built specifically for Disaster Management,Development, and Environmental Management sectors. It is highly configurable so that it can be used in a wide variety of different contexts and is easy to modify to build custom solutions.
Sahana Eden can be accessed from the web or locally from a flash drive, allowing it to be used in environments with poor internet. Local & Web versions can be configured to synchronize to allow data to be shared between them. This helps to keep data synchronized to receive updates regularly and help in responding and recovering from disasters more effectively and efficiently.

Sahana Eden's modular design allows different pieces of functionality to be enabled and disabled as required providing flexible solutions for changing contexts.

The application can be configured to secure sensitive information, while also making data which needs to be shared available in a variety of different formats including Microsoft Excel and PDF. To ensure that Sahana Eden is accessible to every country, it can be translated into multiple languages.

== Framework - Web2py ==
Web2py is an enterprise framework for agile development of secure database-driven web-based applications, written and programmable in Python.web2py is a full-stack framework, meaning that it contains all the components you need to build fully functional web applications.<ref name="name"> Massimo Di Pierro, ''Complete Reference Manual, 6th Edition (pre-release)'' Retrieved from <http://www.web2py.com/init/default/documentation> </ref>
=== Features of Web2py: <ref name = "name"> web2py.com </ref> ===
web2py includes a Database Abstraction Layer (DAL) that writes SQL dynamically so that you, the developer, don't have to. The DAL knows how to generate SQL transparently for SQLite, MySQL, PostgreSQL, MSSQL, FireBird, Oracle, IBM DB2, Informix and Ingres.
* The DAL can also generate function calls for the Google Datastore when running on the Google App Engine.
* web2py differs from other web frameworks in that it is the only framework to fully embrace the Web 2.0 paradigm, where the web is the computer.web2py does not require installation or configuration; it runs on any architecture that can run Python (Windows, Windows CE, Mac OS X, iOS, and Unix/Linux), and the development, deployment, and maintenance phases for the applications can be done via a local or remote web interface. web2py runs with CPython (the C implementation) and PyPy (Python written in Python), on Python versions 2.5, 2.6, and 2.7.
* web2py provides a ticketing system for error events. If an error occurs, a ticket is issued to the user, and the error is logged for the administrator.
* web2py is open source and released under the LGPL version 3 license.

=== Why Web2Py? <ref name = "name"> web2py.com </ref>===
web2py is one of many web application frameworks, but it has compelling and unique features. web2py was originally developed as a teaching tool, with the following primary motivations:
* Easy for users to learn server-side web development without compromising functionality. For this reason, web2py requires no installation and no configuration, has no dependencies (except for the source code distribution, which requires Python 2.5 and its standard library modules), and exposes most of its functionality via a Web interface, including an Integrated Development Environment with Debugger and database interface.
* web2py has been stable from day one because it follows a top-down design; i.e., its API was designed before it was implemented. Even as new functionality has been added, web2py has never broken backwards compatibility, and it will not break compatibility when additional functionality is added in the future.
* web2py proactively addresses the most important security issues which plague many modern web applications, as determined by OWASP[owasp] below.
* web2py is lightweight. Its core libraries, including the Database Abstraction Layer, the template language, and all the helpers amount to 1.4MB. The entire source code including sample applications and images amounts to 10.4MB.
* web2py has a small footprint and is very fast. It uses the Rocket[rocket] WSGI web server developed by Timothy Farrell. It is as fast as Apache with mod_wsgi, and supports SSL and IPv6.
* web2py uses Python syntax for models, controllers, and views, but does not import models and controllers (as all the other Python frameworks do) - instead it executes them. This means that apps can be installed, uninstalled, and modified without having to restart the web server (even in production), and different apps can coexist without their modules interfering with one another.
* web2py uses a Database Abstraction Layer (DAL) instead of an Object Relational Mapper (ORM). From a conceptual point of view, this means that different database tables are mapped into different instances of one Table class and not into different classes, while records are mapped into instances of one Row class, not into instances of the corresponding table class. From a practical point of view, it means that SQL syntax maps almost one-to-one into DAL syntax, and there is no complex metaclass programming going on under the hood as in popular ORMs, which would add latency.

== S3 ==
The Sahana Eden framework (S3) is built on Web2Py and It is used for RAD(Papid Application Development) in all the sahana applications. Eden’s custom CRUD framework.
S3 Framework includes pre configured Filters and Reports that allow the data analysis and report generation a simple task to be performed by directly.

S3 Filter forms allow the user to apply filters to interactive views (e.g. data tables, data lists, reports etc.).

S3 Framework also includes S3Report which is a REST method handler for data analysis. S3Report generates a full page which includes a set of S3Search widgets, a pivot table (a.k.a. contingency table) from S3Resources and graphs based on the Pivot Tables. S3Report responds to the /report URL method. S3Report supports a number of aggregation functions. The following functions have been tested so far:
Function Explanation
list a comma-separated list of all instance values
count the number of instance values
sum the sum of all instance values
avg the average (mean) of all instance values

Table configurations can contain a report_options item, which is a Storage object. This object is used to configure reports and report options form.
The rows, cols, facts and aggregate items are lists of available values for the user to select from.
The defaults property is a Storage object that contains the default values for the report. It can contain a value for rows, cols, fact, aggregate and totals (as described in URL Methods).
The search property is a list of S3Search widgets that will allow the report to be filtered. If no search property is specified, no filter form will be available.

== Openshift Deployment ==

OpenShift is a cloud computing platform which offers "Platform As a Service" cloud. It is based on hosting platform provided by RedHat. It supports Ruby, it is really easy to install and start working on it. OpenShift takes care of maintaining the services underlying the application and scaling the application as needed. OpenShift supports web-application frameworks by supporting each language's preferred web-integration API, with no required changes to the actual framework code. web2py works unmodified on OpenShift.

Since the software is open source, it is available on GitHub. Development can be started after deploying different web applications in different languages on the platform using [http://en.wikipedia.org/wiki/Git_(software) Git].Making Changes to your Application on OpenShift, you make code changes on your local machine, check those changes in locally, and then "push" those changes to OpenShift. One of the primary advantages of Git is that it does not require a continuous online presence in order to run. You can easily check in (in Git terminology, 'commit') and revert changes locally before deciding to upload those changes to OpenShift.

== Confgiuration - Twitter ==

== Twitter Search Results Filter ==
The search filters implemented allows searching of feeds by "Tweeted date" and "Tweeted by"

== Twitter Search After Save ==

== Twitter Search Results Report ==

== Further Work ==
*The twitter search is currently hard coded. The search accesses the msg_twitter_search table and use the keywords stored by the user through the eden portal to search and display twitter feeds on the map.
*Display the twitter icon for twitter search instead of the usual bubble
*Work on twitter links and message which would pop up contents like photos,URL etc. This work is doing the part within the tweet parsing of following embedded URLs to identify if they're Media would be really useful addition to the parser (there is a commented-out stub).
*Create a cron job for searching tweets periodically instead of manual searches as per user’s request.
*Current implementation includes integration of Sahana with Twitter. The future enhancements of the eden project includes allow Sahana Eden to be configured to have a Facebook updates in it

== Summary ==

== Notes ==
{{reflist|1}}

== References ==

<references/>
* http://en.flossmanuals.net/

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:53:10Z

Aagarwa6: /* S3 */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:37:47Z

Aagarwa6: /* Openshift Deployment */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:36:06Z

Aagarwa6: /* Openshift Deployment */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:34:57Z

Aagarwa6: /* Openshift Deployment */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:30:17Z

Aagarwa6: /* Openshift Deployment */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:28:41Z

Aagarwa6: /* Openshift Deployment */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:19:35Z

Aagarwa6: /* Further Work */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:19:09Z

Aagarwa6: /* Further Work */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:16:13Z

Aagarwa6: /* References */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:15:07Z

Aagarwa6: /* Introduction to Eden */

CSC/ECE 517 Fall 2013/oss SocialMediaFeeds

2013-10-30T14:13:59Z

Aagarwa6:

CSC/ECE 517 Fall 2013

2013-10-30T02:54:46Z

Aagarwa6:

* [[ CSC/ECE 517 Fall 2012/ch1 1w23 ph ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w30 nn]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w21 w]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w01 aj]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w24 nv]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w29 rkld]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w25 aras]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w30 ps]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w19 rj]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w18 bs]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w17 pk]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w22 ss]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w12 vn]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w14 st]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w08 cc]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w10 ga ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w26 as ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w27 ma ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w13 aa ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w11 sv ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w07 d ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w20 gq ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w03 ss ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w28 nm ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w02 pp ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w6 zs ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w04 y ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w05 st ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w09 hs ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w32 av ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w48 x ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w43 av]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w46 ka]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w33 aa]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w35 sa ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w39 as ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w31 vm ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w43 sm ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w44 s ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w47 ka ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w34 fs ]]
* [[ CSC/ECE 517 Fall 2013/ch1 1w40 ao ]]
* [[ CSC/ECE 517 Fall 2013/oss fmv ]]
* [[ CSC/ECE 517 Fall 2013/oss vna ]]
* [[ CSC/ECE 517 Fall 2013/oss paa ]]
* [[ CSC/ECE 517 Fall 2013/oss mapFeeds ]]
* [[ CSC/ECE 517 Fall 2013/oss ssp ]]
* [[ CSC/ECE 517 Fall 2013/ch2 0e808 nsv ]]
* [[ CSC/ECE 517 Fall 2013/oss aoa ]]
* [[ CSC/ECE 517 Fall 2013/oss cmh ]]
* [[ CSC/ECE 517 Fall 2013/oss ans ]]
* [[ CSC/ECE 517 Fall 2013/oss ssv]]
* [[ CSC/ECE 517 Fall 2013/oss E818 mra ]]
* [[ CSC/ECE 517 Fall 2013/oss SocialMediaFeeds ]]

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-13T02:25:44Z

Aagarwa6: /* MapReduce Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval[1].

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rails and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*[1] http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-13T02:25:32Z

Aagarwa6: /* Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval[1].

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rails and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*[1] http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-13T02:25:17Z

Aagarwa6: /* Document Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval[1].

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rails and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*[1] http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-13T02:25:02Z

Aagarwa6: /* Key Value Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval[1].

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rails and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*[1] http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:38:09Z

Aagarwa6: /* MapReduce Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval[1].

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*[1] http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:37:31Z

Aagarwa6: /* References */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*[1] http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:36:21Z

Aagarwa6: /* References */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==

*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:34:29Z

Aagarwa6: /* References */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
Specific references:
<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

General References:
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:30:36Z

Aagarwa6: /* References */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
Specific references:
{{Reflist}}
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:27:47Z

Aagarwa6: /* Rais and Redis */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:27:26Z

Aagarwa6: /* Rais and Redis */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
Redis is a extremely quick, key value data store that allows storage of different data types like strings,lists, hashes and sets.Redis keeps all the data in RAM and it periodially writes to disk providing persistence.
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:23:20Z

Aagarwa6: /* MapReduce Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref>

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:22:09Z

Aagarwa6: /* Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''[[File:Graph_data_store.jpg]]'''

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:21:48Z

Aagarwa6: /* Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

'''function'''

[[File:Graph_data_store.jpg]]

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:21:15Z

Aagarwa6: /* MapReduce Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Graph_data_store.jpg]]

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Example of MapReduce Data Store:

'''function''' map(String name, String document):
''// name: document name''
''// document: document contents''
'''for each''' word w '''in''' document:
emit (w, 1)

'''function''' reduce(String word, Iterator partialCounts):
''// word: a word''
''// partialCounts: a list of aggregated partial counts''
sum = 0
'''for each''' pc '''in''' partialCounts:
sum += ParseInt(pc)
emit (word, sum)

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:13:03Z

Aagarwa6: /* Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Graph_data_store.jpg]]

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:12:30Z

Aagarwa6: /* Example of Key Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Example.jpg]]

== MapReduce Data Store: ==
MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:11:50Z

Aagarwa6: /* Introduction to Big Data */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Example.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:11:32Z

Aagarwa6: /* Document Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==
Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.
Example of Document Data Store:
{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Example.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:10:35Z

Aagarwa6: /* Example of key value data store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Key Value Data Store: ==
Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.
Example of key value data store is as follow:

you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

== Document Data Store: ==

{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Example.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:07:34Z

Aagarwa6: /* Example of Document Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Example of key value data store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Document Data Store: ==

{
name: "larry"
unityId: "larryp"
}

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Example.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:07:09Z

Aagarwa6: /* Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Example of key value data store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Example of Document Data Store: ==

{
name: "larry"
unityId: "larryp"
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.

[[File:Example.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:06:27Z

Aagarwa6: /* Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Example of key value data store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Example of Document Data Store: ==

{
name: "larry"
unityId: "larryp"
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

== Graph Data Store: ==
Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.
Example of graph data store can be viewed in the file.
[[File:graph_data_store.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T06:05:40Z

Aagarwa6: /* Example of Graph Data Store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Example of key value data store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Example of Document Data Store: ==

{
name: "larry"
unityId: "larryp"
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

== Graph Data Store: ==
[[File:graph_data_store.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

File:Graph data store.jpg

2013-10-10T06:01:12Z

Aagarwa6: This is an example of Graph Data Store where each node knows its adjacent node. And the nodes can be traversed to search information

This is an example of Graph Data Store where each node knows its adjacent node. And the nodes can be traversed to search information

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T05:59:20Z

Aagarwa6: /* Example of document data store: */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Example of key value data store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Example of Document Data Store: ==

{
name: "larry"
unityId: "larryp"
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

== Example of Graph Data Store: ==
[[File:graph_data_store.jpg]]

== Example of Key Data Store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T05:07:11Z

Aagarwa6: /* Introduction to Big Data */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

== Example of key value data store: ==
Where you have a key in the left column and a value on the right side.

{
Name Mark
Age 20
Size Large
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Example of document data store: ==

{
name: "larry"
unityId: "larryp"
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store

CSC/ECE 517 Fall 2013/ch1 1w47 ka

2013-10-10T05:04:24Z

Aagarwa6: /* Introduction to Big Data */

== Big Data in Rails applications ==

This wiki discusses the implementation and usage of Ruby on Rails (RoR) for Big Data for its analysis and storage using different technologies.

== Introduction to Big Data ==
With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different [http://en.wikipedia.org/wiki/Data_store datastores]. Different types of Data stores are as follows:
*Key-Value Data Store
*Document Data Stores
*Graph Data Stores
*Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

== Example of document data store: ==

{
name: "larry"
unityId: "larryp"
}

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

== Rails and Riak ==
Raik is a NoSQL database that was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, while supporting a variety of additional language drivers such as Python, Java, PHP and Ruby. To use Riak in the application, first Erlang needs to be installed and then riak.

Put
gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git'
gem 'curb'
in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

ripple:
development:
port: 8098
host: localhost

and in all model classes put the following line:
require 'ripple'
class xyz
include Ripple::Document
statement x
statement y
end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

==Rais and Redis ==
As a first step install redis source and run make command. Then start the rails server using following command:
./redis-server

To use redis install redis gem
gem install redis

In the .gemfile put
gem 'redis'
then run bundle install.

Now if the object needs to be saved than a connection object has to be created. For this generate a module as follows:
module RedisMod
@@redis_connection = Redis.new
def self.conn
@@redis_connection
end
end

Now in the model class to save a model object we first need to generate a key as redis is a key-value data store. So first gnerate a key and assign the object to be saved to the key and call conn function to save the record as follows:

def save
#make generate key function using the logic you want
generate_key unless @key
return false unless valid?
# store the key/value pair with @object being the object to be saved.
RedisMod.conn[@key] = @object
true
end

==References ==
*http://www.slideshare.net/vastydeep/ruby-rails-no-sql-and-big-data
*http://www.pikasoft.com/journal/2011/1/2/a-quick-redis-key-value-example-for-the-holidays.html
*http://www.pikasoft.com/journal/2010/7/31/nosql-on-the-cloud-our-first-application.html
*http://jit.nuance9.com/2010/07/ruby-192-rails-3-riak-and-ripple.html
*http://www.pikasoft.com/journal/2011/1/9/nosql-next-up-hadoop-and-cloudera.html
*http://en.wikipedia.org/wiki/Data_store