CSC/ECE 517 Fall 2013/ch1 1w47 ka

Introduction to Big Data

With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different data stores. Different types of Data stores are as follows:

• Key-Value Data Store
• Document Data Stores
• Graph Data Stores
• Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use. Redis and Riak are based on key-value data stores.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different. MongoDB and Cassandra are document Data stores.

An example document in a document data store is as follows: { name: "larry" unityId: "larryp" }

Graph Data Store is a database in which data is represented by nodes and edges. Lookup of elements is possible using pointers to adjacent elements and hence index look-up is not possible. Neo4J uses graph data store.

MapReduce is a program which can be used to retrieve data from a cluster of data nodes using parallel processing. Map() function delegates the work to different data nodes in the cluster and reduce function performs operations on the data retrieved. Hadoop uses the map reduce technology for efficient data storage and retrieval.

Rails and Riak

Riak was designed manly for speed of retrieval. As the data becomes complex it becomes difficult to store using key value pair. Riak was developed using programming language Erlang, so first Erlang needs to be installed and then install riak.

Put gem 'ripple', :git => 'http://github.com/seancribbs/ripple.git' gem 'curb' in the .gemfile of the project and run bundle install.

Next add Ripple into or config/database.yml:

  ripple:
    development:
    port: 8098
    host: localhost

and in all model classes put the following line:

   require 'ripple'
   class xyz
     include Ripple::Document
     statement x
     statement y
   end

After this start riak by going to "path where riak is stored/bin" and running the riak start command.

Now the riak is running and all the CRUD operations will be successfully performed.

After this start riak