CSC/ECE 517 Fall 2013/ch1 1w47 ka

Introduction to Big Data

With the increase in data size it becomes difficult to store the data in relational databases and also processing the data becomes very time consuming. Even if it is feasible to store the data on multiple servers, it becomes difficult to visualize the data all together since it is spread over multiple servers and processing time is also very large. Hence there arises a need of storing and retrieving huge amount of data effectively which involves massive parallel processing to fetch a huge amount of data in less time. This process of storing huge amount of data on multiple servers and processing that data which is not possible using traditional database processing techniques is called big data analysis and this collection of large data sets is called big data.

There are number of frameworks which support big data analysis and storage. To name a few we have Redis, Riak, MongoDB, Cassandra, Neo4J and the biggest of all Hadoop. All of them are based on different data stores. Different types of Data stores are as follows:

• Key-Value Data Store
• Document Data Stores
• Graph Data Stores
• Map Reduce

Key-Value Data Store is similar to hash table storage where a record is accessed using the key. Hence it is very fast but for complex data it is very difficult to use.

Document Data Store is a database which is used for storing, retrieving and managing documents. Each document has a predefined structure and it is uniquely represented by a document key which helps in accessing and storing the document. Each document is similar to a row in relational database. The only difference being each row has a fixed number of attributes(columns) whereas in the documents the number of attributes may be different.

An example document in a document data store is as follows: { name: "larry" unityId: "larryp" }