The large amount of data generated from various sources is known as big data. Due to its volume it must be managed properly for its better utilization. Many new Big Data technologies have been developed to help us handle it efficiently as well as in a time-saving manner. These Big Data technologies can help analyze, process and manage big data. Some famous technologies are Hadoop, parallel computing, In-Memory Computing among others. Many cloud computing technologies are also used for big data processing.
Many important Big Data Technologies for are discussed as an important step to learn Big Data.
Big Data Technologies and Tools
Distributed and Parallel Computing for Big Data management
Distributing computing constitutes to the multiple computing resources that are connected over a network in a distributed manner. This enhances the speed and efficiency of the computing processes. Adding additional resources also helps in faster processing. Parallel computing can carry out various actions and processes simultaneously.
For big data, many organizations use both types of computing in a hybrid format. Many companies also use third party for the analysis of big data. They use specialized tools for big data management, storage and processing. It is a cost-effective technique. The increase of the volume, velocity, variety and veracity of data increases has made companies to adopt powerful hardware approaches that can process big data in a short time frame.
The basic procedures of the applications to achieve this using parallel and distributed computing are:
- Break given tasks into sub-processes
- Survey the resources
- Assign Subtasks to interconnected computers.
Sometimes latency may arise while a big data processing task is underway. It can be defined as the delay in system due to any other delay in the completion of previous individual tasks. It leads to system slowdown. To bypass latency, distributed and parallel computing is used.
Another important element of big data processing is load balancing. It is defined as the sharing of workload over several systems over a network. Virtualization is also a great technique in which a virtual environment is created to handle various hardware and storage required for big data processing and management.
Some techniques of parallel computing include cluster/grid computing, High-Performance Computing and Massively Parallel Computing.
Hadoop
Hadoop can handle both structured as well as unstructured big data. It is open-source and provides advanced tools to handle large volumes of data. It has a great programming model with distributed and parallel computing techniques embedded in its core. Architecturally, it contains one master node and various working nodes. A secure shell must be set up between 2 given nodes for running start up and shut down scripts.
Some salient features of Hadoop are
- Deals with flat files in any format
- No Schema unlike that of other Databases
- Capable to divide files into blocks automatically
- Hadoop is resilient as it stores multiple copies of data.
- Uses parallel processing of tasks, which is faster.
- Master node performs disk management, work allocation, and job management.
- Hadoop performs well without sharing memory
Mapreduce and HDFS
HDFS or Hadoop Distributed File System is apache’s fault-tolerant storage system of Hadoop. It is used to store large files which are in petabytes to terabytes. It uses multiple hosts’ reliability as well as replication of data. Each file in HDFS is split into blocks of size 64 MB by default which is duplicated over a series of data nodes.
MapReduce is a framework that is used to develop programs to process large volumes of data into a usable format. MapReduce programs can be written in various programming languages such as C++, Perl, Ruby, Python, Java etc. MapReduce is fault-tolerant. MapReduce libraries in these languages can be used to create tasks, without the need of any communication between the nodes.
Functioning of Hadoop
- Hadoop uses multiple resources for a particular big data task.
- Core components of Hadoop are HDFS and MapReduce.
- Hadoop divides tasks over multiple nodes, the number of which can vary dynamically.
- Nodes form a cluster that can be changed according to the requirements.
- The operations performed by using MapReduce.
- Mapper function in MapReduce is used for task and recovery management.
- In MapReduce the data is divided into various parts, after which each part is transferred to different server nodes. A code is associated with each server. Then code runs to process the data and does other processing such as job tracking.
- After all jobs are completed, the final result is forwarded.
