--- title: "Hadoop Distributed processing with MapReduce" description: "MapReduce comprises the sequential processing of operations on distributed volumes of data sets. The data comprises of key-value pairs, and the overal" author: "Anonymous User" published: 2016-04-30 updated: 2018-03-14 canonical: https://www.mindstick.com/blog/11049/hadoop-distributed-processing-with-mapreduce category: "hadoop" tags: ["bigdata", "hadoop", "software development"] reading_time: 3 minutes --- # Hadoop Distributed processing with MapReduce MapReduce comprises the sequential processing of operations on distributed volumes of data sets. The data comprises of key-value pairs, and the overall process involves two phases. **First phase** is a map phase and second phase is reduce phase. User-defined MapReduce jobs run on the compute nodes in the cluster. Generally a Map reduce job run through several phases. During the Map phase, [input data](https://www.mindstick.com/forum/161330/what-are-some-efficient-ways-to-handle-large-input-data-in-python) is broken down into a large number of sub fragments, each of the fragments is assigned to a map task. Numerous map tasks are then distributed across the cluster. Every map task processes the key-value pairs from its allotted sub-fragment to produces a set of intermediate key-value pairs. Now, this intermediate data set is sorted through key, and this sorted data is again partitioned into a slots of fragments that matches the number of reduce tasks. **After this in Reduce phase**, every reduce task processes the data fragment which was allotted to it and results an output key-value pair. However, the reduce tasks are also distributed across the cluster and write their results to HDFS when finished. The Hadoop’s MapReduce framework in earlier i.e. pre-version 2 ,Hadoop releases has a single node master service [known as](https://answers.mindstick.com/qa/35703/ricky-ponting-is-also-known-as-what) a JobTracker and several slave node services known as TaskTrackers, one per node in the cluster. When we submit a MapReduce job to the JobTracker, the job is dispatched onto a queue and then executes in accordance with the scheduling rules defined by the analyst or admin. As we might expect, that the JobTracker handles the assignment of map-and-reduce work to the TaskTrackers. In Hadoop 2, a more powerful [resource management](https://www.mindstick.com/articles/341532/how-ai-driven-resource-management-in-servicenow-is-transforming-workforce-planning) system is in place called YARN (short for Yet Another Resource Manager). **YARN provides** generic scheduling and service for management of resources in a way that we can run more than one Map Reduce [applications](https://www.mindstick.com/articles/12847/how-to-choose-the-right-ethernet-cable-for-industrial-applications) on our [Hadoop cluster](https://www.mindstick.com/interview/23034/hadoop-cluster-is-made-up-of-how-many-classes-of-servers). The JobTracker-TaskTracker design could only meant to run MapReduce. Key/value data is the foundation of MapReduce operations which facilitates for a powerful [programming](https://www.mindstick.com/articles/12214/web-development-company-in-india-laid-on-the-foundation-of-concrete-java-programming) model that is unexpectedly widely applicable, and can be seen by the popularity of Hadoop and MapReduce across a wide range of industries and [challenging](https://www.mindstick.com/news/2240/in-a-u-s-lawsuit-challenging-facebook-s-virtual-reality-agreement-mark-zuckerberg-will-testify) scenarios. Most of data is either intrinsically key-value by nature or can be transformed in such a way. It is one of the simplest models with wide applicability and semantics straightforward enough that programs defined in terms of it can be implemented by a Hadoop framework. Since, only the data model itself is not the reason that makes Hadoop useful; its real [power lies](https://answers.mindstick.com/qa/38033/according-to-preamble-the-ultimate-power-lies-in-the-hands-of) in how it utilizes the [effective techniques](https://yourviews.mindstick.com/view/84711/marketing-strategies-for-new-businesses-5-effective-techniques) like parallel execution and “divides and conquer” strategies. --- Original Source: https://www.mindstick.com/blog/11049/hadoop-distributed-processing-with-mapreduce Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.