Explain ResourceManager, NodeManager, and ApplicationMaster – Hadoop Tutorial

In the Hadoop ecosystem, YARN (Yet Another Resource Negotiator) is the resource management layer that handles scheduling and cluster resource allocation. At the core of YARN’s architecture are three major components: ResourceManager, NodeManager, and ApplicationMaster. Together, these components ensure efficient resource utilization, task execution, and monitoring in a Hadoop cluster.

In this Hadoop tutorial, we will explain each of these components in detail, along with their roles and interactions.

1. ResourceManager (RM)

The ResourceManager is the master daemon responsible for managing resources across the Hadoop cluster. It acts as the global resource arbitrator.

Functions of ResourceManager:

• Resource Allocation: Assigns resources (CPU, memory, containers) to applications.

• Scheduling Policies: Works on scheduling policies like Capacity Scheduler, Fair Scheduler, or FIFO.

• Application Management: Accepts job submissions, negotiates resources, and monitors the status.

Key Components of ResourceManager:

1. Scheduler: Responsible for allocating resources based on configured policies.

2. ApplicationManager: Manages the lifecycle of applications, from submission to completion.

2. NodeManager (NM)

The NodeManager is a per-node agent running on each node in the Hadoop cluster. It is responsible for managing the resources on its local node.

Functions of NodeManager:

• Container Management: Launches and monitors containers on the node.

• Resource Reporting: Sends node health status and resource availability to the ResourceManager.

• Task Execution: Works with ApplicationMaster to execute tasks assigned to the node.

In short, the NodeManager ensures that each node contributes its resources effectively to the cluster.

3. ApplicationMaster (AM)

Every application in YARN has its own ApplicationMaster, which runs in its dedicated container. It is responsible for managing the execution of a single job.

Functions of ApplicationMaster:

• Resource Negotiation: Requests resources from the ResourceManager.

• Task Scheduling: Works with NodeManagers to launch containers for tasks.

• Monitoring: Tracks the progress of tasks and handles failures by restarting containers if required.

• Client Communication: Sends updates and job status to the client.

Interaction Between ResourceManager, NodeManager, and ApplicationMaster

1. A client submits an application to the ResourceManager.

2. The ResourceManager allocates a container for the ApplicationMaster.

3. The ApplicationMaster requests additional resources from the ResourceManager.

4. The ResourceManager assigns resources, and the ApplicationMaster coordinates with NodeManagers to launch containers.

5. NodeManagers execute tasks, monitor container health, and report back to the ResourceManager.

6. The ApplicationMaster tracks the job progress and communicates the result to the client.

Example: Running a MapReduce Job on YARN

• The client submits the job to the ResourceManager.

• The ApplicationMaster for the MapReduce job is launched.

• The ApplicationMaster negotiates resources and instructs NodeManagers to launch map and reduce tasks in containers.

• Containers complete the tasks, and the final output is reported back to the client.

Conclusion

The ResourceManager, NodeManager, and ApplicationMaster form the backbone of YARN’s architecture. While the ResourceManager handles global resource allocation, the NodeManager manages resources on individual nodes, and the ApplicationMaster manages job execution. Together, they ensure efficient cluster utilization, scalability, and fault tolerance in Hadoop.