Hadoop extension

  • Latest Dynatrace
  • Extension
  • Published Oct 27, 2025

Enhanced insights for Hadoop HDFS and Yarn services

Get started

Overview

Hadoop monitoring in Dynatrace provides a high-level overview of the main Hadoop components within your cluster. These enhanced insights provides additional metrics directly from the HDFS and YARN services.

Activate this extension in your Dynatrace environment from the in-product Hub and simply select which OneAgents to enable this on.

Use cases

The extension enables insights into the overall health of Hadoop HDFS and YARN services.

Compatibility information

  • Hadoop version 2.4.1+
  • Linux OS
  • For full Hadoop visibility, OneAgent must be installed on all machines running the following Hadoop processes: NameNode, ResourceManager, NodeManager, DataNode, and MRAppMaster

Details

This extension provides extra Hadoop metrics through the use of JMX queries.

  • HDFS - Hadoop Distributed File System
    • Improved visibility into the health of your HDFS NameNodes and DataNodes
  • YARN - Yet Another Resource Negotiator
    • Improved visibility into the health of your YARN NodeManagers, ResourceManagers and MRAppMaster services

Feature sets

When activating your extension using monitoring configuration, you can limit monitoring to one of the feature sets. To work properly the extension has to collect at least one metric after the activation.

In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.

All metrics that aren't categorized into any feature set are considered to be the default and are always reported.

A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.

YARN-NodeManager-ShuffleMetrics
Metric nameMetric keyDescription
ShuffleConnectionshadoop.yarn.nodemanager.ShuffleConnections.countNumber of Shuffle connections
ShuffleOutputByteshadoop.yarn.nodemanager.ShuffleOutputBytes.countTotal shuffle output bytes
ShuffleOutputsFailedhadoop.yarn.nodemanager.ShuffleOutputsFailed.countNumber of failed shuffle outputs
ShuffleOutputsOKhadoop.yarn.nodemanager.ShuffleOutputsOK.countNumber of OK shuffle outputs
YARN-NodeManager-NodeManagerMetrics
Metric nameMetric keyDescription
NodeManager AllocatedContainershadoop.yarn.nodemanager.AllocatedContainersNodeManager Current number of allocated containers
AllocatedGBhadoop.yarn.nodemanager.AllocatedGBCurrent allocated memory in GB
AvailableGBhadoop.yarn.nodemanager.AvailableGBCurrent available memory in GB
ContainersCompletedhadoop.yarn.nodemanager.ContainersCompleted.countTotal number of successfully completed containers
ContainersFailedhadoop.yarn.nodemanager.ContainersFailed.countTotal number of failed containers
ContainersInitinghadoop.yarn.nodemanager.ContainersIniting.countCurrent number of initializing containers
ContainersKilledhadoop.yarn.nodemanager.ContainersKilled.countTotal number of killed containers
ContainersLaunchedhadoop.yarn.nodemanager.ContainersLaunched.countTotal number of launched containers
ContainersRunninghadoop.yarn.nodemanager.ContainersRunning.countCurrent number of running containers
YARN-ResourceManager-ClusterMetrics
Metric nameMetric keyDescription
NumActiveNMshadoop.yarn.resourcemanager.NumActiveNMsCurrent number of active NodeManagers
NumDecommissioningNMshadoop.yarn.resourcemanager.NumDecommissionedNMsCurrent number of NodeManagers being decommissioned
NumLostNMshadoop.yarn.resourcemanager.NumLostNMsCurrent number of lost NodeManagers for not sending heartbeats
NumRebootedNMshadoop.yarn.resourcemanager.NumRebootedNMsCurrent number of rebooted NodeManagers
NumUnhealthyNMshadoop.yarn.resourcemanager.NumUnhealthyNMsCurrent number of unhealthy NodeManagers
HDFS-NameNode-NameNodeInfo
Metric nameMetric keyDescription
NumberOfMissingBlockshadoop.hdfs.namenode.NumberOfMissingBlocks—
CacheCapacityhadoop.hdfs.namenode.CacheCapacityThe total cache capacity of all DataNodes
CacheUsedhadoop.hdfs.namenode.CacheUsedThe total cache used by all DataNodes
YARN-ResourceManager-QueueMetrics
Metric nameMetric keyDescription
AllocatedContainershadoop.yarn.resourcemanager.AllocatedContainersQueue Current number of allocated containers
AllocatedMBhadoop.yarn.resourcemanager.AllocatedMBCurrent allocated memory in MB
AllocatedVCoreshadoop.yarn.resourcemanager.AllocatedVCoresCurrent allocated CPU in virtual cores
AppsCompletedhadoop.yarn.resourcemanager.AppsCompleted.countTotal number of completed applications
AppsFailedhadoop.yarn.resourcemanager.AppsFailed.countTotal number of failed applications
AppsKilledhadoop.yarn.resourcemanager.AppsKilled.countTotal number of killed applications
AppsPendinghadoop.yarn.resourcemanager.AppsPending.countCurrent number of applications that have not yet been assigned by any containers
AppsRunninghadoop.yarn.resourcemanager.AppsRunning.countCurrent number of running applications
AppsSubmittedhadoop.yarn.resourcemanager.AppsSubmitted.countTotal number of submitted applications
AvailableMBhadoop.yarn.resourcemanager.AvailableMBCurrent available memory in MB
AvailableVCoreshadoop.yarn.resourcemanager.AvailableVCoresCurrent available CPU in virtual cores
PendingMBhadoop.yarn.resourcemanager.PendingMBCurrent memory requests in MB that are pending to be fulfilled by the scheduler
PendingVCoreshadoop.yarn.resourcemanager.PendingVCoresCurrent CPU requests in virtual cores that are pending to be fulfilled by the scheduler
ReservedMBhadoop.yarn.resourcemanager.ReservedMBCurrent reserved memory in MB
ReservedVCoreshadoop.yarn.resourcemanager.ReservedVCoresCurrent reserved CPU in virtual cores
HDFS-NameNode-NameNodeActivity
Metric nameMetric keyDescription
FilesAppendedhadoop.hdfs.namenode.FilesAppended.countTotal number of files appended
FilesCreatedhadoop.hdfs.namenode.FilesCreated.countTotal number of files and directories created by create or mkdir operations
FilesDeletedhadoop.hdfs.namenode.FilesDeleted.countTotal number of files and directories deleted by delete or rename operations
FilesRenamedhadoop.hdfs.namenode.FilesRenamed.countTotal number of rename operations (NOT number of files/dirs renamed)
HDFS-DataNode-FSDatasetState
Metric nameMetric keyDescription
DataNode CacheCapacityhadoop.hdfs.datanode.CacheCapacityThe cache capacity of the DataNode
DataNode CacheUsedhadoop.hdfs.datanode.CacheUsedThe cache used the DataNode
DataNode Capacityhadoop.hdfs.datanode.CapacityCurrent raw capacity of the DataNodes in bytes
DataNode DfsUsedhadoop.hdfs.datanode.DfsUsedThe storage space that has been used up by HDFS.
DataNode NumBlocksCachedhadoop.hdfs.datanode.NumBlocksCachedThe number of blocks cached on the DataNode
DataNode NumBlocksFailedToCachehadoop.hdfs.datanode.NumBlocksFailedToCacheThe number of blocks that failed to cache on the DataNode
DataNode NumBlocksFailedToUncachehadoop.hdfs.datanode.NumBlocksFailedToUncacheThe number of failed blocks to remove from cache.
DataNode NumFailedVolumeshadoop.hdfs.datanode.NumFailedVolumesNumber of failed volumes.
DataNode Remaininghadoop.hdfs.datanode.RemainingThe remaining DataNode disk space left in Percent
YARN-MRAppMaster-MRAppMetrics
Metric nameMetric keyDescription
JobsCompletedhadoop.yarn.mrappmaster.JobsCompletedNumber of completed jobs
JobsFailedhadoop.yarn.mrappmaster.JobsFailedNumber of failed jobs
JobsKilledhadoop.yarn.mrappmaster.JobsKilledNumber of killed jobs
JobsPreparinghadoop.yarn.mrappmaster.JobsPreparingNumber of preparing jobs
JobsRunninghadoop.yarn.mrappmaster.JobsRunningNumber of running jobs
MapsCompletedhadoop.yarn.mrappmaster.MapsCompletedNumber of maps completed
MapsFailedhadoop.yarn.mrappmaster.MapsFailedNumber of maps failed
MapsKilledhadoop.yarn.mrappmaster.MapsKilledNumber of maps killed
MapsRunninghadoop.yarn.mrappmaster.MapsRunningNumber of maps running
MapsWaitinghadoop.yarn.mrappmaster.MapsWaitingNumber of maps waiting
ReducesCompletedhadoop.yarn.mrappmaster.ReducesCompletedNumber of completed reduces
ReducesFailedhadoop.yarn.mrappmaster.ReducesFailedNumber of failed reduces
ReducesKilledhadoop.yarn.mrappmaster.ReducesKilledNumber of killed reduces
ReducesRunninghadoop.yarn.mrappmaster.ReducesRunningNumber of running reduces
ReducesWaitinghadoop.yarn.mrappmaster.ReducesWaitingNumber of waiting reduces
HDFS-DataNode-*
Metric nameMetric keyDescription
DataNode BlocksReadhadoop.hdfs.datanode.BlocksRead.countTotal number of blocks read from DataNode
DataNode BlocksRemovedhadoop.hdfs.datanode.BlocksRemoved.countTotal number of blocks removed from DataNode
DataNode BlocksReplicatedhadoop.hdfs.datanode.BlocksReplicated.countTotal number of blocks replicated
DataNode BlocksVerifiedhadoop.hdfs.datanode.BlocksVerified.countTotal number of blocks verified
DataNode BlocksWrittenhadoop.hdfs.datanode.BlocksWritten.countTotal number of blocks written to DataNode
DataNode BytesReadhadoop.hdfs.datanode.BytesRead.countTotal number of bytes read from DataNode
DataNode BytesWrittenhadoop.hdfs.datanode.BytesWritten.countTotal number of bytes written to DataNode
HDFS-NameNode-FSNamesystemState
Metric nameMetric keyDescription
FilesTotalhadoop.hdfs.namenode.FilesTotalCurrent number of files and directories
PendingReplicationBlockshadoop.hdfs.namenode.PendingReplicationBlocksCurrent number of blocks pending to be replicated
UnderReplicatedBlockshadoop.hdfs.namenode.UnderReplicatedBlocksCurrent number of blocks under replicated
ScheduledReplicationBlockshadoop.hdfs.namenode.ScheduledReplicationBlocksCurrent number of blocks scheduled for replications
NumLiveDataNodeshadoop.hdfs.namenode.NumLiveDataNodesNumber of datanodes which are currently live
NumDeadDataNodeshadoop.hdfs.namenode.NumDeadDataNodesNumber of datanodes which are currently dead
NumDecomLiveDataNodeshadoop.hdfs.namenode.NumDecomLiveDataNodesNumber of datanodes which have been decommissioned and are now live
NumDecomDeadDataNodeshadoop.hdfs.namenode.NumDecomDeadDataNodesNumber of datanodes which have been decommissioned and are now dead
VolumeFailuresTotalhadoop.hdfs.namenode.VolumeFailuresTotalTotal number of volume failures across all Datanodes
EstimatedCapacityLostTotalhadoop.hdfs.namenode.EstimatedCapacityLostTotalAn estimate of the total capacity lost due to volume failures
NumDecommissioningDataNodeshadoop.hdfs.namenode.NumDecommissioningDataNodesNumber of datanodes in decommissioning state
NumStaleDataNodeshadoop.hdfs.namenode.NumStaleDataNodesNumber of datanodes marked stale due to delayed hearbeat.
HDFS-NameNode-FSNamesystem
Metric nameMetric keyDescription
CapacityTotalhadoop.hdfs.namenode.CapacityTotalCurrent raw capacity of DataNodes in bytes
CapacityUsedhadoop.hdfs.namenode.CapacityUsedCurrent used capacity across all DataNodes in bytes
CapacityRemaininghadoop.hdfs.namenode.CapacityRemainingCurrent remaining capacity in bytes
TotalLoadhadoop.hdfs.namenode.TotalLoadCurrent number of connections
BlocksTotalhadoop.hdfs.namenode.BlocksTotalCurrent number of allocated blocks in the system
PendingDeletionBlockshadoop.hdfs.namenode.PendingDeletionBlocksCurrent number of blocks pending deletion
CorruptBlockshadoop.hdfs.namenode.CorruptBlocksCurrent number of blocks with corrupt replicas.
CapacityUsedNonDFShadoop.hdfs.namenode.CapacityUsedNonDFSCurrent space used by DataNodes for non DFS purposes in bytes
Hub

Explore in Dynatrace Hub

Enhanced insights for Hadoop HDFS and Yarn services

Related tags
ComputeJMXDistributed computingApacheInfrastructure Observability