Amazon EMR (Elastic MapReduce) monitoring
- How-to guide
- 12-min read
- Published Oct 15, 2020
Dynatrace ingests metrics for multiple preselected namespaces, including Amazon EMR. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.
Prerequisites
To enable monitoring for this service, you need
-
ActiveGate version 1.181+, as follows:
-
For Dynatrace SaaS deployments, you need an Environment ActiveGate or a Multi-environment ActiveGate.
-
For Dynatrace Managed deployments, you can use any kind of ActiveGate.
For role-based access (whether in a SaaS or Managed deployment), you need an Environment ActiveGate installed on an Amazon EC2 host.
-
-
Dynatrace version 1.182+
-
An updated AWS monitoring policy to include the additional AWS services.
To update the AWS IAM policy, use the JSON below, containing the monitoring policy (permissions) for all supporting services.
{"Version": "2012-10-17","Statement": [{"Sid": "VisualEditor0","Effect": "Allow","Action": ["acm-pca:ListCertificateAuthorities","apigateway:GET","apprunner:ListServices","appstream:DescribeFleets","appsync:ListGraphqlApis","athena:ListWorkGroups","autoscaling:DescribeAutoScalingGroups","cloudformation:ListStackResources","cloudfront:ListDistributions","cloudhsm:DescribeClusters","cloudsearch:DescribeDomains","cloudwatch:GetMetricData","cloudwatch:GetMetricStatistics","cloudwatch:ListMetrics","codebuild:ListProjects","datasync:ListTasks","dax:DescribeClusters","directconnect:DescribeConnections","dms:DescribeReplicationInstances","dynamodb:ListTables","dynamodb:ListTagsOfResource","ec2:DescribeAvailabilityZones","ec2:DescribeInstances","ec2:DescribeNatGateways","ec2:DescribeSpotFleetRequests","ec2:DescribeTransitGateways","ec2:DescribeVolumes","ec2:DescribeVpnConnections","ecs:ListClusters","eks:ListClusters","elasticache:DescribeCacheClusters","elasticbeanstalk:DescribeEnvironmentResources","elasticbeanstalk:DescribeEnvironments","elasticfilesystem:DescribeFileSystems","elasticloadbalancing:DescribeInstanceHealth","elasticloadbalancing:DescribeListeners","elasticloadbalancing:DescribeLoadBalancers","elasticloadbalancing:DescribeRules","elasticloadbalancing:DescribeTags","elasticloadbalancing:DescribeTargetHealth","elasticmapreduce:ListClusters","elastictranscoder:ListPipelines","es:ListDomainNames","events:ListEventBuses","firehose:ListDeliveryStreams","fsx:DescribeFileSystems","gamelift:ListFleets","glue:GetJobs","inspector:ListAssessmentTemplates","kafka:ListClusters","kinesis:ListStreams","kinesisanalytics:ListApplications","kinesisvideo:ListStreams","lambda:ListFunctions","lambda:ListTags","lex:GetBots","logs:DescribeLogGroups","mediaconnect:ListFlows","mediaconvert:DescribeEndpoints","mediapackage-vod:ListPackagingConfigurations","mediapackage:ListChannels","mediatailor:ListPlaybackConfigurations","opsworks:DescribeStacks","qldb:ListLedgers","rds:DescribeDBClusters","rds:DescribeDBInstances","rds:DescribeEvents","rds:ListTagsForResource","redshift:DescribeClusters","robomaker:ListSimulationJobs","route53:ListHostedZones","route53resolver:ListResolverEndpoints","s3:ListAllMyBuckets","sagemaker:ListEndpoints","sns:ListTopics","sqs:ListQueues","storagegateway:ListGateways","sts:GetCallerIdentity","swf:ListDomains","tag:GetResources","tag:GetTagKeys","transfer:ListServers","workmail:ListOrganizations","workspaces:DescribeWorkspaces"],"Resource": "*"}]}
If you don't want to add permissions to all services, and just select permissions for certain services, consult the table below. The table contains a set of permissions that are required for All AWS cloud services and, for each supporting service, a list of optional permissions specific to that service.
Permissions required for AWS monitoring integration:
"cloudwatch:GetMetricData""cloudwatch:GetMetricStatistics""cloudwatch:ListMetrics""sts:GetCallerIdentity""tag:GetResources""tag:GetTagKeys""ec2:DescribeAvailabilityZones"
| Name | Permissions |
|---|---|
| All monitored Amazon services required | cloudwatch:GetMetricData,cloudwatch:GetMetricStatistics,cloudwatch:ListMetrics,sts:GetCallerIdentity,tag:GetResources,tag:GetTagKeys,ec2:DescribeAvailabilityZones |
| AWS Certificate Manager Private Certificate Authority | acm-pca:ListCertificateAuthorities |
| Amazon MQ | |
| Amazon API Gateway | apigateway:GET |
| AWS App Runner | apprunner:ListServices |
| Amazon AppStream | appstream:DescribeFleets |
| AWS AppSync | appsync:ListGraphqlApis |
| Amazon Athena | athena:ListWorkGroups |
| Amazon Aurora | rds:DescribeDBClusters |
| Amazon EC2 Auto Scaling | autoscaling:DescribeAutoScalingGroups |
| Amazon EC2 Auto Scaling (built-in) | autoscaling:DescribeAutoScalingGroups |
| AWS Billing | |
| Amazon Keyspaces | |
| AWS Chatbot | |
| Amazon CloudFront | cloudfront:ListDistributions |
| AWS CloudHSM | cloudhsm:DescribeClusters |
| Amazon CloudSearch | cloudsearch:DescribeDomains |
| AWS CodeBuild | codebuild:ListProjects |
| Amazon Cognito | |
| Amazon Connect | |
| Amazon Elastic Kubernetes Service (EKS) | eks:ListClusters |
| AWS DataSync | datasync:ListTasks |
| Amazon DynamoDB Accelerator (DAX) | dax:DescribeClusters |
| AWS Database Migration Service (AWS DMS) | dms:DescribeReplicationInstances |
| Amazon DocumentDB | rds:DescribeDBClusters |
| AWS Direct Connect | directconnect:DescribeConnections |
| Amazon DynamoDB | dynamodb:ListTables |
| Amazon DynamoDB (built-in) | dynamodb:ListTables,dynamodb:ListTagsOfResource |
| Amazon EBS | ec2:DescribeVolumes |
| Amazon EBS (built-in) | ec2:DescribeVolumes |
| Amazon EC2 API | |
| Amazon EC2 (built-in) | ec2:DescribeInstances |
| Amazon EC2 Spot Fleet | ec2:DescribeSpotFleetRequests |
| Amazon Elastic Container Service (ECS) | ecs:ListClusters |
| Amazon ECS Container Insights | ecs:ListClusters |
| Amazon ElastiCache (EC) | elasticache:DescribeCacheClusters |
| AWS Elastic Beanstalk | elasticbeanstalk:DescribeEnvironments |
| Amazon Elastic File System (EFS) | elasticfilesystem:DescribeFileSystems |
| Amazon Elastic Inference | |
| Amazon Elastic Map Reduce (EMR) | elasticmapreduce:ListClusters |
| Amazon Elasticsearch Service (ES) | es:ListDomainNames |
| Amazon Elastic Transcoder | elastictranscoder:ListPipelines |
| Amazon Elastic Load Balancer (ELB) (built-in) | elasticloadbalancing:DescribeInstanceHealth,elasticloadbalancing:DescribeListeners,elasticloadbalancing:DescribeLoadBalancers,elasticloadbalancing:DescribeRules,elasticloadbalancing:DescribeTags,elasticloadbalancing:DescribeTargetHealth |
| Amazon EventBridge | events:ListEventBuses |
| Amazon FSx | fsx:DescribeFileSystems |
| Amazon GameLift | gamelift:ListFleets |
| AWS Glue | glue:GetJobs |
| Amazon Inspector | inspector:ListAssessmentTemplates |
| AWS Internet of Things (IoT) | |
| AWS IoT Analytics | |
| Amazon Managed Streaming for Kafka | kafka:ListClusters |
| Amazon Kinesis Data Analytics | kinesisanalytics:ListApplications |
| Amazon Data Firehose | firehose:ListDeliveryStreams |
| Amazon Kinesis Data Streams | kinesis:ListStreams |
| Amazon Kinesis Video Streams | kinesisvideo:ListStreams |
| AWS Lambda | lambda:ListFunctions |
| AWS Lambda (built-in) | lambda:ListFunctions,lambda:ListTags |
| Amazon Lex | lex:GetBots |
| Amazon Application and Network Load Balancer (built-in) | elasticloadbalancing:DescribeInstanceHealth,elasticloadbalancing:DescribeListeners,elasticloadbalancing:DescribeLoadBalancers,elasticloadbalancing:DescribeRules,elasticloadbalancing:DescribeTags,elasticloadbalancing:DescribeTargetHealth |
| Amazon CloudWatch Logs | logs:DescribeLogGroups |
| AWS Elemental MediaConnect | mediaconnect:ListFlows |
| AWS Elemental MediaConvert | mediaconvert:DescribeEndpoints |
| AWS Elemental MediaPackage Live | mediapackage:ListChannels |
| AWS Elemental MediaPackage Video on Demand | mediapackage-vod:ListPackagingConfigurations |
| AWS Elemental MediaTailor | mediatailor:ListPlaybackConfigurations |
| Amazon VPC NAT Gateways | ec2:DescribeNatGateways |
| Amazon Neptune | rds:DescribeDBClusters |
| AWS OpsWorks | opsworks:DescribeStacks |
| Amazon Polly | |
| Amazon QLDB | qldb:ListLedgers |
| Amazon RDS | rds:DescribeDBInstances |
| Amazon RDS (built-in) | rds:DescribeDBInstances,rds:DescribeEvents,rds:ListTagsForResource |
| Amazon Redshift | redshift:DescribeClusters |
| Amazon Rekognition | |
| AWS RoboMaker | robomaker:ListSimulationJobs |
| Amazon Route 53 | route53:ListHostedZones |
| Amazon Route 53 Resolver | route53resolver:ListResolverEndpoints |
| Amazon S3 | s3:ListAllMyBuckets |
| Amazon S3 (built-in) | s3:ListAllMyBuckets |
| Amazon SageMaker Batch Transform Jobs | |
| Amazon SageMaker Endpoint Instances | sagemaker:ListEndpoints |
| Amazon SageMaker Endpoints | sagemaker:ListEndpoints |
| Amazon SageMaker Ground Truth | |
| Amazon SageMaker Processing Jobs | |
| Amazon SageMaker Training Jobs | |
| AWS Service Catalog | |
| Amazon Simple Email Service (SES) | |
| Amazon Simple Notification Service (SNS) | sns:ListTopics |
| Amazon Simple Queue Service (SQS) | sqs:ListQueues |
| AWS Systems Manager - Run Command | |
| AWS Step Functions | |
| AWS Storage Gateway | storagegateway:ListGateways |
| Amazon SWF | swf:ListDomains |
| Amazon Textract | |
| AWS IoT Things Graph | |
| AWS Transfer Family | transfer:ListServers |
| AWS Transit Gateway | ec2:DescribeTransitGateways |
| Amazon Translate | |
| AWS Trusted Advisor | |
| AWS API Usage | |
| AWS Site-to-Site VPN | ec2:DescribeVpnConnections |
| AWS WAF Classic | |
| AWS WAF | |
| Amazon WorkMail | workmail:ListOrganizations |
| Amazon WorkSpaces | workspaces:DescribeWorkspaces |
Example of JSON policy for one single service.
{"Version": "2012-10-17","Statement": [{"Sid": "VisualEditor0","Effect": "Allow","Action": ["apigateway:GET","cloudwatch:GetMetricData","cloudwatch:GetMetricStatistics","cloudwatch:ListMetrics","sts:GetCallerIdentity","tag:GetResources","tag:GetTagKeys","ec2:DescribeAvailabilityZones"],"Resource": "*"}]}
In this example, from the complete list of permissions you need to select
"apigateway:GET"for Amazon API Gateway"cloudwatch:GetMetricData","cloudwatch:GetMetricStatistics","cloudwatch:ListMetrics","sts:GetCallerIdentity","tag:GetResources","tag:GetTagKeys", and"ec2:DescribeAvailabilityZones"for All AWS cloud services.
Enable monitoring
To learn how to enable service monitoring, see Enable service monitoring.
View service metrics
You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.
View metrics on the custom device overview page
To access the custom device overview page
- Go to Technologies & Processes Classic.
- Filter by service name and select the relevant custom device group.
- Once you select the custom device group, you're on the custom device group overview page.
- The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.
View metrics on your dashboard
You can also view metrics in the Dynatrace web UI on dashboards. There is no preset dashboard available for this service, but you can create your own dashboard.
To check the availability of preset dashboards for each AWS service, see the list below.
| AWS service | Preset dashboard |
|---|---|
| Amazon EC2 Auto Scaling (built-in) |  |
| AWS Lambda (built-in) | |
| Amazon Application and Network Load Balancer (built-in) | |
| Amazon DynamoDB (built-in) | |
| Amazon EBS (built-in) | |
| Amazon EC2 (built-in) | |
| Amazon Elastic Load Balancer (ELB) (built-in) | |
| Amazon RDS (built-in) | |
| Amazon S3 (built-in) | |
| AWS Certificate Manager Private Certificate Authority | |
| All monitored Amazon services | |
| Amazon API Gateway | |
| AWS App Runner | |
| Amazon AppStream |  |
| AWS AppSync | |
| Amazon Athena | |
| Amazon Aurora | |
| Amazon EC2 Auto Scaling | |
| AWS Billing | |
| Amazon Keyspaces | |
| AWS Chatbot | |
| Amazon CloudFront | |
| AWS CloudHSM | |
| Amazon CloudSearch | |
| AWS CodeBuild | |
| Amazon Cognito | |
| Amazon Connect | |
| AWS DataSync | |
| Amazon DynamoDB Accelerator (DAX) | |
| AWS Database Migration Service (AWS DMS) | |
| Amazon DocumentDB | |
| AWS Direct Connect | |
| Amazon DynamoDB | |
| Amazon EBS | |
| Amazon EC2 Spot Fleet | |
| Amazon EC2 API | |
| Amazon Elastic Container Service (ECS) | |
| Amazon ECS Container Insights | |
| Amazon Elastic File System (EFS) | |
| Amazon Elastic Kubernetes Service (EKS) | |
| Amazon ElastiCache (EC) | |
| AWS Elastic Beanstalk | |
| Amazon Elastic Inference | |
| Amazon Elastic Transcoder | |
| Amazon Elastic Map Reduce (EMR) | |
| Amazon Elasticsearch Service (ES) | |
| Amazon EventBridge | |
| Amazon FSx | |
| Amazon GameLift | |
| AWS Glue | |
| Amazon Inspector | |
| AWS Internet of Things (IoT) | |
| AWS IoT Things Graph | |
| AWS IoT Analytics | |
| Amazon Managed Streaming for Kafka | |
| Amazon Kinesis Data Analytics | |
| Amazon Data Firehose | |
| Amazon Kinesis Data Streams | |
| Amazon Kinesis Video Streams | |
| AWS Lambda | |
| Amazon Lex | |
| Amazon CloudWatch Logs | |
| AWS Elemental MediaTailor | |
| AWS Elemental MediaConnect | |
| AWS Elemental MediaConvert | |
| AWS Elemental MediaPackage Live | |
| AWS Elemental MediaPackage Video on Demand | |
| Amazon MQ | |
| Amazon VPC NAT Gateways | |
| Amazon Neptune | |
| AWS OpsWorks | |
| Amazon Polly | |
| Amazon QLDB | |
| Amazon RDS | |
| Amazon Redshift | |
| Amazon Rekognition | |
| AWS RoboMaker | |
| Amazon Route 53 | |
| Amazon Route 53 Resolver | |
| Amazon S3 | |
| Amazon SageMaker Batch Transform Jobs | |
| Amazon SageMaker Endpoints | |
| Amazon SageMaker Endpoint Instances | |
| Amazon SageMaker Ground Truth | |
| Amazon SageMaker Processing Jobs | |
| Amazon SageMaker Training Jobs | |
| AWS Service Catalog | |
| Amazon Simple Email Service (SES) | |
| Amazon Simple Notification Service (SNS) | |
| Amazon Simple Queue Service (SQS) | |
| AWS Systems Manager - Run Command | |
| AWS Step Functions | |
| AWS Storage Gateway | |
| Amazon SWF | |
| Amazon Textract | |
| AWS Transfer Family | |
| AWS Transit Gateway | |
| Amazon Translate | |
| AWS Trusted Advisor | |
| AWS API Usage | |
| AWS Site-to-Site VPN | |
| AWS WAF Classic | |
| AWS WAF | |
| Amazon WorkMail | |
| Amazon WorkSpaces | |
Available metrics
JobFlowId is the main dimension.
| Name | Description | Unit | Statistics | Dimensions | Recommended |
|---|---|---|---|---|---|
| AppsCompleted | The number of applications submitted to YARN that have completed | Count | Sum | JobFlowId, JobId | |
| AppsCompleted | Count | Sum | JobFlowId | ||
| AppsFailed | The number of applications submitted to YARN that have failed to complete | Count | Sum | JobFlowId, JobId | |
| AppsFailed | Count | Sum | JobFlowId | ||
| AppsKilled | The number of applications submitted to YARN that have been killed | Count | Sum | JobFlowId, JobId | |
| AppsKilled | Count | Sum | JobFlowId | ||
| AppsPending | The number of applications submitted to YARN that are in a Pending state | Count | Sum | JobFlowId, JobId | |
| AppsPending | Count | Sum | JobFlowId | ||
| AppsRunning | The number of applications submitted to YARN that are running | Count | Sum | JobFlowId, JobId | |
| AppsRunning | Count | Sum | JobFlowId | | |
| AppsSubmitted | The number of applications submitted to YARN | Count | Sum | JobFlowId, JobId | |
| AppsSubmitted | Count | Sum | JobFlowId | ||
| BackupFailed | Shows if the last backup failed. Set to 0 by default and updated to 1 if the previous backup attempt failed. This metric is only reported for HBase clusters. | Count | Sum | JobFlowId, JobId | |
| BackupFailed | Count | Sum | JobFlowId | ||
| CapacityRemainingGB | The amount of remaining HDFS disk capacity | Count | Sum | JobFlowId, JobId | |
| CapacityRemainingGB | Count | Sum | JobFlowId | ||
| ContainerAllocated | The number of resource containers allocated by the resource manager | Count | Sum | JobFlowId, JobId | |
| ContainerAllocated | Count | Sum | JobFlowId | ||
| ContainerPendingRatio | The ratio (in numbers) of pending containers to containers allocated (ContainerPendingRatio = ContainerPending / ContainerAllocated). If ContainerAllocated = 0, then ContainerPendingRatio = ContainerPending. | Count | Sum | JobFlowId, JobId | |
| ContainerPendingRatio | Count | Sum | JobFlowId | ||
| ContainerPending | The number of containers in the queue that have not yet been allocated | Count | Sum | JobFlowId, JobId | |
| ContainerPending | Count | Sum | JobFlowId | ||
| ContainerReserved | The number of containers reserved | Count | Sum | JobFlowId, JobId | |
| ContainerReserved | Count | Sum | JobFlowId | ||
| CoreNodesPending | The number of core nodes waiting to be assigned (pending requests) | Count | Sum | JobFlowId, JobId | |
| CoreNodesPending | Count | Sum | JobFlowId | ||
| CoreNodesRequested | Count | Sum | JobFlowId, JobId | ||
| CoreNodesRequested | Count | Sum | JobFlowId | ||
| CoreNodesRunning | The number of working core nodes | Count | Sum | JobFlowId, JobId | |
| CoreNodesRunning | Count | Sum | JobFlowId | ||
| CoreUnitsRequested | Count | Sum | JobFlowId, JobId | ||
| CoreUnitsRequested | Count | Sum | JobFlowId | ||
| CoreUnitsRunning | Count | Sum | JobFlowId, JobId | ||
| CoreUnitsRunning | Count | Sum | JobFlowId | ||
| CoreVCPURequested | Count | Sum | JobFlowId, JobId | ||
| CoreVCPURequested | Count | Sum | JobFlowId | ||
| CoreVCPURunning | Count | Sum | JobFlowId, JobId | ||
| CoreVCPURunning | Count | Sum | JobFlowId | ||
| CorruptBlocks | The number of blocks that HDFS reports as corrupted | Count | Sum | JobFlowId, JobId | |
| CorruptBlocks | Count | Sum | JobFlowId | ||
| DfsPendingReplicationBlocks | The status of block replication: blocks being replicated, age of replication requests, and unsuccessful replication requests | Count | Sum | JobFlowId, JobId | |
| DfsPendingReplicationBlocks | Count | Sum | JobFlowId | ||
| HDFSBytesRead | The number of bytes read from HDFS | Count | Sum | JobFlowId, JobId | |
| HDFSBytesRead | Count | Sum | JobFlowId | ||
| HDFSBytesWritten | The number of bytes written to HDFS | Count | Sum | JobFlowId, JobId | |
| HDFSBytesWritten | Count | Sum | JobFlowId | ||
| HDFSUtilization | The percentage of HDFS storage currently used | Percent | Average | JobFlowId, JobId | |
| HDFSUtilization | Percent | Average | JobFlowId | | |
| HbaseBackupFailed | Shows if the last backup failed. Set to 0 by default and updated to 1 if the previous backup attempt failed. This metric is only reported for HBase clusters. | Count | Minimum | JobFlowId, JobId | |
| HbaseBackupFailed | Count | Minimum | JobFlowId | ||
| IsIdle | Indicates that a cluster is no longer performing work, but is still alive and accruing charges. Set to 1 if no tasks are running and no jobs are running, and to 0 otherwise. This value is checked at five-minute intervals and a value of 1 indicates only that the cluster was idle when checked, not that it was idle for the entire five minutes. | Count | Minimum | JobFlowId, JobId | |
| IsIdle | Count | Minimum | JobFlowId | | |
| JobsFailed | The number of jobs in the cluster that have failed | Count | Sum | JobFlowId, JobId | |
| JobsFailed | Count | Sum | JobFlowId | ||
| JobsRunning | The number of jobs in the cluster that are currently running | Count | Sum | JobFlowId, JobId | |
| JobsRunning | Count | Sum | JobFlowId | ||
| LiveDataNodes | The percentage of data nodes that are receiving work from Hadoop | Count | Sum | JobFlowId, JobId | |
| LiveDataNodes | Count | Sum | JobFlowId | ||
| LiveTaskTrackers | The percentage of task trackers that are functional | Percent | Average | JobFlowId, JobId | |
| LiveTaskTrackers | Percent | Average | JobFlowId | ||
| MRActiveNodes | The number of nodes presently running MapReduce tasks or jobs. Equivalent to YARN metric mapred.resourcemanager.NoOfActiveNodes | Count | Sum | JobFlowId, JobId | |
| MRActiveNodes | Count | Sum | JobFlowId | ||
| MRDecommissionedNodes | The number of nodes allocated to MapReduce applications that have been marked in a Decommissioned state | Count | Sum | JobFlowId, JobId | |
| MRDecommissionedNodes | Count | Sum | JobFlowId | ||
| MRLostNodes | The number of nodes allocated to MapReduce that have been marked in a Lost state | Count | Sum | JobFlowId, JobId | |
| MRLostNodes | Count | Sum | JobFlowId | ||
| MRRebootedNodes | The number of nodes available to MapReduce that have been rebooted and marked in a Rebooted state | Count | Sum | JobFlowId, JobId | |
| MRRebootedNodes | Count | Sum | JobFlowId | ||
| MRTotalNodes | The number of nodes presently available to MapReduce jobs | Count | Sum | JobFlowId, JobId | |
| MRTotalNodes | Count | Sum | JobFlowId | ||
| MRUnhealthyNodes | The number of nodes available to MapReduce jobs marked in an Unhealthy state | Count | Sum | JobFlowId, JobId | |
| MRUnhealthyNodes | Count | Sum | JobFlowId | ||
| MapSlotsOpen | The unused map task capacity. This is calculated as the maximum number of map tasks for a given cluster, less the total number of map tasks currently running in that cluster. | Count | Sum | JobFlowId, JobId | |
| MapSlotsOpen | Count | Sum | JobFlowId | ||
| MapTasksRemaining | The number of remaining map tasks for each job | Count | Sum | JobFlowId, JobId | |
| MapTasksRemaining | Count | Sum | JobFlowId | ||
| MapTasksRunning | The number of running map tasks for each job | Count | Sum | JobFlowId, JobId | |
| MapTasksRunning | Count | Sum | JobFlowId | | |
| MemoryAllocatedMB | The amount of memory allocated to the cluster | Count | Sum | JobFlowId, JobId | |
| MemoryAllocatedMB | Count | Sum | JobFlowId | ||
| MemoryAvailableMB | The amount of memory available for allocation | Count | Sum | JobFlowId, JobId | |
| MemoryAvailableMB | Count | Sum | JobFlowId | ||
| MemoryReservedMB | The amount of memory reserved for allocation | Count | Sum | JobFlowId, JobId | |
| MemoryReservedMB | Count | Sum | JobFlowId | ||
| MemoryTotalMB | The total amount of memory in the cluster | Count | Sum | JobFlowId, JobId | |
| MemoryTotalMB | Count | Sum | JobFlowId | ||
| MissingBlocks | The number of blocks in which HDFS has no replicas. These might be corrupt blocks. | Count | Sum | JobFlowId, JobId | |
| MissingBlocks | Count | Sum | JobFlowId | ||
| MostRecentBackupDuration | The amount of time it took the previous backup to complete. This metric is set regardless of whether the last completed backup succeeded or failed. While the backup is ongoing, this metric returns the number of minutes after the backup started. This metric is only reported for HBase clusters. | Count | Sum | JobFlowId, JobId | |
| MostRecentBackupDuration | Count | Sum | JobFlowId | ||
| PendingDeletionBlocks | The number of blocks marked for deletion | Count | Sum | JobFlowId, JobId | |
| PendingDeletionBlocks | Count | Sum | JobFlowId | ||
| ReduceSlotsOpen | Unused reduce task capacity. This is calculated as the maximum reduce task capacity for a given cluster, less the number of reduce tasks currently running in that cluster. | Count | Sum | JobFlowId, JobId | |
| ReduceSlotsOpen | Count | Sum | JobFlowId | ||
| ReduceTasksRemaining | The number of remaining reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. | Count | Sum | JobFlowId | |
| ReduceTasksRunning | The number of running reduce tasks for each job. If you have a scheduler installed and multiple jobs running, multiple graphs are generated. | Count | Sum | JobFlowId, JobId | |
| ReduceTasksRunning | Count | Sum | JobFlowId | ||
| RemainingMapTasksPerSlot | The ratio of the total map tasks remaining to the total map slots available in the cluster | Percent | Average | JobFlowId, JobId | |
| RemainingMapTasksPerSlot | Percent | Average | JobFlowId | ||
| S3BytesRead | The number of bytes read from Amazon S3. This metric aggregates MapReduce jobs only, and does not apply for other workloads on EMR. | Count | Sum | JobFlowId, JobId | |
| S3BytesRead | Count | Sum | JobFlowId | ||
| S3BytesWritten | The number of bytes written to Amazon S3. This metric aggregates MapReduce jobs only, and does not apply for other workloads on EMR. | Count | Sum | JobFlowId, JobId | |
| S3BytesWritten | Count | Sum | JobFlowId | ||
| TaskNodesPending | The number of task nodes waiting to be assigned (pending requests) | Count | Sum | JobFlowId, JobId | |
| TaskNodesPending | Count | Sum | JobFlowId | ||
| TaskNodesRequested | Count | Sum | JobFlowId, JobId | ||
| TaskNodesRequested | Count | Sum | JobFlowId | ||
| TaskNodesRunning | The number of working task nodes | Count | Sum | JobFlowId, JobId | |
| TaskNodesRunning | Count | Sum | JobFlowId | ||
| TaskUnitsRequested | Count | Sum | JobFlowId, JobId | ||
| TaskUnitsRequested | Count | Sum | JobFlowId | ||
| TaskUnitsRunning | Count | Sum | JobFlowId, JobId | ||
| TaskUnitsRunning | Count | Sum | JobFlowId | ||
| TaskVCPURequested | Count | Sum | JobFlowId, JobId | ||
| TaskVCPURequested | Count | Sum | JobFlowId | ||
| TaskVCPURunning | Count | Sum | JobFlowId, JobId | ||
| TaskVCPURunning | Count | Sum | JobFlowId | ||
| TimeSinceLastSuccessfulBackup | The number of elapsed minutes after the last successful HBase backup started on your cluster. This metric is only reported for HBase clusters. | Count | Sum | JobFlowId, JobId | |
| TimeSinceLastSuccessfulBackup | Count | Sum | JobFlowId | ||
| TotalLoad | The total number of concurrent data transfers | Count | Sum | JobFlowId, JobId | |
| TotalLoad | Count | Sum | JobFlowId | ||
| TotalNodesRequested | Count | Sum | JobFlowId, JobId | ||
| TotalNodesRequested | Count | Sum | JobFlowId | ||
| TotalNodesRunning | Count | Sum | JobFlowId, JobId | ||
| TotalNodesRunning | Count | Sum | JobFlowId | ||
| TotalUnitsRequested | Count | Sum | JobFlowId, JobId | ||
| TotalUnitsRequested | Count | Sum | JobFlowId | ||
| TotalUnitsRunning | Count | Sum | JobFlowId, JobId | ||
| TotalUnitsRunning | Count | Sum | JobFlowId | ||
| TotalVCPURequested | Count | Sum | JobFlowId | ||
| TotalVCPURunning | Count | Sum | JobFlowId, JobId | ||
| TotalVCPURunning | Count | Sum | JobFlowId | ||
| UnderReplicatedBlocks | The number of blocks that need to be replicated one or more times | Count | Sum | JobFlowId, JobId | |
| UnderReplicatedBlocks | Count | Sum | JobFlowId | ||
| YARNMemoryAvailablePercentage | he percentage of remaining memory available to YARN (YARNMemoryAvailablePercentage = MemoryAvailableMB / MemoryTotalMB) | Percent | Average | JobFlowId, JobId | |
| YARNMemoryAvailablePercentage | Percent | Average | JobFlowId |
Related tags
Infrastructure Observability