AWS Glue monitoring

Dynatrace ingests metrics for multiple preselected namespaces, including AWS Glue. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.

Prerequisites

To enable monitoring for this service, you need

ActiveGate version 1.181+, as follows:
- For Dynatrace SaaS deployments, you need an Environment ActiveGate or a Multi-environment ActiveGate.
- For Dynatrace Managed deployments, you can use any kind of ActiveGate.
  
  For role-based access (whether in a SaaS or Managed deployment), you need an Environment ActiveGate installed on an Amazon EC2 host.
Dynatrace version 1.182+
An updated AWS monitoring policy to include the additional AWS services.
To update the AWS IAM policy, use the JSON below, containing the monitoring policy (permissions) for all supporting services.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "acm-pca:ListCertificateAuthorities",
        "apigateway:GET",
        "apprunner:ListServices",
        "appstream:DescribeFleets",
        "appsync:ListGraphqlApis",
        "athena:ListWorkGroups",
        "autoscaling:DescribeAutoScalingGroups",
        "cloudformation:ListStackResources",
        "cloudfront:ListDistributions",
        "cloudhsm:DescribeClusters",
        "cloudsearch:DescribeDomains",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "codebuild:ListProjects",
        "datasync:ListTasks",
        "dax:DescribeClusters",
        "directconnect:DescribeConnections",
        "dms:DescribeReplicationInstances",
        "dynamodb:ListTables",
        "dynamodb:ListTagsOfResource",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeNatGateways",
        "ec2:DescribeSpotFleetRequests",
        "ec2:DescribeTransitGateways",
        "ec2:DescribeVolumes",
        "ec2:DescribeVpnConnections",
        "ecs:ListClusters",
        "eks:ListClusters",
        "elasticache:DescribeCacheClusters",
        "elasticbeanstalk:DescribeEnvironmentResources",
        "elasticbeanstalk:DescribeEnvironments",
        "elasticfilesystem:DescribeFileSystems",
        "elasticloadbalancing:DescribeInstanceHealth",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeRules",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticmapreduce:ListClusters",
        "elastictranscoder:ListPipelines",
        "es:ListDomainNames",
        "events:ListEventBuses",
        "firehose:ListDeliveryStreams",
        "fsx:DescribeFileSystems",
        "gamelift:ListFleets",
        "glue:GetJobs",
        "inspector:ListAssessmentTemplates",
        "kafka:ListClusters",
        "kinesis:ListStreams",
        "kinesisanalytics:ListApplications",
        "kinesisvideo:ListStreams",
        "lambda:ListFunctions",
        "lambda:ListTags",
        "lex:GetBots",
        "logs:DescribeLogGroups",
        "mediaconnect:ListFlows",
        "mediaconvert:DescribeEndpoints",
        "mediapackage-vod:ListPackagingConfigurations",
        "mediapackage:ListChannels",
        "mediatailor:ListPlaybackConfigurations",
        "opsworks:DescribeStacks",
        "qldb:ListLedgers",
        "rds:DescribeDBClusters",
        "rds:DescribeDBInstances",
        "rds:DescribeEvents",
        "rds:ListTagsForResource",
        "redshift:DescribeClusters",
        "robomaker:ListSimulationJobs",
        "route53:ListHostedZones",
        "route53resolver:ListResolverEndpoints",
        "s3:ListAllMyBuckets",
        "sagemaker:ListEndpoints",
        "sns:ListTopics",
        "sqs:ListQueues",
        "storagegateway:ListGateways",
        "sts:GetCallerIdentity",
        "swf:ListDomains",
        "tag:GetResources",
        "tag:GetTagKeys",
        "transfer:ListServers",
        "workmail:ListOrganizations",
        "workspaces:DescribeWorkspaces"
      ],
      "Resource": "*"
    }
  ]
}

If you don't want to add permissions to all services, and just select permissions for certain services, consult the table below. The table contains a set of permissions that are required for All AWS cloud services and, for each supporting service, a list of optional permissions specific to that service.

Permissions required for AWS monitoring integration:

"cloudwatch:GetMetricData"
"cloudwatch:GetMetricStatistics"
"cloudwatch:ListMetrics"
"sts:GetCallerIdentity"
"tag:GetResources"
"tag:GetTagKeys"
"ec2:DescribeAvailabilityZones"

Name

Permissions

All monitored Amazon services required

cloudwatch:GetMetricData,
cloudwatch:GetMetricStatistics,
cloudwatch:ListMetrics,
sts:GetCallerIdentity,
tag:GetResources,
tag:GetTagKeys,
ec2:DescribeAvailabilityZones

AWS Certificate Manager Private Certificate Authority

acm-pca:ListCertificateAuthorities

Amazon MQ

Amazon API Gateway

apigateway:GET

AWS App Runner

apprunner:ListServices

Amazon AppStream

appstream:DescribeFleets

AWS AppSync

appsync:ListGraphqlApis

Amazon Athena

athena:ListWorkGroups

Amazon Aurora

rds:DescribeDBClusters

Amazon EC2 Auto Scaling

autoscaling:DescribeAutoScalingGroups

Amazon EC2 Auto Scaling (built-in)

autoscaling:DescribeAutoScalingGroups

AWS Billing

Amazon Keyspaces

AWS Chatbot

Amazon CloudFront

cloudfront:ListDistributions

AWS CloudHSM

cloudhsm:DescribeClusters

Amazon CloudSearch

cloudsearch:DescribeDomains

AWS CodeBuild

codebuild:ListProjects

Amazon Cognito

Amazon Connect

Amazon Elastic Kubernetes Service (EKS)

eks:ListClusters

AWS DataSync

datasync:ListTasks

Amazon DynamoDB Accelerator (DAX)

dax:DescribeClusters

AWS Database Migration Service (AWS DMS)

dms:DescribeReplicationInstances

Amazon DocumentDB

rds:DescribeDBClusters

AWS Direct Connect

directconnect:DescribeConnections

Amazon DynamoDB

dynamodb:ListTables

Amazon DynamoDB (built-in)

dynamodb:ListTables,
dynamodb:ListTagsOfResource

Amazon EBS

ec2:DescribeVolumes

Amazon EBS (built-in)

ec2:DescribeVolumes

Amazon EC2 API

Amazon EC2 (built-in)

ec2:DescribeInstances

Amazon EC2 Spot Fleet

ec2:DescribeSpotFleetRequests

Amazon Elastic Container Service (ECS)

ecs:ListClusters

Amazon ECS Container Insights

ecs:ListClusters

Amazon ElastiCache (EC)

elasticache:DescribeCacheClusters

AWS Elastic Beanstalk

elasticbeanstalk:DescribeEnvironments

Amazon Elastic File System (EFS)

elasticfilesystem:DescribeFileSystems

Amazon Elastic Inference

Amazon Elastic Map Reduce (EMR)

elasticmapreduce:ListClusters

Amazon Elasticsearch Service (ES)

es:ListDomainNames

Amazon Elastic Transcoder

elastictranscoder:ListPipelines

Amazon Elastic Load Balancer (ELB) (built-in)

elasticloadbalancing:DescribeInstanceHealth,
elasticloadbalancing:DescribeListeners,
elasticloadbalancing:DescribeLoadBalancers,
elasticloadbalancing:DescribeRules,
elasticloadbalancing:DescribeTags,
elasticloadbalancing:DescribeTargetHealth

Amazon EventBridge

events:ListEventBuses

Amazon FSx

fsx:DescribeFileSystems

Amazon GameLift

gamelift:ListFleets

AWS Glue

glue:GetJobs

Amazon Inspector

inspector:ListAssessmentTemplates

AWS Internet of Things (IoT)

AWS IoT Analytics

Amazon Managed Streaming for Kafka

kafka:ListClusters

Amazon Kinesis Data Analytics

kinesisanalytics:ListApplications

Amazon Data Firehose

firehose:ListDeliveryStreams

Amazon Kinesis Data Streams

kinesis:ListStreams

Amazon Kinesis Video Streams

kinesisvideo:ListStreams

AWS Lambda

lambda:ListFunctions

AWS Lambda (built-in)

lambda:ListFunctions,
lambda:ListTags

Amazon Lex

lex:GetBots

Amazon Application and Network Load Balancer (built-in)

Amazon CloudWatch Logs

logs:DescribeLogGroups

AWS Elemental MediaConnect

mediaconnect:ListFlows

AWS Elemental MediaConvert

mediaconvert:DescribeEndpoints

AWS Elemental MediaPackage Live

mediapackage:ListChannels

AWS Elemental MediaPackage Video on Demand

mediapackage-vod:ListPackagingConfigurations

AWS Elemental MediaTailor

mediatailor:ListPlaybackConfigurations

Amazon VPC NAT Gateways

ec2:DescribeNatGateways

Amazon Neptune

rds:DescribeDBClusters

AWS OpsWorks

opsworks:DescribeStacks

Amazon Polly

Amazon QLDB

qldb:ListLedgers

Amazon RDS

rds:DescribeDBInstances

Amazon RDS (built-in)

rds:DescribeDBInstances,
rds:DescribeEvents,
rds:ListTagsForResource

Amazon Redshift

redshift:DescribeClusters

Amazon Rekognition

AWS RoboMaker

robomaker:ListSimulationJobs

Amazon Route 53

route53:ListHostedZones

Amazon Route 53 Resolver

route53resolver:ListResolverEndpoints

Amazon S3

s3:ListAllMyBuckets

Amazon S3 (built-in)

s3:ListAllMyBuckets

Amazon SageMaker Batch Transform Jobs

Amazon SageMaker Endpoint Instances

sagemaker:ListEndpoints

Amazon SageMaker Endpoints

sagemaker:ListEndpoints

Amazon SageMaker Ground Truth

Amazon SageMaker Processing Jobs

Amazon SageMaker Training Jobs

AWS Service Catalog

Amazon Simple Email Service (SES)

Amazon Simple Notification Service (SNS)

sns:ListTopics

Amazon Simple Queue Service (SQS)

sqs:ListQueues

AWS Systems Manager - Run Command

AWS Step Functions

AWS Storage Gateway

storagegateway:ListGateways

Amazon SWF

swf:ListDomains

Amazon Textract

AWS IoT Things Graph

AWS Transfer Family

transfer:ListServers

AWS Transit Gateway

ec2:DescribeTransitGateways

Amazon Translate

AWS Trusted Advisor

AWS API Usage

AWS Site-to-Site VPN

ec2:DescribeVpnConnections

AWS WAF Classic

AWS WAF

Amazon WorkMail

workmail:ListOrganizations

Amazon WorkSpaces

workspaces:DescribeWorkspaces

Example of JSON policy for one single service.

{
  "Version": "2012-10-17",
  "Statement": [
          {
                  "Sid": "VisualEditor0",
                  "Effect": "Allow",
                  "Action": [
                          "apigateway:GET",
                          "cloudwatch:GetMetricData",
                          "cloudwatch:GetMetricStatistics",
                          "cloudwatch:ListMetrics",
                          "sts:GetCallerIdentity",
                          "tag:GetResources",
                          "tag:GetTagKeys",
                          "ec2:DescribeAvailabilityZones"
                  ],
                  "Resource": "*"
          }
      ]
}

In this example, from the complete list of permissions you need to select

"apigateway:GET" for Amazon API Gateway
"cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", and "ec2:DescribeAvailabilityZones" for All AWS cloud services.

Endpoint

Service

autoscaling.<REGION>.amazonaws.com

Amazon EC2 Auto Scaling (built-in), Amazon EC2 Auto Scaling

lambda.<REGION>.amazonaws.com

AWS Lambda (built-in), AWS Lambda

elasticloadbalancing.<REGION>.amazonaws.com

Amazon Application and Network Load Balancer (built-in), Amazon Elastic Load Balancer (ELB) (built-in)

dynamodb.<REGION>.amazonaws.com

Amazon DynamoDB (built-in), Amazon DynamoDB

ec2.<REGION>.amazonaws.com

Amazon EBS (built-in), Amazon EC2 (built-in), Amazon EBS, Amazon EC2 Spot Fleet, Amazon VPC NAT Gateways, AWS Transit Gateway, AWS Site-to-Site VPN

rds.<REGION>.amazonaws.com

Amazon RDS (built-in), Amazon Aurora, Amazon DocumentDB, Amazon Neptune, Amazon RDS

s3.<REGION>.amazonaws.com

Amazon S3 (built-in)

acm-pca.<REGION>.amazonaws.com

AWS Certificate Manager Private Certificate Authority

apigateway.<REGION>.amazonaws.com

Amazon API Gateway

apprunner.<REGION>.amazonaws.com

AWS App Runner

appstream2.<REGION>.amazonaws.com

Amazon AppStream

appsync.<REGION>.amazonaws.com

AWS AppSync

athena.<REGION>.amazonaws.com

Amazon Athena

cloudfront.amazonaws.com

Amazon CloudFront

cloudhsmv2.<REGION>.amazonaws.com

AWS CloudHSM

cloudsearch.<REGION>.amazonaws.com

Amazon CloudSearch

codebuild.<REGION>.amazonaws.com

AWS CodeBuild

datasync.<REGION>.amazonaws.com

AWS DataSync

dax.<REGION>.amazonaws.com

Amazon DynamoDB Accelerator (DAX)

dms.<REGION>.amazonaws.com

AWS Database Migration Service (AWS DMS)

directconnect.<REGION>.amazonaws.com

AWS Direct Connect

ecs.<REGION>.amazonaws.com

Amazon Elastic Container Service (ECS), Amazon ECS Container Insights

elasticfilesystem.<REGION>.amazonaws.com

Amazon Elastic File System (EFS)

eks.<REGION>.amazonaws.com

Amazon Elastic Kubernetes Service (EKS)

elasticache.<REGION>.amazonaws.com

Amazon ElastiCache (EC)

elasticbeanstalk.<REGION>.amazonaws.com

AWS Elastic Beanstalk

elastictranscoder.<REGION>.amazonaws.com

Amazon Elastic Transcoder

es.<REGION>.amazonaws.com

Amazon Elasticsearch Service (ES)

events.<REGION>.amazonaws.com

Amazon EventBridge

fsx.<REGION>.amazonaws.com

Amazon FSx

gamelift.<REGION>.amazonaws.com

Amazon GameLift

glue.<REGION>.amazonaws.com

AWS Glue

inspector.<REGION>.amazonaws.com

Amazon Inspector

kafka.<REGION>.amazonaws.com

Amazon Managed Streaming for Kafka

models.lex.<REGION>.amazonaws.com

Amazon Lex

logs.<REGION>.amazonaws.com

Amazon CloudWatch Logs

api.mediatailor.<REGION>.amazonaws.com

AWS Elemental MediaTailor

mediaconnect.<REGION>.amazonaws.com

AWS Elemental MediaConnect

mediapackage.<REGION>.amazonaws.com

AWS Elemental MediaPackage Live

mediapackage-vod.<REGION>.amazonaws.com

AWS Elemental MediaPackage Video on Demand

opsworks.<REGION>.amazonaws.com

AWS OpsWorks

qldb.<REGION>.amazonaws.com

Amazon QLDB

redshift.<REGION>.amazonaws.com

Amazon Redshift

robomaker.<REGION>.amazonaws.com

AWS RoboMaker

route53.amazonaws.com

Amazon Route 53

route53resolver.<REGION>.amazonaws.com

Amazon Route 53 Resolver

api.sagemaker.<REGION>.amazonaws.com

Amazon SageMaker Endpoints, Amazon SageMaker Endpoint Instances

sns.<REGION>.amazonaws.com

Amazon Simple Notification Service (SNS)

sqs.<REGION>.amazonaws.com

Amazon Simple Queue Service (SQS)

storagegateway.<REGION>.amazonaws.com

AWS Storage Gateway

swf.<REGION>.amazonaws.com

Amazon SWF

transfer.<REGION>.amazonaws.com

AWS Transfer Family

workmail.<REGION>.amazonaws.com

Amazon WorkMail

workspaces.<REGION>.amazonaws.com

Amazon WorkSpaces

Enable monitoring

To learn how to enable service monitoring, see Enable service monitoring.

View service metrics

You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.

View metrics on the custom device overview page

To access the custom device overview page

Go to Technologies & Processes or Technologies & Processes Classic (latest Dynatrace).
Filter by service name and select the relevant custom device group.
Once you select the custom device group, you're on the custom device group overview page.
The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.

View metrics on your dashboard

You can also view metrics in the Dynatrace web UI on dashboards. There is no preset dashboard available for this service, but you can create your own dashboard.

To check the availability of preset dashboards for each AWS service, see the list below.

AWS service

Preset dashboard

Amazon EC2 Auto Scaling (built-in)

AWS Lambda (built-in)

Amazon Application and Network Load Balancer (built-in)

Amazon DynamoDB (built-in)

Amazon EBS (built-in)

Amazon EC2 (built-in)

Amazon Elastic Load Balancer (ELB) (built-in)

Amazon RDS (built-in)

Amazon S3 (built-in)

AWS Certificate Manager Private Certificate Authority

All monitored Amazon services

Amazon API Gateway

AWS App Runner

Amazon AppStream

AWS AppSync

Amazon Athena

Amazon Aurora

Amazon EC2 Auto Scaling

AWS Billing

Amazon Keyspaces

AWS Chatbot

Amazon CloudFront

AWS CloudHSM

Amazon CloudSearch

AWS CodeBuild

Amazon Cognito

Amazon Connect

AWS DataSync

Amazon DynamoDB Accelerator (DAX)

AWS Database Migration Service (AWS DMS)

Amazon DocumentDB

AWS Direct Connect

Amazon DynamoDB

Amazon EBS

Amazon EC2 Spot Fleet

Amazon EC2 API

Amazon Elastic Container Service (ECS)

Amazon ECS Container Insights

Amazon Elastic File System (EFS)

Amazon Elastic Kubernetes Service (EKS)

Amazon ElastiCache (EC)

AWS Elastic Beanstalk

Amazon Elastic Inference

Amazon Elastic Transcoder

Amazon Elastic Map Reduce (EMR)

Amazon Elasticsearch Service (ES)

Amazon EventBridge

Amazon FSx

Amazon GameLift

AWS Glue

Amazon Inspector

AWS Internet of Things (IoT)

AWS IoT Things Graph

AWS IoT Analytics

Amazon Managed Streaming for Kafka

Amazon Kinesis Data Analytics

Amazon Data Firehose

Amazon Kinesis Data Streams

Amazon Kinesis Video Streams

AWS Lambda

Amazon Lex

Amazon CloudWatch Logs

AWS Elemental MediaTailor

AWS Elemental MediaConnect

AWS Elemental MediaConvert

AWS Elemental MediaPackage Live

AWS Elemental MediaPackage Video on Demand

Amazon MQ

Amazon VPC NAT Gateways

Amazon Neptune

AWS OpsWorks

Amazon Polly

Amazon QLDB

Amazon RDS

Amazon Redshift

Amazon Rekognition

AWS RoboMaker

Amazon Route 53

Amazon Route 53 Resolver

Amazon S3

Amazon SageMaker Batch Transform Jobs

Amazon SageMaker Endpoints

Amazon SageMaker Endpoint Instances

Amazon SageMaker Ground Truth

Amazon SageMaker Processing Jobs

Amazon SageMaker Training Jobs

AWS Service Catalog

Amazon Simple Email Service (SES)

Amazon Simple Notification Service (SNS)

Amazon Simple Queue Service (SQS)

AWS Systems Manager - Run Command

AWS Step Functions

AWS Storage Gateway

Amazon SWF

Amazon Textract

AWS Transfer Family

AWS Transit Gateway

Amazon Translate

AWS Trusted Advisor

AWS API Usage

AWS Site-to-Site VPN

AWS WAF Classic

AWS WAF

Amazon WorkMail

Amazon WorkSpaces

Available metrics

JobName is the main dimension.

Name

Description

Statistics

Unit

Dimensions

Recommended

glue.driver.aggregate.bytesRead

The number of bytes read from all data sources by all completed Spark tasks running in all executors

Sum

Bytes

JobName, JobRunId, Type

glue.driver.aggregate.elapsedTime

The ETL elapsed time in milliseconds (doesn't include the job bootstrap times)

Sum

Milliseconds

JobName, JobRunId, Type

glue.driver.aggregate.numCompletedStages

The number of completed stages in a job

Sum

Count

JobName, JobRunId, Type

glue.driver.aggregate.numCompletedTasks

The number of completed tasks in a job

Sum

Count

JobName, JobRunId, Type

glue.driver.aggregate.numFailedTasks

The number of failed tasks in a job

Sum

Count

JobName, JobRunId, Type

glue.driver.aggregate.numKilledTasks

The number of tasks killed in a job

Sum

Count

JobName, JobRunId, Type

glue.driver.aggregate.recordsRead

The number of records read from all data sources by all completed Spark tasks running in all executors

Sum

Count

JobName, JobRunId, Type

glue.driver.aggregate.shuffleBytesWritten

The number of bytes written by all executors to shuffle data between them since the previous report (aggregated by the AWS Glue metrics dashboard as the number of bytes written for this purpose during the previous minute)

Sum

Bytes

JobName, JobRunId, Type

glue.driver.aggregate.shuffleLocalBytesRead

The number of bytes read by all executors to shuffle data between them since the previous report (aggregated by the AWS Glue metrics dashboard as the number of bytes read for this purpose during the previous minute)

Sum

Bytes

JobName, JobRunId, Type

glue.driver.BlockManager.disk.diskSpaceUsed_MB

The number of megabytes of disk space used across all executors

Average

Megabytes

JobName, JobRunId, Type

glue.driver.ExecutorAllocationManager.executors.numberAllExecutors

The number of actively running job executors

Average

Count

JobName, JobRunId, Type

glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors

The number of maximum (actively running and pending) job executors needed to satisfy the current load

Maximum

Count

JobName, JobRunId, Type

glue.driver.jvm.heap.usage

The fraction of memory used by the JVM heap (scale: 0-1) for this driver

Average

Percent

JobName, JobRunId, Type

glue.ALL.jvm.heap.usage

The fraction of memory used by the JVM heap (scale: 0-1) for all executors

Average

Percent

JobName, JobRunId, Type

glue.driver.jvm.heap.used

The number of memory bytes used by the JVM heap for the driver

Average

Bytes

JobName, JobRunId, Type

glue.ALL.jvm.heap.used

The number of memory bytes used by the JVM heap for all executors

Average

Bytes

JobName, JobRunId, Type

glue.driver.s3.filesystem.read_bytes

The number of bytes read from Amazon S3 by the driver since the previous report (aggregated by the AWS Glue metrics dashboard as the number of bytes read during the previous minute)

Sum

Bytes

JobName, JobRunId, Type

glue.ALL.s3.filesystem.read_bytes

The number of bytes read from Amazon S3 by all executors since the previous report (aggregated by the AWS Glue metrics dashboard as the number of bytes read during the previous minute)

Sum

Bytes

JobName, JobRunId, Type

glue.driver.s3.filesystem.write_bytes

The number of bytes written to Amazon S3 by the driver since the previous report (aggregated by the AWS Glue metrics dashboard as the number of bytes written during the previous minute)

Sum

Bytes

JobName, JobRunId, Type

glue.ALL.s3.filesystem.write_bytes

The number of bytes written to Amazon S3 by all executors since the previous report (aggregated by the AWS Glue metrics dashboard as the number of bytes written during the previous minute)

Sum

Bytes

JobName, JobRunId, Type

glue.driver.system.cpuSystemLoad

The fraction of CPU system load used (scale: 0-1) by the driver

Average

Percent

JobName, JobRunId, Type

glue.ALL.system.cpuSystemLoad

The fraction of CPU system load used (scale: 0-1) by all executors

Average

Percent

JobName, JobRunId, Type