Amazon Kinesis (Data Analytics, Data Firehose, Data Streams, Video Streams) monitoring

Dynatrace ingests metrics for multiple preselected namespaces, including Amazon Kinesis. You can view metrics for each service instance, split metrics into multiple dimensions, and create custom charts that you can pin to your dashboards.

Prerequisites

To enable monitoring for this service, you need

  • ActiveGate version 1.181+, as follows:

    • For Dynatrace SaaS deployments, you need an Environment ActiveGate or a Multi-environment ActiveGate.

    • For Dynatrace Managed deployments, you can use any kind of ActiveGate.

      For role-based access (whether in a SaaS or Managed deployment), you need an Environment ActiveGate installed on an Amazon EC2 host.

  • Dynatrace version 1.182+

  • An updated AWS monitoring policy to include the additional AWS services.
    To update the AWS IAM policy, use the JSON below, containing the monitoring policy (permissions) for all supporting services.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "acm-pca:ListCertificateAuthorities",
        "apigateway:GET",
        "apprunner:ListServices",
        "appstream:DescribeFleets",
        "appsync:ListGraphqlApis",
        "athena:ListWorkGroups",
        "autoscaling:DescribeAutoScalingGroups",
        "cloudformation:ListStackResources",
        "cloudfront:ListDistributions",
        "cloudhsm:DescribeClusters",
        "cloudsearch:DescribeDomains",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "codebuild:ListProjects",
        "datasync:ListTasks",
        "dax:DescribeClusters",
        "directconnect:DescribeConnections",
        "dms:DescribeReplicationInstances",
        "dynamodb:ListTables",
        "dynamodb:ListTagsOfResource",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeNatGateways",
        "ec2:DescribeSpotFleetRequests",
        "ec2:DescribeTransitGateways",
        "ec2:DescribeVolumes",
        "ec2:DescribeVpnConnections",
        "ecs:ListClusters",
        "eks:ListClusters",
        "elasticache:DescribeCacheClusters",
        "elasticbeanstalk:DescribeEnvironmentResources",
        "elasticbeanstalk:DescribeEnvironments",
        "elasticfilesystem:DescribeFileSystems",
        "elasticloadbalancing:DescribeInstanceHealth",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeRules",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticmapreduce:ListClusters",
        "elastictranscoder:ListPipelines",
        "es:ListDomainNames",
        "events:ListEventBuses",
        "firehose:ListDeliveryStreams",
        "fsx:DescribeFileSystems",
        "gamelift:ListFleets",
        "glue:GetJobs",
        "inspector:ListAssessmentTemplates",
        "kafka:ListClusters",
        "kinesis:ListStreams",
        "kinesisanalytics:ListApplications",
        "kinesisvideo:ListStreams",
        "lambda:ListFunctions",
        "lambda:ListTags",
        "lex:GetBots",
        "logs:DescribeLogGroups",
        "mediaconnect:ListFlows",
        "mediaconvert:DescribeEndpoints",
        "mediapackage-vod:ListPackagingConfigurations",
        "mediapackage:ListChannels",
        "mediatailor:ListPlaybackConfigurations",
        "opsworks:DescribeStacks",
        "qldb:ListLedgers",
        "rds:DescribeDBClusters",
        "rds:DescribeDBInstances",
        "rds:DescribeEvents",
        "rds:ListTagsForResource",
        "redshift:DescribeClusters",
        "robomaker:ListSimulationJobs",
        "route53:ListHostedZones",
        "route53resolver:ListResolverEndpoints",
        "s3:ListAllMyBuckets",
        "sagemaker:ListEndpoints",
        "sns:ListTopics",
        "sqs:ListQueues",
        "storagegateway:ListGateways",
        "sts:GetCallerIdentity",
        "swf:ListDomains",
        "tag:GetResources",
        "tag:GetTagKeys",
        "transfer:ListServers",
        "workmail:ListOrganizations",
        "workspaces:DescribeWorkspaces"
      ],
      "Resource": "*"
    }
  ]
}

If you don't want to add permissions to all services, and just select permissions for certain services, consult the table below. The table contains a set of permissions that are required for All AWS cloud services and, for each supporting service, a list of optional permissions specific to that service.

Permissions required for AWS monitoring integration:
  • "cloudwatch:GetMetricData"
  • "cloudwatch:GetMetricStatistics"
  • "cloudwatch:ListMetrics"
  • "sts:GetCallerIdentity"
  • "tag:GetResources"
  • "tag:GetTagKeys"
  • "ec2:DescribeAvailabilityZones"
Name
Permissions
All monitored Amazon services required
cloudwatch:GetMetricData,
cloudwatch:GetMetricStatistics,
cloudwatch:ListMetrics,
sts:GetCallerIdentity,
tag:GetResources,
tag:GetTagKeys,
ec2:DescribeAvailabilityZones
AWS Certificate Manager Private Certificate Authority
acm-pca:ListCertificateAuthorities
Amazon MQ
Amazon API Gateway
apigateway:GET
AWS App Runner
apprunner:ListServices
Amazon AppStream
appstream:DescribeFleets

Example of JSON policy for one single service.

{
  "Version": "2012-10-17",
  "Statement": [
          {
                  "Sid": "VisualEditor0",
                  "Effect": "Allow",
                  "Action": [
                          "apigateway:GET",
                          "cloudwatch:GetMetricData",
                          "cloudwatch:GetMetricStatistics",
                          "cloudwatch:ListMetrics",
                          "sts:GetCallerIdentity",
                          "tag:GetResources",
                          "tag:GetTagKeys",
                          "ec2:DescribeAvailabilityZones"
                  ],
                  "Resource": "*"
          }
      ]
}

In this example, from the complete list of permissions you need to select

  • "apigateway:GET" for Amazon API Gateway
  • "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "sts:GetCallerIdentity", "tag:GetResources", "tag:GetTagKeys", and "ec2:DescribeAvailabilityZones" for All AWS cloud services.

Enable monitoring

To learn how to enable service monitoring, see Enable service monitoring.

View service metrics

You can view the service metrics in your Dynatrace environment either on the custom device overview page or on your Dashboards page.

View metrics on the custom device overview page

To access the custom device overview page

  1. Go to Technologies & Processes or Technologies & Processes Classic (latest Dynatrace).
  2. Filter by service name and select the relevant custom device group.
  3. Once you select the custom device group, you're on the custom device group overview page.
  4. The custom device group overview page lists all instances (custom devices) belonging to the group. Select an instance to view the custom device overview page.

View metrics on your dashboard

You can also view metrics in the Dynatrace web UI on dashboards. There is no preset dashboard available for this service, but you can create your own dashboard.

To check the availability of preset dashboards for each AWS service, see the list below.

AWS service
Preset dashboard
Amazon EC2 Auto Scaling (built-in)
Not applicable
AWS Lambda (built-in)
Not applicable
Amazon Application and Network Load Balancer (built-in)
Not applicable
Amazon DynamoDB (built-in)
Not applicable
Amazon EBS (built-in)
Not applicable
Amazon EC2 (built-in)
Not applicable

Available metrics

Amazon Kinesis Data Analytics

Application is the main dimension.

Name
Description
Unit
Statistics
Dimensions
Recommended
Bytes
The number of bytes read (per input stream) or written (per output stream).
Bytes
Sum
Application, Flow, Id
Applicable
InputProcessing.DroppedRecords
The number of records returned by a Lambda function that were marked with Dropped status.
Count
Sum
Application, Flow, Id
InputProcessing.Duration
The time taken for each AWS Lambda function invocation performed by Kinesis Data Analytics.
Milliseconds
Multi
Application, Flow, Id
InputProcessing.OkBytes
The sum of bytes of the records returned by a Lambda function that were marked with OK status.
Bytes
Sum
Application, Flow, Id
InputProcessing.OkRecords
The number of records returned by a Lambda function that were marked with OK status.
Count
Sum
Application, Flow, Id
InputProcessing.ProcessingFailedRecords
The number of records returned by a Lambda function that were marked with ProcessingFailed status.
Count
Sum
Application, Flow, Id

Amazon Data Firehose

DeliveryStreamName is the main dimension.

Name
Description
Unit
Statistics
Dimensions
Recommended
BackupToS3.Bytes
The number of bytes delivered to Amazon S3 for backup over the specified time period. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled.
Bytes
Sum
Region
BackupToS3.Bytes
Bytes
Sum
DeliveryStreamName
BackupToS3.DataFreshness
Age (from getting into Amazon Data Firehose to now) of the oldest record in Amazon Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled.
Seconds
Maximum
Region
BackupToS3.DataFreshness
Seconds
Maximum
DeliveryStreamName
BackupToS3.Records
The number of records delivered to Amazon S3 for backup over the specified time period. Amazon Data Firehose emits this metric when backup to Amazon S3 is enabled.
Count
Sum
Region
BackupToS3.Records
Count
Sum
DeliveryStreamName

Amazon Kinesis Data Streams (KDS)

StreamName is the main dimension.

Name
Description
Unit
Statistics
Dimensions
Recommended
GetRecords.Bytes
The number of bytes retrieved from the Kinesis stream, measured over the specified time period. Minimum, maximum, and average statistics represent the bytes in a single GetRecords operation for the stream in the specified time period.
Bytes
Sum
StreamName
GetRecords.Bytes
Bytes
Multi
StreamName
GetRecords.Bytes
Bytes
Count
StreamName
GetRecords.IteratorAgeMilliseconds
The age of the last record in all GetRecords calls made against a Kinesis stream, measured over the specified time period. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream. The minimum and maximum statistics can be used to track the progress of Kinesis consumer applications. A value of 0 indicates that the records being read are completely caught up with the stream.
Milliseconds
Multi
StreamName
Applicable
GetRecords.IteratorAgeMilliseconds
Milliseconds
Count
StreamName
GetRecords.Latency
The time taken per GetRecords operation, measured over the specified time period
Milliseconds
Multi
StreamName

Amazon Kinesis Video Streams

StreamName is the main dimension.

Name
Description
Unit
Statistics
Dimensions
Recommended
GetHLSMasterPlaylist.Latency
Latency of the GetHLSMasterPlaylist API calls for the given stream name
Milliseconds
Multi
StreamName
GetHLSMasterPlaylist.Requests
Number of GetHLSMasterPlaylist API requests for a given stream
Count
Sum
StreamName
GetHLSMasterPlaylist.Success
The rate of success, 1 being the value for every successful request, and 0 the value for every failure
Count
Average
StreamName
GetHLSMediaPlaylist.Latency
Latency of the GetHLSMediaPlaylist API calls for the given stream name
Milliseconds
Multi
StreamName
GetHLSMediaPlaylist.Requests
Number of GetHLSMediaPlaylist API requests for a given stream
Count
Sum
StreamName
GetHLSMediaPlaylist.Success
The rate of success, 1 being the value for every successful request, and 0 the value for every failure
Count
Average
StreamName