CloudWatch Logs

Latest Dynatrace
How-to guide

At present, Amazon Data Firehose log stream is the primary log ingest method (push-based).

Amazon Data Firehose allows to create a fully managed serverless log pipeline, where the destination is the Dynatrace log ingest APIs. This architecture:

Allows swift registration of AWS log producers with not much effort.
Auto-scales to meet log throughput demand and can efficiently handle back-pressure with configurable buffers and smart retry policy.

Onboard AWS logs

When using the recommended flow to create an AWS connection, the CloudFormation stack creates a Firehose stream per enabled monitored AWS region. You can modify the AWS Regions where logs can be pushed from by updating the AWS connection's CloudFormation main stack.

If you choose not to deploy AWS logs during the onboarding, update the AWS connection's CloudFormation main stack and set AWS logs to TRUE, then set the desired AWS Regions, and update the stack:

Go to the AWS CloudFormation console and locate the primary CloudFormation stack (the name follows the connection name: Connection Friendly name):
Initiate a stack update.
Find the Log Ingest parameters section and enable the logs ingest (set to TRUE), provide a combination of AWS Regions (separated by commas).
Start the update.

Modify or remove AWS CloudWatch Logs

You may wish to modify/remove AWS CloudWatch logs (Via Firehose) ingest or add/remove AWS Regions.

Go to the AWS CloudFormation console and locate the primary CloudFormation stack (the name follows the connection name, for example MyProd3East).
Initiate a stack update.
Use Existing Stack.
In Parameters > Log ingest, set the Enable log ingest parameter to TRUE Or FALSE, if set to TRUE modify the AWS Region(s), separated by commas.
Start the update.

AWS logs and AWS connection

While it's possible to forward logs into your Dynatrace environments without creating an AWS connection, we recommend onboarding logs (and any other signal) starting first with the creation of an AWS connection.

Forwarding logs from an AWS account without an existing AWS connection will not support log records entity linking or signal/entity enrichment.

Log ingest sources

CloudWatch Logs: This is a common and recommended log ingest type, where multiple AWS services are shipped with CloudWatch Logs support. In this type, the AWS service can send logs into a designated log group, which can then subscribe to the Dynatrace created Firehose data log stream allowing an end-to-end log ingest in minutes.

In this mode, Dynatrace SaaS attempts to map the log record producer (AWS resource) to the corresponding Dynatrace Smartscape node (or log record entity linking). Success in linking means the enrichment of the log records with the Dynatrace common attributes including AWS tags. Linking and enrichment unlocks multiple upstream platform use cases.
Direct push to Amazon Data Firehose stream: For certain AWS services, you can forward the logs directly into the Dynatrace-created Firehose stream.

In this mode, Dynatrace can't perform log record entity linking or log record enrichment.

AWS does not add the required rich resource metadata to its generated log records, so Dynatrace is unable to map the log record (AWS resource) to the Dynatrace entity (AWS resource), resulting in an inability to link and enrich those log records. We therefore highly recommend choosing the CloudWatch log groups as a source (when possible).

You can implement a workaround to allow the linking and enrichment of those log records, by using a Firehose stream per log.

Supported AWS service resources

This table lists supported AWS service resources for log records linking and enrichment—with CloudWatch as the logs source.

Service name	Service resource	Log group name requirement
AWS Lambda	`AWS::Lambda::Function`	default log group name¹
Amazon RDS	`AWS::RDS::DBInstance`	default log group name²
Amazon SNS	`AWS::SNS::Topic`	default log group name²
AWS App Runner	`AWS::AppRunner::Service`	default log group name²
Amazon EKS	`AWS::EKS::Cluster`	default log group name²
Amazon Route 53	`AWS::Route53::HostedZone`	default log group name⁶
Amazon API Gateway	`AWS::ApiGateway::RestApi` `AWS::ApiGatewayV2::Api`	default log group name³
AWS ElastiCache	`AWS::ElastiCache::CacheCluster`	default log group name⁴
Amazon OpenSearch Service	`AWS::OpenSearchService::Domain`	default log group name²
Amazon Managed Streaming for Apache Kafka	`AWS::MSK::Cluster` `AWS::KafkaConnect::Connector`	default log group name⁵

The log group name should not be modified. Renaming will prevent the log source from being depicted in context using Clouds Clouds.

The log group name can be modified.

The log group name must be named in this pattern: (ApiGateway) API-Gateway-Access-Logs_<rest-api-id>/<stage-name> (ApiGateway), ApiGatewayV2 log group names must start with: API-GatewayV2-Access-Logs.

⁴

The log group name must start with this pattern: /aws/route53/.

⁵

The MSK Broker log group names must start with: /aws/msk/broker , The MSK Connector log group names must start with: /aws/msk/broker.

⁶

The log group name must start with this pattern: /aws/elasticache/.