Subscribe AWS log groups to be ingested into Dynatrace

  • Latest Dynatrace
  • How-to guide
  • Published Sep 25, 2025
  • Preview

This page explains how to subscribe selected CloudWatch log groups to the designated Firehose stream ingesting logs into Dynatrace.

Prerequisites

AWS log ingestion (push-based) must be enabled and deployed within the designated AWS account as part of connection setup and management.

For details, see Onboard AWS logs.

Confirm that the Dynatrace log ingest stack was deployed correctly

To ensure that the Dynatrace log ingest CloudFormation stack was deployed correctly

  1. In the AWS Console, go to CloudFormation.
  2. Find a stack that name starts with: StackSet-DynatraceLogsIngest- followed by your Dynatrace Environment ID and alphanumerical characters. For example: 
    StackSet-DynatraceLogsIngest-abc12345-bd714890-987c-11f0-8b1f-0e8082238e73-c43acbc9-7840-4b73-affc-2052f5c73c0f
  3. In the Events tab, ensure all events have completed successfully and there are no failed events.
  4. Take a note of the stack name.
  5. In the Output tab, take a note of the outputs:
    • CloudWatchLogsRoleArn: ARN of the IAM role to use when creating a CloudWatch subscription filter.
    • FirehoseDeliveryStreamArn: ARN of the Firehose delivery stream pushing data to Dynatrace.

The log ingest StackSet instance deploys a single Firehose stream in each user enabled region.

Subscribe log groups to ingest AWS CloudWatch Logs

When a connection has been created with the Log ingest enabled option turned on, the next step is to subscribe CloudWatch log groups.

You might have your own GitOps processes to subscribe log groups already; if not, see the options below on how to do so.

To fetch the shell script, run the command below in a bash shell.

wget -q https://dynatrace-data-acquisition.s3.us-east-1.amazonaws.com/aws/deployment/cfn/da-log-groups-subscription.sh -O da-log-groups-subscription.sh && chmod +x ./da-log-groups-subscription.sh

You can use CloudShell, which is available in the AWS Console. Otherwise, if you have AWS CLI configured, you can use a bash-compliant shell. The scripts assume that REGION is set on the shell or AWS CLI configuration.

Subscribe by listing log group names

You can specify all the log group names in the script parameter, making sure to replace:

  • <cloudformation_stack_name> with name collected in prerequisites.

  • <your_log_group_list> with a space-separated list of the log group names you want to subscribe to.

    Example list: /aws/lambda/my-lambda /aws/apigateway/my-api

./da-log-groups-subscription.sh subscribe --stack-name <cloudformation_stack_name> --log-groups <your_log_group_list>

Subscribe by reading log groups from file

You can list all the group names in a text file and specify the file name as a script parameter.

  1. Create a file and list each log group name on a separate line.

  2. Save the file.

  3. Run the command below, making sure to replace:

    • <cloudformation_stack_name> with name collected in prerequisites
    • <your_file_name> with the actual file name.
    ./da-log-groups-subscription.sh subscribe --stack-name <cloudformation_stack_name> --log-groups-from-file <your_file_name>
Log groups auto-discovery

To simplify the file creation, you can use the auto-discovery command below to list the names of all log groups in your account. You can adjust the list manually before subscribing. Make sure to replace: <your_log_groups_file> with the file name to which you want to redirect the output.

./da-log-groups-subscription.sh discover-log-groups > <your_log_groups_file>

To list the names of log groups matching specific pattern, use the command below. Make sure to replace:

  • <your_log_groups_file> with the file name to which you want to redirect the output.
  • <log_group_name_pattern> with the string to match in log group name.
./da-log-groups-subscription.sh discover-log-groups --log-group-name-pattern <log_group_name_pattern> > <your_log_groups_file>

Subscribe with a subscription filter pattern

By default, you subscribe to all the logs in the log group. Use this option if you want to restrict the logs you subscribe to.

See Filter and pattern syntax for more details.

Run the command below, making sure to replace <your_log_group_list> and <your_filter_pattern> with your values.

./da-log-groups-subscription.sh subscribe --log-groups <your_log_group_list> --filter-pattern <your_filter_pattern>

Unsubscribe from log groups

To stop forwarding logs to Dynatrace, use one of the two options below to unsubscribe the CloudWatch log groups.

Unsubscribe by listing the log group names

Run the command below, making sure to replace <your_log_group_list> with a space-separated list of the log group names you want to unsubscribe from. Make sure to replace:

  • <cloudformation_stack_name> with name collected in prerequisites.
  • <your_log_group_list> with a space-separated list of the log group names you want to subscribe to. Example list: /aws/lambda/my-lambda /aws/apigateway/my-api
./da-log-groups-subscription.sh unsubscribe --stack-name <cloudformation_stack_name> --log-groups <your_log_group_list>

Unsubscribe by reading log groups from a file

Run the command below, making sure to replace:

./da-log-groups-subscription.sh unsubscribe --stack-name <cloudformation_stack_name> --log-groups-from-file <your_file_name>

View AWS logs in the Dynatrace environment

After configuring log subscription streaming, you should be able to view and analyze AWS logs in Logs Logs. Go to Logs & Events or Notebooks Notebooks, and filter for AWS logs.

Logs ingested via Amazon Data Firehose will be enriched with the stream name attribute key: dt.da.aws.data_firehose.arn; the attribute value will be resolved to the Amazon Resource Name (ARN) of the Firehose stream.

Logs from AWS services with entity linking support will automatically be displayed in Clouds Clouds for context analysis.

If you don't see any logs after 10 minutes, see Troubleshooting.

Troubleshooting

  1. Verify in the AWS Console that logs from CloudWatch are sent into Firehose.

    In the AWS Console, go to Amazon Data Firehose and select the Firehose instance. Check Firehose delivery stream metrics (incoming PUT requests, incoming bytes).

  2. If there's no data sent to Firehose, verify that subscribed CloudWatch log groups are producing new logs.

  3. If the subscription filter was created manually, verify that the correct IAM role and Firehose instance were selected when creating the subscription filter.

  4. Verify that logs are successfully sent from Firehose to Dynatrace. Check Firehose Delivery stream metrics (HTTP endpoint delivery success, records delivered to HTTP endpoint). In case of errors, check AWS Firehose CloudWatch Logs for details.

  5. If the Destination error logs tab (in the AWS Firehose console) displays this error message:

    Delivery to the endpoint was unsuccessful. See Troubleshooting HTTP Endpoints in the Firehose documentation for more information. Response received with status code. 403: "requestId":"xxxx,"errorMessage":"The authorization token does not provide the necessary permissions. details: missing\_scopes=\[data-acquisition:logs:ingest]

    Verify that:

    1. The Platform ingest token is assigned with the correct permission scope (data-acquisition:logs:ingest).

    2. The Dynatrace IAM user who created the platform token is also assigned with same token permission scope (data-acquisition:logs:ingest).

    3. The Platform ingest token has not expired.

      For more information on Dynatrace IAM, see Create an AWS connection.

Limitations

  • You can use only two subscription filters per log group. If you attempt to create a subscription filter that exceeds the limit, the AWS LimitExceededException error occurs.
  • Only new CloudWatch Logs (produced after creating log group subscription) are sent to Firehose and pushed to Dynatrace.
Related tags
Infrastructure ObservabilityLogsLogsNotebooksNotebooksCloudsClouds