Ingest AWS logs from Amazon S3

Three things:

Everything else—the Lambda, SQS queues, IAM role, and event routing—is created for you by a single CloudFormation template.

No. The S3 logs forwarder can be deployed into your AWS account(s) without an AWS connection—you only need a valid ingest platform token and the required AWS permissions.

Be aware of the trade-off: without an AWS connection (with at least topology collection) for the source account, logs are still ingested, yet you node(entity) linking will not work, cloud metadata enrichment, and logs-in-context. If topology-aware analysis matters for a given account, onboard it as an AWS connection.

Both architectures are supported—pick the one that matches your estate and make sure the Layer ARN or ZIP package matches it. For the deployment package, the Lambda Layer option is the default and simplest (no build tools, no SAM CLI, no Python required); the ZIP option is available if you'd rather package the function code yourself.

One note: keep your CloudFormation stack name to 47 characters or fewer, or deployment will fail.

Yes. Any text-based log can be ingested as a generic log, and a stream of JSON entries can be ingested using the generic_json_stream source. You can then process/enrich those logs at ingestion time in Dynatrace or parse them at query time with DQL.

For more advanced needs, you can define custom log processing rules so the Lambda extracts entries the way you want.

The most common reasons:

Every log record is automatically annotated with log.source.aws.s3.bucket.name, log.source.aws.s3.key.name, and cloud.log_forwarder (the ARN of the forwarding Lambda, useful when you run more than one instance).

For supported AWS services, the forwarder also extracts service-specific attributes such as aws.account.id and aws.region. If you've onboarded the source account as an AWS connection, records are additionally linked to their Smartscape entity.

Yes. Log forwarding rules support custom annotations (for example team: payments or environment: dev), which are added as attributes to the matching log entries.

Use Logs in Dynatrace, or run a DQL query. For example:

You choose from three mutually exclusive options at deploy time: reference an existing AWS Secrets Manager secret (recommended), store the token as a SSM Parameter Store SecureString parameter, or pass it as plain text—in which case the stack creates a Secrets Manager secret for you.

The token is never hardcoded into the function.

Yes, once you grant it access. Add the KmsKeyArns parameter (a comma-separated list of key ARNs) to your deploy command so the Lambda can decrypt the objects it reads.

The CloudFormation template creates the required IAM role and permissions for you. If your organization enforces an IAM role path (for example: /engineering/platform/), set the IamRolePath parameter so the role is created where your governance policy expects it.

The forwarder retries a failed object up to three times. If it still fails, the SQS message is moved to a Dead Letter Queue (DLQ), where it's retained for up to one day.

You can inspect the DLQ and the Lambda's own logs to find the cause, and if it was a transient issue you can redrive the messages from the DLQ to reprocess them. You can tune this with MaximumSQSMessageRetries (default 3), SQSVisibilityTimeout (default 420s), and SQSLongPollingMaxSeconds (default 20s).

The template configures a CloudWatch alarm that fires whenever messages land in the DLQ. Add the NotificationsEmail parameter to receive email alerts, or subscribe your own tooling (for example: PagerDuty or ServiceNow) to the SNS topic the stack creates.

The Lambda publishes its own CloudWatch metrics under a namespace named after your stack, including LogFilesProcessed, LogProcessingFailures, DynatraceHTTP204Success, DynatraceHTTP429Throttled, LogMessagesTrimmed, and processing/ingestion timing metrics.

The Lambda also emits its own diagnostic logs, so first-line investigation happens entirely in your AWS account without needing to open a Dynatrace support ticket. If you run multiple deployments in the same account and region, each publishes its own metrics under a deployment dimension matching its stack name.

The forwarder is event-driven: new objects landing in S3 trigger the Lambda on arrival, so throughput scales automatically with log volume. It has been tested at 10 GB/min.

For high-throughput scenarios we recommend raising the MaximumLambdaConcurrency parameter.

Upgrades are in place: run a CloudFormation stack update with the new templates (and, for Layer deployments, the new Layer ARN)—no redeployment or migration needed.

Updates to the latest version are supported from v0.4.4 onward. To roll back, repeat the update using the previous version tag.

Both are possible. The S3BucketNames parameter forwards everything from the listed buckets. To forward only specific key prefixes (for example AWSLogs/123456789012/CloudTrail/), deploy the per-bucket configuration template instead—it supports up to 10 prefixes per bucket. Don't do both for the same bucket, or objects will be ingested twice.

Yes. The forwarder supports a centralized model—one Lambda in a central logging account and region pulling logs from buckets in many source accounts and regions—as well as a distributed model (a forwarder per account and/or region).

The centralized, cross-account/cross-region pattern is a common enterprise setup: source accounts forward S3 object-created notifications cross-account to the central event bus, and the central Lambda reaches back into each source bucket to fetch the object.

Note that this centralized pattern is a well-known customer pattern rather than a Dynatrace-recommended one—you remain responsible for how spoke accounts ship or replicate logs into the central buckets.

Each per-bucket configuration adds an inline statement to the Lambda's IAM role, and IAM caps a role's inline policy size at 10,240 characters. In practice that's roughly 20–25 buckets with the template as shipped. If you need more, you can optimize the policy (for example, using wildcards for buckets that share a common name prefix).

Three are supported:

Deploy the optional AppConfig template and set LogForwarderConfigurationLocation=aws-appconfig.

Your log-forwarding-rules and log-processing-rules then live in AWS AppConfig, and the Lambda picks up edits within about a minute—no Lambda redeployment required.

Yes, the two methods are complementary and fully supported side by side. A common split is Firehose for application logs already in CloudWatch, and S3 ingest for S3-native sources like CloudTrail and VPC Flow Logs.

Which one fits a given source depends on where the logs already land and your operational constraints; review the comparison in the release notes and choose per source.

Treat it as a migration, not an in-place upgrade. Deploy the new S3 log forwarder alongside your existing classic forwarder. Confirm logs arrive on the new cloud platform monitoring endpoints, then gradually stop shipping from the classic forwarder—for example by deactivating its access tokens or its Lambda function.