OneAgent version 1.225+
OpenTelemetry interoperability connects the Dynatrace AWS Lambda extension to the OpenTelemetry Java API to use the instrumentation packages and extensions. You can then monitor technologies like databases or messaging frameworks that aren't supported by Dynatrace AWS Lambda extension out of the box.
Ensure that OpenTelemetry interoperability is enabled.
OpenTelemetry for Java provides several instrumentation packages in their OpenTelemetry Java instrumentation repository that can be used in combination with the Dynatrace AWS Lambda extension for additional visibility into certain technologies.
Dynatrace version 1.277+ The following example shows how the OpenTelemetry AWS SDK instrumentation packages can be used to monitor calls to a DynamoDB database.
Depending on the AWS SDK version you use, add one of the following packages to your package manager configuration (for example, Maven or Gradle).
AWS SDK version
Package to add
Notes
v1
Experimental span attributes must be enabled via system property (otel.instrumentation.aws-sdk.experimental-span-attributes=true
) or environment variable (OTEL_INSTRUMENTATION_AWS_SDK_EXPERIMENTAL_SPAN_ATTRIBUTES=true
).
Once the package has been added as a dependency, instrumentation is automatically added to capture requests to DynamoDB.
For the instrumentation to capture an item, query the item from DynamoDB using the AWS SDK, as in the code example below.
import java.util.Map;import com.amazonaws.services.lambda.runtime.Context;import com.amazonaws.services.lambda.runtime.RequestHandler;import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPEvent;import com.amazonaws.services.lambda.runtime.events.APIGatewayV2HTTPResponse;import software.amazon.awssdk.services.dynamodb.DynamoDbClient;import software.amazon.awssdk.services.dynamodb.model.AttributeValue;import software.amazon.awssdk.services.dynamodb.model.GetItemRequest;public class SimpleDynamoDBSample implements RequestHandler<APIGatewayV2HTTPEvent, APIGatewayV2HTTPResponse> {public APIGatewayV2HTTPResponse handleRequest(APIGatewayV2HTTPEvent input, Context context) {GetItemRequest request = GetItemRequest.builder().key(Map.of("mykey", AttributeValue.fromN("42"))).tableName("MyTable").build();try (DynamoDbClient dynamoDbClient = DynamoDbClient.builder().build()) {dynamoDbClient.getItem(request);} catch (Exception e) {return APIGatewayV2HTTPResponse.builder().withBody("error!").withStatusCode(500).build();}return APIGatewayV2HTTPResponse.builder().withBody("success!").withStatusCode(200).build();}}
After running the code example above, you'll see the DynamoDB database and individual requests in the related database service.
OpenTelemetry Java can be used in an SDK-like approach to trace additional operations that aren't covered by Dynatrace out of the box.
@Overridepublic String handleRequest(Object input, Context context) {Tracer tracer = GlobalOpenTelemetry.getTracer("instrumentation-library-name", "1.0.0");Span span = tracer.spanBuilder("do some work").startSpan();try {span.setAttribute("foo", "bar");// ....return "Hello from OpenTelemetry Java!";} finally {span.end();}}
The Dynatrace AWS Lambda extension captures only spans created via tracers from GlobalOpenTelemetry
and might not work if you try to manually (re)configure GlobalOpenTelemetry
.
OneAgent version 1.267+
You can use open-source instrumentation packages to trace AWS SQS and SNS messages and collect them via the Dynatrace AWS Lambda extension.
To enable automatic instrumentation of SQS messages, make sure you have installed the following.
AWS SDK version 2.2+ for sending the messages.
You can use the v1 and v2 SDK in the same Lambda function if you need version v1 in your function.
io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2-autoconfigure version 1.27+ added as a runtime-only dependency.
The full set of dependencies used in the examples, in Gradle format, is as follows.
dependencies {// Instrumentation for SQS dependenciesimplementation(platform('io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:1.27.0-alpha'))runtimeOnly('io.opentelemetry.instrumentation:opentelemetry-aws-sdk-2.2-autoconfigure')// Only needed if you want to trace messages from batches separatelyimplementation('io.opentelemetry:opentelemetry-api') // (Version taken from otel-instrumentation BOM)// Dependencies for the AWS SDK itself -- you should already have this in your Lambda if you send SQS messagesimplementation(platform('software.amazon.awssdk:bom:2.20.85'))implementation('software.amazon.awssdk:sqs') // Uses version from above BOM// Basic AWS Lambda dependencies -- you should already have this in your Lambdaimplementation('com.amazonaws:aws-lambda-java-events:3.6.0') // SQSEvent input, etcimplementation('com.amazonaws:aws-lambda-java-core:1.2.1') // RequestHandler interface, etc}
The example below shows a Lambda handler that sends a SQS message.
package com.dynatrace.example.lambda;import com.amazonaws.services.lambda.runtime.Context;import com.amazonaws.services.lambda.runtime.RequestHandler;import software.amazon.awssdk.services.sqs.SqsClient;import software.amazon.awssdk.services.sqs.model.SendMessageResponse;public class MessageSender implements RequestHandler<Object, Void> {private static final SqsClient client = SqsClient.create();@Overridepublic Void handleRequest(Object input, Context context) {final SendMessageResponse resp = client.sendMessage(builder -> builder.queueUrl("[your SQS-queue URL]").messageBody("[your payload]"));System.out.printf("Sent message with ID %s (send request ID %s)%n",resp.messageId(),resp.responseMetadata().requestId());return null;}}
Note that no extra code is required for tracing because the runtimeOnly
dependency added in the first
step automatically installs a handler that creates a span behind the scenes in the sendMessage
call using
the OpenTelemetry API, and OneAgent picks up that span automatically.
Additionally, you need to enable OneAgent's OpenTelemetry integration and set the environment variable
OTEL_INSTRUMENTATION_AWS_SDK_EXPERIMENTAL_USE_PROPAGATOR_FOR_MESSAGING
to true
in your Lambda function's configuration.
Receiving messages works out of the box when you use an AWS Lambda with an SQS trigger monitored with the Dynatrace AWS Lambda extension.
Because a span can have only a single parent, if your Lambda function receives a batch of multiple messages, you need to manually create spans to process each message if you want to track them separately and have them linked to the sender.
If that is enough for you, or you have a maximum batch size of one configured, or you send messages only rarely, no additional code or configuration is required.
If you want to gain more insights into a batch scenario with multiple messages, you first need to apply some configuration to allow you to override the parent span.
To configure the Dynatrace AWS Lambda extension to allow setting parent spans manually:
For the environment variables configuration method, set the DT_OPEN_TELEMETRY_ALLOW_EXPLICIT_PARENT
environment variable to true
:
DT_OPEN_TELEMETRY_ALLOW_EXPLICIT_PARENT=true
For the JSON file configuration method, in dtconfig.json
, set the following field to true
:
{...other configuration properties..."OpenTelemetry": {"AllowExplicitParent": "true"}}
Then new spans can be created with the parent span extracted from each received SQS message.
The following code processes messages one by one and links them to the sender as a parent span.
package com.dynatrace.example.lambda;import java.util.Map;import com.amazonaws.services.lambda.runtime.Context;import com.amazonaws.services.lambda.runtime.RequestHandler;import com.amazonaws.services.lambda.runtime.events.SQSEvent;import io.opentelemetry.api.GlobalOpenTelemetry;import io.opentelemetry.api.trace.Span;import io.opentelemetry.api.trace.Tracer;import io.opentelemetry.context.Scope;import io.opentelemetry.context.propagation.TextMapGetter;public class MessageIngress implements RequestHandler<SQSEvent, Void> {private static final Tracer tracer = GlobalOpenTelemetry.getTracer("message-ingress-app");@Overridepublic Void handleRequest(SQSEvent input, Context context) {for (SQSEvent.SQSMessage message: input.getRecords()) {Span span = tracer.spanBuilder(message.getEventSource() + " process").setSpanKind(SpanKind.CONSUMER) // MUST be either CONSUMER or SERVER.setParent(GlobalOpenTelemetry.getPropagators().getTextMapPropagator().extract(io.opentelemetry.context.Context.current(),message.getMessageAttributes(),SqsMessageRecordGetter.INSTANCE)).startSpan();try (Scope ignored = span.makeCurrent()) {handleMessage(message);} catch (Throwable e) {span.recordException(e);throw e;} finally {span.end();}}return null;}private void handleMessage(SQSEvent.SQSMessage message) {// This is where your actual handling code would goSystem.out.printf("Handling message with ID %s...%n", message.getMessageId());}private enum SqsMessageRecordGetter implements TextMapGetter<Map<String, SQSEvent.MessageAttribute>> {INSTANCE;@Overridepublic Iterable<String> keys(Map<String, SQSEvent.MessageAttribute> carrier) {return carrier.keySet();}@Overridepublic String get(Map<String, SQSEvent.MessageAttribute> carrier, String key) {if (carrier == null) {return null;}SQSEvent.MessageAttribute messageAttribute = carrier.get(key);return messageAttribute == null ? null : messageAttribute.getStringValue();}}}
This code requires a dependency on the OpenTelemetry Java API and requires enabling OneAgent's OpenTelemetry integration as explained above in Use OpenTelemetry Java API.
When you deploy the two Lambda functions, add OneAgent, create an SQS queue that the MessageSender
sends to, and add a trigger to MessageIngress
to receive from that queue, and then invoke the MessageSender
function (for example, using the default test event), you should get a distributed trace that looks something like this:
You may wonder why both the receiving function and the processing span are direct children of the Sqs.SendMessage
span instead of the processing span being a child of the receiving lambda function.
This structure is required to be prepared for batch receives. When you send multiple messages to the queue quickly, your receiver can process a batch of up to 10 messages at once. In this case, there are multiple sender spans from different traces, and the trace structure implemented in the sample code ensures that each process span is linked to the Sqs.SendMessage
span of the respective message. Because a span can have only a single parent, the receiving Lambda invoke span can be a child of only one (random) sender.