Trace .NET Lambda functions
Dynatrace uses OpenTelemetry to monitor AWS Lambda invocations.
Prerequisites
Ensure that you use a supported AWS Lambda runtime and that you have followed the initial configuration steps described in Configuration for monitoring AWS Lambda with OpenTelemetry before using the packages below.
The following NuGet packages can be used to cover different aspects of AWS Lambda tracing:
- recommended OpenTelemetry.Instrumentation.AWSLambda version 1.2.0-beta.1+—contains methods to trace incoming AWS Lambda invocations, such as calls triggered by AWS SQS/SNS messages.
- recommended OpenTelemetry.Instrumentation.AWS version 1.1.0-beta.1+—traces outgoing AWS SDK calls to other AWS Lambda invocations and calls to AWS services, such as DynamoDB, SQS, and SNS.
- required
Dynatrace.OpenTelemetry.Instrumentation.AwsLambda
—enables linking through outgoing AWS Lambda Invoke SDK calls from one Lambda to another. This is the only way to link these kinds of requests; however, if you don't need such linking or don't use the AWS Lambda client SDK to invoke or receive other Lambda invocations, you can replace the package withDynatrace.OpenTelemetry
.
Installation
Any of the above-listed packages can be installed via the CLI. For example, Dynatrace.OpenTelemetry.Instrumentation.AwsLambda
can be installed using the following command:
1dotnet add package Dynatrace.OpenTelemetry.Instrumentation.AwsLambda
Some packages may require you to specify a version explicitly or use the --prerelease
command line flag, such as
dotnet add package --prerelease OpenTelemetry.Instrumentation.AWSLambda
.
Compatibility with OpenTelemetry and System.Diagnostics.DiagnosticSource
versions
Periodically, we need to upgrade the minimum version of the OpenTelemetry
NuGet package
our components depend on, and consequently, the minimum version of the System.Diagnostics.DiagnosticSource
library.
This table lists the compatibility between Dynatrace.OpenTelemetry
, OpenTelemetry
, and System.Diagnostics.DiagnosticSource
versions.
Dynatrace.OpenTelemetry version | Minimum OpenTelemetry version | Minimum System.Diagnostics.DiagnosticSource version |
---|---|---|
1.273 and earlier | 1.1.0 | 5.0.1 |
1.275+ | 1.3.1 | 6.0.0 |
You don't usually need to worry about these dependencies as they're defined for you in our NuGet package. This
means that when you upgrade Dynatrace.OpenTelemetry
, NuGet might implicitly upgrade your OpenTelemetry
or
System.Diagnostics.DiagnosticSource
version if you are currently on an earlier one.
Initialization
The initialization code for AWS Lambda tracing in your Function.cs
file could look as follows (where Function
is the configured Lambda handler class):
1using System.Threading.Tasks;2using Amazon.Lambda.Core;3using Dynatrace.OpenTelemetry;4using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;5using OpenTelemetry;6using OpenTelemetry.Instrumentation.AWSLambda;7using OpenTelemetry.Trace;89[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]1011namespace Examples.AwsFunctionApp12{13 public class Function14 {15 private static readonly TracerProvider TracerProvider;1617 static Function()18 {19 DynatraceSetup.InitializeLogging();20 TracerProvider = Sdk.CreateTracerProviderBuilder()21 .AddDynatrace()22 // Configures AWS Lambda invocations tracing23 .AddAWSLambdaConfigurations(c => c.DisableAwsXRayContextExtraction = true)24 // Instrumentation for creation of span (Activity) representing AWS SDK call.25 // Can be omitted if there are no outgoing AWS SDK calls to other AWS Lambdas and/or calls to AWS services like DynamoDB and SQS.26 .AddAWSInstrumentation(c => c.SuppressDownstreamInstrumentation = true)27 // Adds injection of Dynatrace-specific context information in certain SDK calls (e.g. Lambda Invoke).28 // Can be omitted if there are no outgoing calls to other Lambdas via the AWS Lambda SDK.29 .AddDynatraceAwsSdkInjection()30 .Build();31 }32 }33}
- Setting the option
DisableAwsXRayContextExtraction
totrue
is required to skip Amazon X-Ray parent extraction, which may conflict with the Dynatrace propagation. - If the option
SuppressDownstreamInstrumentation
is set totrue
, HTTP child nodes will not be shown under AWS SDK calls.
Tracing incoming AWS Lambda calls
Example 1: Trace an AWS Lambda invoked via AWS SDK
In addition to the initialization part provided above, the handler method of a Lambda invoked via AWS SDK could look as follows:
1using System.Threading.Tasks;2using Amazon.Lambda.Core;3using Dynatrace.OpenTelemetry;4using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;5using OpenTelemetry;6using OpenTelemetry.Instrumentation.AWSLambda;7using OpenTelemetry.Trace;89[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]1011namespace Examples.AwsFunctionApp12{13 public class Function14 {15 private static readonly TracerProvider TracerProvider;1617 // Use initialization code from the "Initialization" section of the docs1819 public Task FunctionHandlerAsync(object input, ILambdaContext context)20 {21 var propagationContext = AwsLambdaHelpers.ExtractPropagationContext(context);22 return AWSLambdaWrapper.TraceAsync(TracerProvider, FunctionHandlerInternalAsync, input, context, propagationContext.ActivityContext);23 }2425 private Task FunctionHandlerInternalAsync(object input, ILambdaContext context)26 {27 // This is just an example of function handler and should be replaced by actual code.2829 return Task.CompletedTask;30 }31 }32}
- A parent context is extracted explicitly using the
AwsLambdaHelpers
class from theDynatrace.OpenTelemetry.Instrumentation.AwsLambda
package. - An activity tracing the incoming request and the handler is created by the
TraceAsync
method. TraceAsync
should be used when you trace an async function or a function returning a task. That way, the activity ends only when the task completes.
Example 2: Trace an AWS Lambda invoked via Amazon API Gateway (incoming HTTP request)
In addition to the initialization part provided above, the Lambda handler invoked via Amazon API Gateway could look as follows:
1using Amazon.Lambda.APIGatewayEvents;2using Amazon.Lambda.Core;3using Dynatrace.OpenTelemetry;4using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;5using OpenTelemetry;6using OpenTelemetry.Instrumentation.AWSLambda;7using OpenTelemetry.Trace;89[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]1011namespace Examples.AwsFunctionApp12{13 public class Function14 {15 private static readonly TracerProvider TracerProvider;1617 // Use initialization code from the "Initialization" section of the docs1819 public APIGatewayHttpApiV2ProxyResponse FunctionHandler(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)20 {21 return AWSLambdaWrapper.Trace(TracerProvider, FunctionHandlerInternal, request, context);22 }2324 private APIGatewayHttpApiV2ProxyResponse FunctionHandlerInternal(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)25 {26 // This is just an example of function handler and should be replaced by actual code.2728 return new APIGatewayHttpApiV2ProxyResponse29 {30 StatusCode = 200,31 Body = "Example function result",32 };33 }34 }35}
- A parent context is extracted from the incoming request in the
Trace
(orTraceAsync
) method. - An activity tracing the incoming request and the handler is created by the
Trace
method. - In general, the
Trace
/TraceAsync
methods support any trigger, but extended support is available for theAPIGatewayProxyRequest
andAPIGatewayHttpApiV2ProxyRequest
trigger types. For more details about request/response types, consult the GitHub documentation. Trace
should only be used when you have a function returning something other than aTask
. For the asynchronous handler,TraceAsync
should be used instead.
Example 3: Tracing without the AwsLambda
package
If you prefer not to use the OpenTelemetry.Instrumentation.AWSLambda
package, you can manually create an activity for Lambda. Note that this involves quite a bit of work, as Dynatrace requires certain activity tags (span attributes) to detect the service (conforming to the OpenTelemetry FaaS trace conventions
and resource conventions). You also need to manually extract the parent context.
For this example, only the Dynatrace.OpenTelemetry.Instrumentation.AwsLambda
package is required.
1using System;2using System.Collections.Generic;3using System.Diagnostics;4using System.Reflection;5using Amazon.Lambda.APIGatewayEvents;6using Amazon.Lambda.Core;7using Dynatrace.OpenTelemetry;8using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;9using OpenTelemetry;10using OpenTelemetry.Context.Propagation;11using OpenTelemetry.Trace;1213[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]1415namespace Examples.AwsFunctionApp16{17 public class Function18 {19 private static readonly TracerProvider TracerProvider;20 private static readonly ActivitySource ActivitySource;2122 static Function()23 {24 DynatraceSetup.InitializeLogging();25 var activitySourceName = Assembly.GetExecutingAssembly().GetName().Name;26 ActivitySource = new ActivitySource(activitySourceName);27 TracerProvider = Sdk.CreateTracerProviderBuilder()28 .AddSource(activitySourceName)29 .AddDynatrace()30 .Build();31 }3233 public static IEnumerable<KeyValuePair<string, object>> GetFunctionTags(ILambdaContext context, string trigger)34 {35 return new KeyValuePair<string, object>[]36 {37 new("faas.name", context.FunctionName),38 new("faas.id", context.InvokedFunctionArn),39 new("faas.trigger", trigger),40 new("cloud.platform", "aws_lambda"),41 new("cloud.provider", "aws"),42 new("cloud.region", Environment.GetEnvironmentVariable("AWS_REGION")),43 };44 }4546 public APIGatewayProxyResponse FunctionHandler(APIGatewayHttpApiV2ProxyRequest apiGatewayProxyEvent, ILambdaContext context)47 {48 try49 {50 var parentContext = ExtractParentContext(apiGatewayProxyEvent, context);51 using var activity = ActivitySource.StartActivity(ActivityKind.Server, parentContext, GetFunctionTags(context, "http"));52 return new APIGatewayProxyResponse53 {54 StatusCode = 200,55 Body = "Example function result",56 };57 }58 catch (Exception ex)59 {60 context.Logger.LogLine($"Exception occurred while handling request: {ex.Message}");61 throw;62 }63 finally64 {65 TracerProvider?.ForceFlush();66 }67 }6869 private static ActivityContext ExtractParentContext(APIGatewayHttpApiV2ProxyRequest apiGatewayProxyEvent, ILambdaContext context)70 {71 var propagationContext = AwsLambdaHelpers.ExtractPropagationContext(context);72 if (propagationContext == default)73 {74 propagationContext = Propagators.DefaultTextMapPropagator.Extract(default, apiGatewayProxyEvent, HeaderValuesGetter);75 }7677 return propagationContext.ActivityContext;78 }7980 private static IEnumerable<string> HeaderValuesGetter(APIGatewayHttpApiV2ProxyRequest apiGatewayProxyEvent, string name) =>81 (apiGatewayProxyEvent.Headers != null && apiGatewayProxyEvent.Headers.TryGetValue(name.ToLowerInvariant(), out var value)) ? new[] { value } : null;82 }83}
Tracing AWS SDK calls
You can use open-source instrumentation NuGet packages to trace Amazon DynamoDB calls using client like AmazonDynamoDBClient
or SQS and SNS calls using clients like AmazonSQSClient
or AmazonSimpleNotificationServiceClient
.
To set up tracing of AWS SDK calls,
-
Make sure that the following packages are added to your project:
OpenTelemetry.Instrumentation.AWSLambda
OpenTelemetry.Instrumentation.AWS
To learn more about the packages, see Prerequisites
-
After you add packages to your project, add the following initialization code:
1using Dynatrace.OpenTelemetry;2using OpenTelemetry;3using OpenTelemetry.Trace;45namespace Examples.AwsFunctionApp6{7 public class Function8 {9 private static readonly TracerProvider TracerProvider;1011 static Function()12 {13 DynatraceSetup.InitializeLogging();14 TracerProvider = Sdk.CreateTracerProviderBuilder()15 .AddDynatrace()16 .AddAWSLambdaConfigurations(c =>17 {18 c.DisableAwsXRayContextExtraction = true;19 c.SetParentFromBatch = true;20 })21 // Instrumentation used for tracing outgoing calls to AWS services via AWS SDK (including Amazon DynamoDB, SQS/SNS).22 // Can be omitted if no outgoing AWS SDK calls expected.23 .AddAWSInstrumentation(c => c.SuppressDownstreamInstrumentation = true)24 .Build();25 }26 }27}SuppressDownstreamInstrumentation
When set totrue
, HTTP child nodes will not be shown under AWS SDK calls.- SQS
SetParentFromBatch
It controls whether the parent Activity should be set when a potentially batched event is received where multiple parents would be available (e.g. SQS). When set totrue
the parent is set using one of the received messages. Else, the parent is not set. In both cases, links will be created. DisableAwsXRayContextExtraction
We reccomend to set it totrue
to skip Amazon X-Ray parent extraction and avoid conflicts with the Dynatrace propagation.
Tracing of Amazon DynamoDB operations
To trace Amazon DynamoDB calls, set up AWS SDK instrumentation. No additional code is required.
In the following example, the DynamoDB operations DescribeTable
, UpdateItem
, and GetItem
are represented as separate span child nodes with a common parent outbound-aws-dotnet in eu-central-1
(Lambda function performing DynamoDB operations):
Tracing of outgoing SQS/SNS messages
To trace outgoing SQS/SNS messages, set up AWS SDK instrumentation. No additional code is required.
Tracing of incoming SQS/SNS messages
To trace incoming SQS/SNS messages,
- Set up AWS SDK instrumentation.
- Wrap the function handler into one of the tracing methods of the
AWSLambdaWrapper
class:
1using Amazon.Lambda.Core;2using Amazon.Lambda.SQSEvents;3using Dynatrace.OpenTelemetry;4using OpenTelemetry;5using OpenTelemetry.Instrumentation.AWSLambda;6using OpenTelemetry.Trace;7using System.Threading.Tasks;89namespace Examples.AwsFunctionApp10{11 public class Function12 {13 private static readonly TracerProvider TracerProvider;1415 static Function()16 {17 // See "Set up tracing for AWS SDK calls" section above.18 }1920 public Task Handler(SQSEvent sqsEvent, ILambdaContext context) =>21 AWSLambdaWrapper.TraceAsync(tracerProvider, HandlerInternal, sqsEvent, context);2223 private Task HandlerInternal(SQSEvent sqsEvent, ILambdaContext context)24 {25 // This is just an example of async function handler and it should be replaced by actual code.2627 return Task.CompletedTask;28 }29 }30}
Special considerations for HttpClient instrumentation
Because with AWS Lambda the function execution environment might be suspended at any time after the function handler execution, the span export might be interrupted or delayed (for example, in some cases, the span might not be exported at all) and outgoing HTTP requests will typically appear only with the next function invocation.
However, outgoing HTTP request activities related to AWS Runtime API might be captured by OpenTelemetry HttpClient instrumentation in the current function invocation, even if unexpected. To avoid unexpected outgoing HTTP requests in the current function invocation, we recommend configuring the following filter when initializing the HttpClient instrumentation.
1var tracerProvider = Sdk.CreateTracerProviderBuilder()2 // Initialization code similar to previous examples...3 .AddHttpClientInstrumentation(op =>4 {5 op.FilterHttpRequestMessage = req => Activity.Current?.Parent?.IsAllDataRequested ?? false;6 })7 .Build();