Trace AWS Lambda .NET Core functions with OpenTelemetry .NET

New version available

While you can use the vanilla OpenTelemetry capabilties to instrument your AWS Lambda functions for .NET as described within this tutorial, we recommend to follow our up-leveled version to
trace AWS Lambda .NET functions, which heavily reduces instrumentation effort and gives deeper and better insights into your function invocations.

In Feb 2021, AWS announced Support for AWS Distro for OpenTelemetry .NET Tracing. For general information about AWS Distro for OpenTelemetry, see the AWS Distro for OpenTelemetry page.

For tracing AWS Lambda for other languages such as Java, Node.JS, and Python using the Dynatrace AWS Lambda Layer, see Deploy OneAgent as Lambda extension.

Prerequisites

The following prerequisites and limitations apply:

  • Dynatrace version 1.222+
  • W3C Trace Context is enabled
    1. Go to Settings > Preferences > OneAgent features.
    2. Turn on Send W3C Trace Context HTTP headers.

The OpenTelemetry Protocol (OTLP) exporters for .NET currently support gRPC and HTTP 1.1 with binary Protocol Buffers (Protobuf) payload transports. Supported corresponding protocol values are grpc and http/protobuf. Configuration options can be set either via environment variables or explicitly in code.

Instrument AWS Lambda .NET Core functions

Dynatrace uses OpenTelemetry trace ingest to provide end-to-end visibility to your AWS Lambda .NET Core functions.

To instrument your AWS Lambda .NET Core functions

Step 1 Set up export

To ingest gRPC via the Dynatrace Trace API, you need to use an OpenTelemetry collector between Dynatrace and the exporter. You can choose to either self-host a collector or use the AWS Distro for OpenTelemetry Collector (ADOT Collector).

If you use environment variables for setup, you need to set the following value:

  • For OTEL_EXPORTER_OTLP_PROTOCOL: grpc

Add the ARN of the Lambda Layer ADOT Collector

AWS region

Lambda layers are a rationalized resource, meaning that they can only be used in the AWS region in which they are published. Make sure to use the layer in the same region as your Lambda functions.

Collector layer: aws-otel-collector-ver-0-27-0.

For a complete list of the AWS-managed OpenTelemetry Lambda layers, see AWS Distro for OpenTelemetry - AWS Lambda respository

Lambda layer ARN format is:

arn:aws:lambda:\<region>:901920570463:layer:<layer>:1

Configure ADOT Collector

The configuration of the ADOT Collector follows the OpenTelemetry standard.

By default, the ADOT Lambda layer uses the config.yaml file, which exports OpenTelemetry data to AWS X-Ray. To export the data to Dynatrace, you need to customize the configuration using the OpenTelemetry OTLP exporter.

To customize the collector configuration, add a configuration YAML file to your function code. After the file has been deployed with a Lambda function, create environment variable OPENTELEMETRY_COLLECTOR_CONFIG_FILE on your Lambda function and set it to /var/task/<path>/<to>/<filename>. This tells the extension where to find the collector configuration.

Here is a sample configuration file, collector.yaml, in the root directory:

  1. Copy collector.yaml in the root directory
  2. Set an environment variable OPENTELEMETRY_COLLECTOR_CONFIG_FILE to /var/task/<path>/<to>/<file>
receivers:
otlp:
protocols:
grpc:
exporters:
otlphttp:
endpoint: "https://<YOUR-TENANT-ID>.live.dynatrace.com/api/v2/otlp"
headers: {"Authorization": "Api-Token <YOUR-DYNATRACE-API-TOKEN>"}
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp]

For further details on configuration, see Best practices for OpenTelemetry traces.

You can set this via the Lambda console or the AWS CLI. With the CLI, use the following command:

aws lambda update-function-configuration --function-name Function --environment Variables={OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/collector.yaml}

You can also configure environment variables via CloudFormation template:

Function:
Type: AWS::Serverless::Function
Properties:
...
Environment:
Variables:
OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml

Step 2 Add dependencies

Add the following dependencies via NuGet to your project:

OpenTelemetry.Exporter.OpenTelemetryProtocol

If you are using the AWS SDK to interact with other AWS services, you can add auto-instrumentation using the ADOT SDK for .NET

OpenTelemetry.Contrib.Instrumentation.AWS

OpenTelemetry also provides other auto-instrumentation libraries available as NuGet packages

Step 3 Add OpenTelemetry Tracer

The AWS Distro for OpenTelemetry doesn't provide a wrapper layer for .NET as it does for other languages. You need to add a tracer to your code and create a trace root span.

The following sample uses an AWS Lambda Proxy Integration and gRPC transport.

If you don't set the Protocol property of the OtlpExporterOptions class via environment variables or in code, it will be initialized as OtlpExportProtocol.Grpc by default.

public class Functions
{
public Functions() {}
//Defines the OpenTelemetry resource attribute "service.name" which is mandatory
private const string servicename = "AWS Lambda";
//Defines the OpenTelemetry Instrumentation Scope.
private const string activitySource = "MyCompany.MyProduct.MyLibrary";
//Provides the API for starting/stopping activities.
private static readonly ActivitySource MyActivitySource = new ActivitySource(activitySource);
public async Task<APIGatewayProxyResponse> Get(APIGatewayProxyRequest request, ILambdaContext context)
{
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport",true);
//Initialize OpenTelemetry Tracer
using (Sdk.CreateTracerProviderBuilder()
.SetSampler(new AlwaysOnSampler())
.AddSource(activitySource)
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(servicename))
.AddAWSInstrumentation() //Add auto-instrumentation for AWS SDK
.AddHttpClientInstrumentation() //Add auto-instrumentation for AWS SDK
.AddOtlpExporter(otlpOptions =>
{
//Use a local endpoint for AWS Lambda ADOT Collector Layer
//or an endpoint configured via environment variable.
var collectorUrl = Environment.GetEnvironmentVariable("COLLECTOR_URL") ?? "http://localhost:55680";
otlpOptions.Endpoint = new Uri(collectorUrl);
})
.Build())
{
//create root-span, connecting with trace-parent read from the http-header
using (var activity = MyActivitySource.StartActivity("Invoke", ActivityKind.Server, request.Headers["traceparent"]))
{
//.....
//... YOUR CODE GOES HERE
//....
}
}
}
}