Trace Google Cloud Functions in Go with OpenTelemetry

This guide shows how to instrument Google Cloud Functions in Go with OpenTelemetry and export the traces to Dynatrace. To learn more about how Dynatrace works with OpenTelemetry, see OpenTelemetry.

To learn about how to monitor Google Cloud Functions with Dynatrace-enhanced OpenTelemetry traces, see Integrate on Google Cloud Functions GoLang.

Prerequisites

The following prerequisites and limitations apply:

Dynatrace version 1.222+
W3C Trace Context is enabled
1. From the Dynatrace menu, go to Settings > Preferences > OneAgent features.
2. Turn on Send W3C Trace Context HTTP headers.
Cloud Functions Go Runtime 1.16+

Instrument Google Cloud Functions

Dynatrace uses OpenTelemetry Trace Ingest to provide end-to-end visibility to your Google Cloud Functions.

To instrument your Google Cloud Functions

Add OpenTelemetry dependencies

Set up OpenTelemetry

Instrument the function entry point

Instrument outgoing requests

Add OpenTelemetry dependencies

Use the following commands to add the required OpenTelemetry dependencies to your function:

1go get go.opentelemetry.io/otel
2go get go.opentelemetry.io/otel/sdk
3go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

Set up OpenTelemetry

To make sure traces are collected, linked, and exported to Dynatrace, you need to set up and configure OpenTelemetry accordingly. For this, the Dynatrace endpoint and an authentication token are required.

To determine the endpoint

Open Dynatrace.
Check the address line of your browser. The URL will match one of the following patterns:
- Dynatrace SaaS: https://{your-environment-id}.live.dynatrace.com/...
- Dynatrace Managed: https://{your-domain}/e/{your-environment-id}/...
Replace the ... part with api/v2/otlp to get the URL you will need to configure the OpenTelemetry exporter.
- Dynatrace SaaS: https://{your-environment-id}.live.dynatrace.com/api/v2/otlp
- Dynatrace Managed: https://{your-domain}/e/{your-environment-id}/api/v2/otlp

To create an authentication token

In the Dynatrace menu, go to Access tokens and select Generate new token.
Provide a Token name.
In the Search scopes box, search for Ingest OpenTelemetry traces and select the checkbox.
Select Generate token.
Select Copy to copy the token to your clipboard.
Save the token in a safe place; you can't display it again, and you will need it to configure the OpenTelemetry exporter.

Here is how to set up the OpenTelemetry tracing pipeline:

1package otelsetup
2
3import (
4    "context"
5    "log"
6
7    "go.opentelemetry.io/otel"
8    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
9    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
10    "go.opentelemetry.io/otel/propagation"
11    "go.opentelemetry.io/otel/sdk/resource"
12    sdk "go.opentelemetry.io/otel/sdk/trace"
13    semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
14)
15
16func InitTracing(serviceName string, serviceVersion string) *sdk.TracerProvider {
17    client := otlptracehttp.NewClient()
18    exporter, err := otlptrace.New(context.Background(), client)
19    if err != nil {
20        log.Fatal(err)
21    }
22
23    // create resource
24    r, err := resource.Merge(
25        resource.Default(),
26        resource.NewWithAttributes(
27            // customizable resource attributes
28            semconv.SchemaURL,
29            semconv.ServiceNameKey.String(serviceName),
30            semconv.ServiceVersionKey.String(serviceVersion),
31        ),
32    )
33
34    tracerProvider := sdk.NewTracerProvider(
35        sdk.WithBatcher(exporter),
36        sdk.WithResource(r),
37    )
38    otel.SetTracerProvider(tracerProvider)
39
40    // setup W3C trace context as global propagator
41    otel.SetTextMapPropagator(propagation.TraceContext{})
42
43    return tracerProvider
44}

To configure the exporter to your tenant, add the following environment variables when deploying your Google Cloud function:

OTEL_EXPORTER_OTLP_ENDPOINT: set it to the previously determined endpoint.
OTEL_EXPORTER_OTLP_HEADERS: set it to Authorization=Api-Token <TOKEN>, where <TOKEN> is the previously created authentication token.

Alternatively, the endpoint and authentication token can be configured in code by providing them as options to otlptracehttp.NewClient.

Instrument the function entry point

To instrument invocations to a Google Cloud Function with OpenTelemetry, you need to

Create a span around the entry point of the function to trace invocations.
Extract and link the parent span from the propagated context. (To learn about W3C Trace Context, see our W3C Trace Context introduction.)

For certain libraries, OpenTelemetry Go already provides instrumentations that you can use to take care of these things.

The following sections show you how to instrument certain types of Google Cloud Functions:

Instrument an HTTP Google Cloud Function

The entry point of an HTTP Google Cloud Function mostly matches the standard http.Handler interface. OpenTelemetry Go already provides an instrumentation for this interface. To add it as a dependency to your function, use the following command:

1go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Because this instrumentation works with an http.Handler interface, it requires your entry point function to have the name ServeHTTP. Also, because the Go Runtime might terminate right after a function invocation, spans must be exported to Dynatrace beforehand.

To take care of this, create a wrapper function that instruments your actual handler and flushes the spans after invocation:

1package instrumentor
2
3import (
4    "context"
5    "net/http"
6
7    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
8    semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
9    "go.opentelemetry.io/otel/trace"
10)
11
12type Flush interface {
13    ForceFlush(context.Context) error
14}
15
16type HttpHandler = func(w http.ResponseWriter, r *http.Request)
17
18func InstrumentedHandler(functionName string, function HttpHandler, flusher Flush) HttpHandler {
19    opts := []trace.SpanStartOption{
20        // customizable span attributes
21        trace.WithAttributes(semconv.FaaSTriggerHTTP),
22    }
23
24    // create instrumented handler
25    handler := otelhttp.NewHandler(
26        http.HandlerFunc(function), functionName, otelhttp.WithSpanOptions(opts...),
27    )
28
29    return func(w http.ResponseWriter, r *http.Request) {
30        // call the actual handler
31        handler.ServeHTTP(w, r)
32
33        // flush spans
34        flusher.ForceFlush(r.Context())
35    }
36}

Putting everything together, here is how you use it in your function:

1package myfunction
2
3import (
4    "net/http"
5
6    "instrumentor"
7    "otelsetup"
8)
9
10var InstrumentedHandler instrumentor.HttpHandler
11
12func init() {
13    tracerProvider := otelsetup.InitTracing("my-service", "1.0.0")
14    InstrumentedHandler = instrumentor.InstrumentedHandler("my-function", Handler, tracerProvider)
15}
16
17func Handler(w http.ResponseWriter, r *http.Request) {
18    // Your code goes here
19}

When deploying your function to GCP, make sure to use InstrumentedHandler as the entry point to your Google Cloud Function.

Instrument a Pub/Sub Google Cloud Function

A Pub/Sub Google Cloud Function is triggered by the Pub/Sub message event. The event is unmarshalled by GCP into a message object that matches the type you defined in the entry point of your function. This type usually looks similar to the following:

1type PubSubMessage struct {
2    Data        []byte            `json:"data"`
3    Attributes  map[string]string `json:"attributes"`
4    MessageId   string            `json:"messageId"`
5    PublishTime string            `json:"publishTime"`
6    OrderingKey string            `json:"orderingKey"`
7}

OpenTelemetry currently does not provide an instrumentation for Pub/Sub, so instrumenting a Pub/Sub Google Cloud Function requires a little more work.

In the following snippet, you can see how to create a wrapper function that instruments invocations to your Pub/Sub handler. This wrapper creates the corresponding span and uses the Attributes map on the PubSubMessage to extract and link to the parent span from the propagated context.

1package instrumentor
2
3import (
4    "context"
5    "fmt"
6
7    "go.opentelemetry.io/otel"
8    "go.opentelemetry.io/otel/codes"
9    "go.opentelemetry.io/otel/propagation"
10    semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
11    "go.opentelemetry.io/otel/trace"
12)
13
14const (
15    instrumentationName = "my.company.com/my-pubsub-handler-instrumentation-name"
16    instrumentationVer  = "0.1.0"
17)
18
19type PubSubHandler = func(context.Context, PubSubMessage) error
20
21type Flush interface {
22    ForceFlush(context.Context) error
23}
24
25func InstrumentedHandler(topicID string, handler PubSubHandler, flush Flush) PubSubHandler {
26    return func(ctx context.Context, msg PubSubMessage) error {
27        // create span
28        ctx, span := beforePubSubHandlerInvoke(ctx, topicID, msg)
29        defer span.End()
30
31        // call actual handler function
32        err := handler(ctx, msg)
33
34        // update span with handler result
35        afterPubSubHandlerInvoke(span, err)
36
37        // flush spans
38        flush.ForceFlush(ctx)
39
40        return err
41    }
42}
43
44func beforePubSubHandlerInvoke(ctx context.Context, topicID string, msg PubSubMessage) (context.Context, trace.Span) {
45    if msg.Attributes != nil {
46        // extract propagated span
47        propagator := otel.GetTextMapPropagator()
48        ctx = propagator.Extract(ctx, propagation.MapCarrier(msg.Attributes))
49    }
50
51    opts := []trace.SpanStartOption{
52        trace.WithSpanKind(trace.SpanKindConsumer),
53        trace.WithAttributes(
54            //customizable attributes
55            semconv.FaaSTriggerPubsub,
56            semconv.MessagingSystemKey.String("pubsub"),
57            semconv.MessagingDestinationKey.String(topicID),
58            semconv.MessagingDestinationKindTopic,
59            semconv.MessagingOperationProcess,
60            semconv.MessagingMessageIDKey.String(msg.MessageId),
61        ),
62    }
63
64    tracer := otel.GetTracerProvider().Tracer(
65        instrumentationName, trace.WithInstrumentationVersion(instrumentationVer),
66    )
67
68    return tracer.Start(ctx, fmt.Sprintf("%s process", topicID), opts...)
69}
70
71func afterPubSubHandlerInvoke(span trace.Span, err error) {
72    if err != nil {
73        span.RecordError(err)
74        span.SetStatus(codes.Error, err.Error())
75    }
76}

Putting everything together, here is how to use the instrumented handler in your function:

1package myfunction
2
3import (
4    "context"
5
6    "instrumentor"
7    "otelsetup"
8)
9
10var InstrumentedHandler instrumentor.PubSubHandler
11
12func init() {
13    tracerProvider := otelsetup.InitTracing("my-service", "1.0.0")
14    InstrumentedHandler = instrumentor.InstrumentedHandler("my-topic", Handler, tracerProvider)
15}
16
17func Handler(ctx context.Context, msg PubSubMessage) error {
18    // Your code goes here
19    return nil
20}

When deploying your function to GCP, make sure to use InstrumentedHandler as the entry point to your Google Cloud Function.

Instrument outgoing requests

To achieve end-to-end tracing, it is important to also make sure your outgoing requests are instrumented.

The following sections show how to instrument certain outgoing requests:

OpenTelemetry Go uses context.Context to link a newly created span to its parent, so when using an instrumentation or creating a span manually, make sure to pass it the context.Context instance that was passed to your Google Cloud Function (or an instance derived from it). Otherwise, your trace will not be linked properly.

Instrument outgoing HTTP requests

OpenTelemetry Go provides an instrumentation for tracing outgoing HTTP calls. Add it as a dependency to your function by using the following command:

1go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Here is how you can use this instrumentation in your code:

1import (
2    "context"
3    "net/http"
4
5    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
6)
7
8func makeHttpRequest(ctx context.Context, url string) {
9    // create an instrumented HTTP client
10    client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
11
12    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
13    if err != nil {
14        // error handling
15        return
16    }
17
18    res, err := client.Do(req)
19    if err != nil {
20        // error handling
21        return
22    }
23    defer res.Body.Close()
24
25    // response handling code goes here
26}

Do not use convenience functions such as GET or POST on the standard http.Client, because they do not accept a context.Context object. To make sure that your HTTP request is properly linked, either create a request with a context object as in the sample above, or use one of the convenience functions (such as otelhttp.Get or otelhttp.Put) of the HTTP instrumentation.
Make sure to close or fully read the response body. Otherwise, the outgoing request will not be instrumented properly.

Instrument Pub/Sub publish request

For the Pub/Sub client, there is currently no instrumentation in OpenTelemetry Go. Check out the following snippet to see how you can use the OpenTelemetry Go API to instrument Pub/Sub publish operations:

1import (
2    "context"
3    "fmt"
4
5    "cloud.google.com/go/pubsub"
6
7    "go.opentelemetry.io/otel"
8    "go.opentelemetry.io/otel/codes"
9    "go.opentelemetry.io/otel/propagation"
10    semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
11    "go.opentelemetry.io/otel/trace"
12)
13
14const (
15    instrumentationName = "my.company.com/my-pubsub-instrumentation-lib"
16    instrumentationVer  = "0.1.0"
17)
18
19
20func PublishMessage(ctx context.Context, client *pubsub.Client, msg *pubsub.Message, topicID string) (string, error) {
21    // create span
22    ctx, span := beforePublishMessage(ctx, topicID, msg)
23    defer span.End()
24
25    // Send Pub/Sub message
26    messageID, err := client.Topic(topicID).Publish(ctx, msg).Get(ctx)
27
28    // enrich span with publish result
29    afterPublishMessage(span, messageID, err)
30
31    return messageID, err
32}
33
34func beforePublishMessage(ctx context.Context, topicID string, msg *pubsub.Message) (context.Context, trace.Span) {
35    opts := []trace.SpanStartOption{
36        trace.WithSpanKind(trace.SpanKindProducer),
37        trace.WithAttributes(
38            // customizable span attributes
39            semconv.MessagingSystemKey.String("pubsub"),
40            semconv.MessagingDestinationKey.String(topicID),
41            semconv.MessagingDestinationKindTopic,
42        ),
43    }
44
45    tracer := otel.Tracer(
46        instrumentationName, trace.WithInstrumentationVersion(instrumentationVer),
47    )
48    ctx, span := tracer.Start(ctx, fmt.Sprintf("%s send", topicID), opts...)
49
50    if msg.Attributes == nil {
51        msg.Attributes = make(map[string]string)
52    }
53
54    // propagate Span across process boundaries
55    otel.GetTextMapPropagator().Inject(ctx, propagation.MapCarrier(msg.Attributes))
56
57    return ctx, span
58}
59
60func afterPublishMessage(span trace.Span, messageID string, err error) {
61    if err != nil {
62        span.RecordError(err)
63        span.SetStatus(codes.Error, err.Error())
64    } else {
65        span.SetAttributes(semconv.MessagingMessageIDKey.String(messageID))
66    }
67}

The above snippet propagates the outgoing span by injecting it into the Attributes field on the Pub/Sub message. An instrumented Pub/Sub function will extract this propagated span to link the trace together.

Verify that the traces are ingested into Dynatrace

A few minutes after invoking your Google Cloud Functions, look for your spans:

In the Dynatrace menu, go to Distributed traces and select the Ingested traces tab.
Your spans will be part of an existing PurePath distributed trace if the root of your call is already being monitored by the OneAgent.

If your Google Cloud Function is not getting any traffic, there will be no traces.

(Optional) Configure data capture to meet privacy requirements

While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data that's stored.

To view your custom span attributes, you need to allow them in the Dynatrace web UI first:

Span attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
Resource attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.