Trace Google Cloud Functions in Go with OpenTelemetry
This guide shows how to instrument Google Cloud Functions in Go with OpenTelemetry and export the traces to Dynatrace. To learn more about how Dynatrace works with OpenTelemetry, see OpenTelemetry and Dynatrace.
To learn about how to monitor Google Cloud Functions with Dynatrace-enhanced OpenTelemetry traces, see Integrate on Google Cloud Functions GoLang.
Prerequisites
The following prerequisites and limitations apply:
- Dynatrace version 1.222+
- W3C Trace Context is enabled
- Go to Settings > Preferences > OneAgent features.
- Turn on Send W3C Trace Context HTTP headers.
- Cloud Functions Go Runtime 1.16+
Instrument Google Cloud Functions
Dynatrace uses OpenTelemetry Trace Ingest to provide end-to-end visibility to your Google Cloud Functions.
To instrument your Google Cloud Functions
Add OpenTelemetry dependencies
Set up OpenTelemetry
Instrument the function entry point
Instrument outgoing requests
Add OpenTelemetry dependencies
Use the following commands to add the required OpenTelemetry dependencies to your function:
go get go.opentelemetry.io/otelgo get go.opentelemetry.io/otel/sdkgo get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
Set up OpenTelemetry
To make sure traces are collected, linked, and exported to Dynatrace, you need to set up and configure OpenTelemetry accordingly. For this, the Dynatrace endpoint and an authentication token are required.
To determine the endpoint
- Open Dynatrace.
- Check the address line of your browser. The URL will match one of the following patterns:
- Dynatrace SaaS:
https://{your-environment-id}.live.dynatrace.com/...
- Dynatrace Managed:
https://{your-domain}/e/{your-environment-id}/...
- Dynatrace SaaS:
- Replace the
...
part withapi/v2/otlp
to get the URL you will need to configure the OpenTelemetry exporter.- Dynatrace SaaS:
https://{your-environment-id}.live.dynatrace.com/api/v2/otlp
- Dynatrace Managed:
https://{your-domain}/e/{your-environment-id}/api/v2/otlp
- Dynatrace SaaS:
To create an authentication token
- Go to Access Tokens > Generate new token.
- Provide a Token name.
- In the Search scopes box, search for
Ingest OpenTelemetry traces
and select the checkbox. - Select Generate token.
- Select Copy to copy the token to your clipboard.
- Save the token in a safe place; you can't display it again, and you will need it to configure the OpenTelemetry exporter.
Here is how to set up the OpenTelemetry tracing pipeline:
package otelsetupimport ("context""log""go.opentelemetry.io/otel""go.opentelemetry.io/otel/exporters/otlp/otlptrace""go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp""go.opentelemetry.io/otel/propagation""go.opentelemetry.io/otel/sdk/resource"sdk "go.opentelemetry.io/otel/sdk/trace"semconv "go.opentelemetry.io/otel/semconv/v1.7.0")func InitTracing(serviceName string, serviceVersion string) *sdk.TracerProvider {client := otlptracehttp.NewClient()exporter, err := otlptrace.New(context.Background(), client)if err != nil {log.Fatal(err)}// create resourcer, err := resource.Merge(resource.Default(),resource.NewWithAttributes(// customizable resource attributessemconv.SchemaURL,semconv.ServiceNameKey.String(serviceName),semconv.ServiceVersionKey.String(serviceVersion),),)tracerProvider := sdk.NewTracerProvider(sdk.WithBatcher(exporter),sdk.WithResource(r),)otel.SetTracerProvider(tracerProvider)// setup W3C trace context as global propagatorotel.SetTextMapPropagator(propagation.TraceContext{})return tracerProvider}
To configure the exporter to your tenant, add the following environment variables when deploying your Google Cloud function:
OTEL_EXPORTER_OTLP_ENDPOINT
: set it to the previously determined endpoint.OTEL_EXPORTER_OTLP_HEADERS
: set it toAuthorization=Api-Token <TOKEN>
, where<TOKEN>
is the previously created authentication token.
Alternatively, the endpoint and authentication token can be configured in code by providing them as options to otlptracehttp.NewClient
.
Instrument the function entry point
To instrument invocations to a Google Cloud Function with OpenTelemetry, you need to
- Create a span around the entry point of the function to trace invocations.
- Extract and link the parent span from the propagated context. (To learn about W3C Trace Context, see our W3C Trace Context introduction.)
For certain libraries, OpenTelemetry Go already provides instrumentations that you can use to take care of these things.
The following sections show you how to instrument certain types of Google Cloud Functions:
Instrument an HTTP Google Cloud Function
The entry point of an HTTP Google Cloud Function mostly matches the standard http.Handler
interface. OpenTelemetry Go already provides an instrumentation for this interface. To add it as a dependency to your function, use the following command:
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
Because this instrumentation works with an http.Handler
interface, it requires your entry point function to have the name ServeHTTP
. Also, because the Go Runtime might terminate right after a function invocation, spans must be exported to Dynatrace beforehand.
To take care of this, create a wrapper function that instruments your actual handler and flushes the spans after invocation:
package instrumentorimport ("context""net/http""go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"semconv "go.opentelemetry.io/otel/semconv/v1.7.0""go.opentelemetry.io/otel/trace")type Flush interface {ForceFlush(context.Context) error}type HttpHandler = func(w http.ResponseWriter, r *http.Request)func InstrumentedHandler(functionName string, function HttpHandler, flusher Flush) HttpHandler {opts := []trace.SpanStartOption{// customizable span attributestrace.WithAttributes(semconv.FaaSTriggerHTTP),}// create instrumented handlerhandler := otelhttp.NewHandler(http.HandlerFunc(function), functionName, otelhttp.WithSpanOptions(opts...),)return func(w http.ResponseWriter, r *http.Request) {// call the actual handlerhandler.ServeHTTP(w, r)// flush spansflusher.ForceFlush(r.Context())}}
Putting everything together, here is how you use it in your function:
package myfunctionimport ("net/http""instrumentor""otelsetup")var InstrumentedHandler instrumentor.HttpHandlerfunc init() {tracerProvider := otelsetup.InitTracing("my-service", "1.0.0")InstrumentedHandler = instrumentor.InstrumentedHandler("my-function", Handler, tracerProvider)}func Handler(w http.ResponseWriter, r *http.Request) {// Your code goes here}
When deploying your function to GCP, make sure to use InstrumentedHandler
as the entry point to your Google Cloud Function.
Instrument a Pub/Sub Google Cloud Function
A Pub/Sub Google Cloud Function is triggered by the Pub/Sub message event. The event is unmarshalled by GCP into a message object that matches the type you defined in the entry point of your function. This type usually looks similar to the following:
type PubSubMessage struct {Data []byte `json:"data"`Attributes map[string]string `json:"attributes"`MessageId string `json:"messageId"`PublishTime string `json:"publishTime"`OrderingKey string `json:"orderingKey"`}
OpenTelemetry currently does not provide an instrumentation for Pub/Sub, so instrumenting a Pub/Sub Google Cloud Function requires a little more work.
In the following snippet, you can see how to create a wrapper function that instruments invocations to your Pub/Sub handler. This wrapper creates the corresponding span and uses the Attributes
map on the PubSubMessage
to extract and link to the parent span from the propagated context.
package instrumentorimport ("context""fmt""go.opentelemetry.io/otel""go.opentelemetry.io/otel/codes""go.opentelemetry.io/otel/propagation"semconv "go.opentelemetry.io/otel/semconv/v1.7.0""go.opentelemetry.io/otel/trace")const (instrumentationName = "my.company.com/my-pubsub-handler-instrumentation-name"instrumentationVer = "0.1.0")type PubSubHandler = func(context.Context, PubSubMessage) errortype Flush interface {ForceFlush(context.Context) error}func InstrumentedHandler(topicID string, handler PubSubHandler, flush Flush) PubSubHandler {return func(ctx context.Context, msg PubSubMessage) error {// create spanctx, span := beforePubSubHandlerInvoke(ctx, topicID, msg)defer span.End()// call actual handler functionerr := handler(ctx, msg)// update span with handler resultafterPubSubHandlerInvoke(span, err)// flush spansflush.ForceFlush(ctx)return err}}func beforePubSubHandlerInvoke(ctx context.Context, topicID string, msg PubSubMessage) (context.Context, trace.Span) {if msg.Attributes != nil {// extract propagated spanpropagator := otel.GetTextMapPropagator()ctx = propagator.Extract(ctx, propagation.MapCarrier(msg.Attributes))}opts := []trace.SpanStartOption{trace.WithSpanKind(trace.SpanKindConsumer),trace.WithAttributes(//customizable attributessemconv.FaaSTriggerPubsub,semconv.MessagingSystemKey.String("pubsub"),semconv.MessagingDestinationKey.String(topicID),semconv.MessagingDestinationKindTopic,semconv.MessagingOperationProcess,semconv.MessagingMessageIDKey.String(msg.MessageId),),}tracer := otel.GetTracerProvider().Tracer(instrumentationName, trace.WithInstrumentationVersion(instrumentationVer),)return tracer.Start(ctx, fmt.Sprintf("%s process", topicID), opts...)}func afterPubSubHandlerInvoke(span trace.Span, err error) {if err != nil {span.RecordError(err)span.SetStatus(codes.Error, err.Error())}}
Putting everything together, here is how to use the instrumented handler in your function:
package myfunctionimport ("context""instrumentor""otelsetup")var InstrumentedHandler instrumentor.PubSubHandlerfunc init() {tracerProvider := otelsetup.InitTracing("my-service", "1.0.0")InstrumentedHandler = instrumentor.InstrumentedHandler("my-topic", Handler, tracerProvider)}func Handler(ctx context.Context, msg PubSubMessage) error {// Your code goes herereturn nil}
When deploying your function to GCP, make sure to use InstrumentedHandler
as the entry point to your Google Cloud Function.
Instrument outgoing requests
To achieve end-to-end tracing, it is important to also make sure your outgoing requests are instrumented.
The following sections show how to instrument certain outgoing requests:
OpenTelemetry Go uses context.Context
to link a newly created span to its parent, so when using an instrumentation or creating a span manually, make sure to pass it the context.Context
instance that was passed to your Google Cloud Function (or an instance derived from it). Otherwise, your trace will not be linked properly.
Instrument outgoing HTTP requests
OpenTelemetry Go provides an instrumentation for tracing outgoing HTTP calls. Add it as a dependency to your function by using the following command:
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
Here is how you can use this instrumentation in your code:
import ("context""net/http""go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp")func makeHttpRequest(ctx context.Context, url string) {// create an instrumented HTTP clientclient := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}req, err := http.NewRequestWithContext(ctx, "GET", url, nil)if err != nil {// error handlingreturn}res, err := client.Do(req)if err != nil {// error handlingreturn}defer res.Body.Close()// response handling code goes here}
- Do not use convenience functions such as
GET
orPOST
on the standardhttp.Client
, because they do not accept acontext.Context
object. To make sure that your HTTP request is properly linked, either create a request with a context object as in the sample above, or use one of the convenience functions (such asotelhttp.Get
orotelhttp.Put
) of the HTTP instrumentation. - Make sure to close or fully read the response body. Otherwise, the outgoing request will not be instrumented properly.
Instrument Pub/Sub publish request
For the Pub/Sub client, there is currently no instrumentation in OpenTelemetry Go. Check out the following snippet to see how you can use the OpenTelemetry Go API to instrument Pub/Sub publish operations:
import ("context""fmt""cloud.google.com/go/pubsub""go.opentelemetry.io/otel""go.opentelemetry.io/otel/codes""go.opentelemetry.io/otel/propagation"semconv "go.opentelemetry.io/otel/semconv/v1.7.0""go.opentelemetry.io/otel/trace")const (instrumentationName = "my.company.com/my-pubsub-instrumentation-lib"instrumentationVer = "0.1.0")func PublishMessage(ctx context.Context, client *pubsub.Client, msg *pubsub.Message, topicID string) (string, error) {// create spanctx, span := beforePublishMessage(ctx, topicID, msg)defer span.End()// Send Pub/Sub messagemessageID, err := client.Topic(topicID).Publish(ctx, msg).Get(ctx)// enrich span with publish resultafterPublishMessage(span, messageID, err)return messageID, err}func beforePublishMessage(ctx context.Context, topicID string, msg *pubsub.Message) (context.Context, trace.Span) {opts := []trace.SpanStartOption{trace.WithSpanKind(trace.SpanKindProducer),trace.WithAttributes(// customizable span attributessemconv.MessagingSystemKey.String("pubsub"),semconv.MessagingDestinationKey.String(topicID),semconv.MessagingDestinationKindTopic,),}tracer := otel.Tracer(instrumentationName, trace.WithInstrumentationVersion(instrumentationVer),)ctx, span := tracer.Start(ctx, fmt.Sprintf("%s send", topicID), opts...)if msg.Attributes == nil {msg.Attributes = make(map[string]string)}// propagate Span across process boundariesotel.GetTextMapPropagator().Inject(ctx, propagation.MapCarrier(msg.Attributes))return ctx, span}func afterPublishMessage(span trace.Span, messageID string, err error) {if err != nil {span.RecordError(err)span.SetStatus(codes.Error, err.Error())} else {span.SetAttributes(semconv.MessagingMessageIDKey.String(messageID))}}
The above snippet propagates the outgoing span by injecting it into the Attributes
field on the Pub/Sub message. An instrumented Pub/Sub function will extract this propagated span to link the trace together.
Verify that the traces are ingested into Dynatrace
A few minutes after invoking your Google Cloud Functions, look for your spans:
- Go to Distributed Traces and select the Ingested traces tab.
- Your spans will be part of an existing PurePath distributed trace if the root of your call is already being monitored by the OneAgent.
If your Google Cloud Function is not getting any traffic, there will be no traces.
(Optional) Configure data capture to meet privacy requirements
While Dynatrace automatically captures all OpenTelemetry attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data stored.
To view your custom attributes, you need to allow them in the Dynatrace web UI first. To learn how to configure attribute storage and masking, see Attribute redaction.