Known limitations for Go support
Before you start using Go application monitoring, make sure that you are aware of the known limitations.
Support limited to official, stable Go releases
Go support is limited to official, stable Go releases compiled with the Golang toolchain.
OneAgent doesn't support binaries compiled using the gccgo toolchain.
Application binaries must be dynamically linked
This restriction applies only to Linux systems and if Go static monitoring is disabled.
OneAgent fully automatic injection requires dynamically linked application binaries. Dynamic linking is automatically applied when the application uses certain standard runtime library packages, for example, net/http
.
In all other cases, you can enforce dynamic linking through the -ldflags="-linkmode=external"
command line option. Note that disabling cgo, for example, using CGO_ENABLED=0
, is not supported, and OneAgent will reject the resulting application binary.
Consider the following minimalistic Go application called GoMinimal.go
:
1package main2import "fmt"34func main() {5 fmt.Print("Enter text: ")6 var input string7 fmt.Scanln(&input)8 fmt.Print(input)9}
Building the application results in a statically linked application binary:
1go build GoMinimal.go2file GoMinimal3GoMinimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
You can enforce dynamic linking with -ldflags="-linkmode=external"
:
1go build -ldflags="-linkmode=external" GoMinimal.go2file GoMinimal3GoMinimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32
Applications can't be built with -linkshared
option
Go supports dynamic linking of the Go standard library. This build mode is rarely used, and OneAgent won't inject into applications built this way.
Consider the following minimalistic Go application called GoMinimal.go
:
1go install -buildmode=shared -linkshared std2go build -linkshared GoMinimal.go
OneAgent will reject the resulting application binary.
Applications that load Go plugins aren't supported
A Go plugin is a package compiled using the -buildmode=plugin
build flag to produce a shared object file. This build mode is rarely used, and OneAgent will disable deep monitoring when an application actually loads a Go plugin.
Vendored third-party packages aren't supported
Go vendoring is used to include local copies of external dependencies in the project repository. This approach was used to pin versions of third-party packages before Go module support was added.
OneAgent will not monitor vendored packages. For example, gRPC services are supported only if you use Go modules or if you import go-grpc directly without using a dependency management system.
Possible limitations without a symbol table
By default, Go generates application binaries that contain a symbol table.
Currently there are no known limitations for stripped binaries in Dynatrace. However, we can't guarantee that all current features will continue to work in future Go versions or that all features added later will be supported in stripped binaries.
We therefore recommend that you build Go binaries that contain a symbol table and avoid the use of command line parameters or external tools that might suppress it.
- Don't use the external tool
strip
(strip <Go binary>
). - Don't compile with
go build -ldflags="-s"
. The-s
flag strips away the symbol table. - Don't run
go run <application>
. This rarely used command builds and runs applications on the fly. Because the output application file is temporary (the file is deleted automatically after the app termination), the application binary contains no symbol table.
In Dynatrace, monitoring of stripped binaries is enabled by default. However, you can disable the feature for the tenant by turning off Go stripped binaries on the OneAgent features settings page.
Applications built with race detector enabled aren't supported
An application built with -race
flag contains a built-in data race detector.
This build mode is mostly used in a development environment and OneAgent won't inject into applications built this way.
Creation stack profiling of OS threads is disabled
OneAgent does not support the predefined threadcreate
profile. Thread creation profiling results of Go applications monitored by OneAgent will contain empty stack traces only.
Support for static monitoring
OneAgent version 1.203+
Automatic injection for statically linked Go binaries is supported by OneAgent if
-
The parent process is dynamically linked. This also applies to applications running as container payload.
In this example, the parent process is a
/bin/sh
shell that starts a statically linked Go binary. The following code launches the/bin/sh
shell and executes the provided command.1/bin/sh -c '/StaticGoMinimal <optional app arguments>'You can use the
file
command to verify if an application is dynamically or statically linked.1$ file -L /bin/sh2/bin/sh: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked3$ file -L /StaticGoMinimal4/StaticGoMinimal: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked -
Kubernetes Classic full-stack injection The statically linked Go binary is running as Docker container entrypoint.
1FROM alpine:3.112COPY StaticGoMinimal /3ENTRYPOINT ["/StaticGoMinimal"]
If your setup is not supported by automatic injection, we recommend calling the static Go application via a shell (/bin/sh -c '/StaticGoMinimal <optional app arguments>'
).
To enable Go static application monitoring, see Enable Go static monitoring.
Limitations
-
Static Go applications running as container entrypoints are not supported in cloud-native full-stack injection.
Automatic injection of statically linked Go applications running as container entrypoints is not supported when using the cloud-native full-stack injection deployment option in Kubernetes/OpenShift.
To overcome this limitation, you can change the container entrypoint from a statically linked Go application to a dynamically linked application such as a shell.
For example, in a cloud-native full-stack injection deployment, the following code starts an unsupported statically linked Go application.
1FROM alpine:3.112COPY StaticGoMinimal /3ENTRYPOINT ["/StaticGoMinimal"]By changing the container entryproint from a statically linked Go application to a dynamically linked application, we obtain the following code that launches
/bin/sh
and executes the/StaticGoMinimal
command.1FROM alpine:3.112COPY StaticGoMinimal /3ENTRYPOINT ["/bin/sh", "-c", "'/StaticGoMinimal'"] -
Static Go applications that use cgo are not supported.
OneAgent rejects monitoring of static Go binaries that use cgo and, therefore, have a static dependency on the C system library libc. This is because the statically linked version of libc might conflict with the one used by OneAgent.
To overcome this limitation, you can build the Go application as a dynamically linked executable that dynamically links to libc. This will ensure that both the Go application and OneAgent use the same version of libc, which is the one available on the host.
-
Static Go monitoring requires the
SYS_PTRACE
capability in Docker containers.The
SYS_PTRACE
capability is enabled by default for Docker 19.03.0+ and Linux Kernel 4.8+. It allows system calls between processes running in a container, which is a requirement for Go static monitoring.You can overcome this limitation for Docker versions earlier than 19.03.0 or Linux Kernel versions earlier than 4.8 by running the container with the
SYS_PTRACE
capability as shown below.1docker run --cap-add=SYS_PTRACE <container> ... -
Docker images that don't provide a C system library are not supported.
OneAgent requires a C system library to be available on the monitored host.
To overcome this limitation, you can change the base image of a container to one that provides a C system library.
An example of a Docker image that doesn't provide a C system library is the scratch image.
1FROM scratch2COPY StaticGoMinimal /3CMD ["/StaticGoMinimal"]An example of an image that provides a C system library is the alpine image.
1FROM alpine:3.112COPY StaticGoMinimal /3CMD ["/StaticGoMinimal"]
Side effects
The file proc/<pId>/exe
refers to an executable named oneagentdynamizer
instead of the Go application binary, it is contained in the proc pseudo-filesystem that provides an interface to kernel data structures of running processes. This may cause system tools like ps
or top
to display oneagentdynamizer
instead of the Go binary name in their output.
Support for musl libc
The musl libc library is a drop-in replacement for the glibc library. Dynatrace supports musl-based Go applications, such as those built on Alpine Linux.
There is one additional requirement for building a dynamically linked application binary. You should use the Go toolchain for alpine (golang:<version>-alpine) and add -ldflags="-linkmode external"
(or add -linkmode external
to an existing -ldflags
) to the build command line to enforce usage of the system linker. This is not required for statically linked Go applications watched by Go static monitoring.
While musl libc does closely mimic the glibc functionalities, there are subtle behavioral differences between the two. Moreover, Go doesn't officially support the musl-based Go toolchain, which means Go toolchain binaries can't be downloaded from the golang.org website.
In addition, there is a serious issue with how Go uses musl libc. This limits the extent to which Dynatrace can support musl-based applications. The Go toolchain includes an internal linker that generates musl-based application binaries that don't correctly initialize musl libc at application startup. This issue prevents Dynatrace from monitoring these applications. In such a case, the following message is displayed on the relevant application process page:
Activation of deep monitoring was unsuccessful, Monitoring of Go musl binaries built with Go internal linker is not supported
If you use the system linker to generate the application binary, it adds startup code that correctly initializes shared objects. Also, adding -ldflags="linkmode external"
to the build command line enforces usage of the system linker. The resulting binary will execute with a correctly initialized libc, allowing Dynatrace to monitor such an application.
Consider the following minimalistic Go application called GoMinimal.go
:
1package main2import "fmt"34func main() {5 fmt.Print("Enter text: ")6 var input string7 fmt.Scanln(&input)8 fmt.Print(input)9}
The following multi-stage docker file yields a valid dynamically linked Go musl binary in stage 1 and runs the application in stage 2.
1# --- Stage 1:2# Use Golang toolchain for alpine to build the application.3FROM golang:1.13.5-alpine as builder4RUN apk update && apk add gcc libc-dev56# Copy local code, for example, GoMinimal.go, to the container image.7COPY ./GoMinimal.go ./GoMinimal.go89# Build dynamically linked Go binary.10RUN go build -ldflags="-linkmode=external" GoMinimal.go11# or add '-linkmode=external' to existing ldflags:12# e.g.: go build -ldflags="-linkmode=external <other linker flags>" GoMinimal.go1314# --- Stage 2:15# Use a Docker multi-stage build to create a lean production image.16FROM alpine:3.1117# Install ca-certificates and libc6-compat for Go programs to work properly.18RUN apk add --no-cache ca-certificates libc6-compat1920# Copy the binary to the production image from the builder stage.21COPY --from=builder /go/GoMinimal /GoMinimal2223# Run the application on container startup.24CMD ["/GoMinimal"]
Build the container and run the application:
1docker build -t gominimal-alpine .2docker run --interactive gominimal-alpine