Extend traces using OpenTelemetry

OpenTelemetry is a collection of tools, APIs, and SDKs that enable you to use telemetry data (distributed traces, metrics, and logs) to get insights into your application's performance and behavior.

OpenTelemetry with the z/OS Java code module enables you to enrich or extend distributed traces.

OpenTelemetry metrics and logs are currently not supported by the z/OS Java code module.

OpenTelemetry interoperability

OpenTelemetry version 1.0+

Enabling OpenTelemetry interoperability connects the z/OS Java code module to the OpenTelemetry API. When enabled, the code module redirects certain OpenTelemetry API usage (for example, GlobalOpenTelemetry) to the internal Dynatrace OpenTelemetry SDK.

The z/OS Java code module forwards the captured OpenTelemetry Spans, via the zDC subsystem and zRemote module, to your Dynatrace environment.

Enable OpenTelemetry interoperability

OpenTelemetry interoperability is disabled by default. To enable it, add the OpenTelemetry: EnableIntegration attribute to your dtconfig.json file as shown in the following example.

{
  "OpenTelemetry": {
    "EnableIntegration": true
  }
}

Alternatively, you can add the open-telemetry-enable-integration option to the JVM argument of the z/OS Java code module:

-javaagent:/PATH_TO/dynatrace-oneagent-zos-java.jar=open-telemetry-enable-integration=true

OpenTelemetry instrumentation samples

To support various use cases, OpenTelemetry enables you to add vendor-neutral custom instrumentation to your applications. Instrumenting applications with OpenTelemetry requires programming knowledge and access to the application’s code. To learn how to instrument your application, refer to OpenTelemetry documentation and OpenTelemetry Java documentation.

See the examples below for using OpenTelemetry Java.

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapPropagator;

public class AuditService {

    public static void sendAuditEntry(Order order) {

        // Your application code: declare an audit entry
        Map<String, String> auditEntry = new HashMap<>();
        auditEntry.put("name", order.getId().toString());
        auditEntry.put("description", order.getCustomer());

        // OpenTelemetry: Inject current context of audit entry and propagate it
        TextMapPropagator propagator = GlobalOpenTelemetry.getPropagators().getTextMapPropagator();
        propagator.inject(
            Context.current(),
            auditEntry,
            Map::put
        );

        // Your application code: write audit entry to objectOutputStream (Socket)
        Socket socket = new Socket(host, port);
        ObjectOutputStream objectOutputStream = new ObjectOutputStream(socket.getOutputStream())) {
        objectOutputStream.writeObject(auditEntry);
    }
}

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;
import io.opentelemetry.context.propagation.TextMapGetter;

public class AuditService {

    private static void receivedAuditEntry(Map<String, String> auditEntry) {

        // OpenTelemetry: declare the tracer, create a span and define it's scope
        Span span = GlobalOpenTelemetry.getTracer("auditing-center", "0.0.1")
            .spanBuilder("auditEntry")
            .setSpanKind(SpanKind.SERVER)
            .setAttribute("auditName", auditEntry.get("name"))
            .startSpan();
        try (Scope scope = span.makeCurrent()) {

            // Your application code: process audit entry
            // ...
        }
        span.end();
    }

    public static void readAuditEntryFromSocket(Socket socket) {
        ObjectInputStream objectInputStream = new ObjectInputStream(socket.getInputStream());
        Object input = objectInputStream.readObject();

        // OpenTelemetry: extract context of audit entry
        Context context = GlobalOpenTelemetry.getPropagators().getTextMapPropagator()
            .extract(
                Context.current(),
                input,
                new TextMapGetter<Map<String, String>>(){ /* ... */ });

        try (Scope scope = context.makeCurrent()) {
            receivedAuditEntry((Map<String, String>) input);
        }
        // ...
    }
}

Suppress spans from specific instrumentations

You can suppress spans coming from a particular instrumentation scope/library. To do so, add the library name to the OpenTelemetry: DisabledSensors parameter in name via the dtconfig.json file. You can use an asterisk (*) to exclude all instrumentation scope/library names starting with the preceding string. You can't use asterisk at the beginning or in the middle of a library name.

{
  "OpenTelemetry": {
    "EnableIntegration": true,
    "DisabledSensors": [
      "com.example.MyLib",
      "opentelemetry.urllib3*"
    ]
  }
}

Alternatively, you can add the open-telemetry-disabled-sensors option to the JVM argument of the z/OS Java code module:

-javaagent:/PATH_TO/dynatrace-oneagent-zos-java.jar=open-telemetry-disabled-sensors=com.example.MyLib:opentelemetry.urllib3*

The examples above suppress spans from the com.example.MyLib instrumentation scope/library and spans from all libraries starting with the name opentelemetry.urllib3.

Rules for spans that Dynatrace will report

Depending on the SpanKind, Dynatrace will suppress some OpenTelemetry spans:

Suppressed spans will not be visible in distributed traces.

Define a request attribute for span attributes

You can define a request attribute for any captured span attribute. To do so

To learn more details about span attributes and how to capture them, see Span settings.

You can find the Attribute key of your spans on the Distributed traces page in the Code level tab under Span attributes.