OpenPipeline limits

  • Latest Dynatrace
  • Reference
  • 4-min read

The following page lists the default limits of Dynatrace OpenPipeline.

Configuration scope limits

Limitations specific to configuration scopes might override OpenPipeline generic limits. For limits specific to the configuration scope, see

Limits specific to fields

Fields with limits for all configuration scopes

  • The following fields can be viewed-only; editing via OpenPipeline is not supported.

    • dt.ingest.*
    • dt.openpipeline.*
    • dt.retain.*
    • dt.system.*

  • The following fields are added after the Processing stage when Dynatrace runs its entity detection. You can use them only in stages after the Processing stage, but not in pre-processing, routing, or the Processing stage.

    • dt.entity.aws_lambda_function
    • dt.entity.cloud_application
    • dt.entity.cloud_application_instance
    • dt.entity.cloud_application_names
    • dt.entity.custom_device
    • dt.entity.<genericEntityType>
    • dt.entity.kubernetes_cluster
    • dt.entity.kubernetes_node
    • dt.entity.kubernetes_service
    • dt.entity.service
    • dt.env_vars.dt_tags
    • dt.kubernetes.cluster.id
    • dt.kubernetes.cluster.name
    • dt.loadtest.custom_entity.enriched_custom_device_name
    • dt.process.name1
    • dt.source_entity
    • k8s.cluster.name2
    1

    To obtain equivalent results before the Processing stage, you can use dt.process_group.detected_name instead.

    2

    OneAgent version 1.309Dynatrace Operator version 1.4.2+The field is available before the Processing stage if OneAgent Log module is running in standalone mode.

Fields with limits for metrics

The use of the following fields for metrics in OpenPipeline is limited.

  • Fields excluded from dynamic route matching conditions and in the Processing stage

    • dt.entity.*

  • Fields excluded from the Processing stage

    • dt.system.monitoring_source
    • metric.key
    • metric.type
    • timestamp
    • value

Fields with limits for spans

The use of the following fields for spans in OpenPipeline is limited.

  • Fields excluded from dynamic route matching conditions and in the Processing stage

    • dt.entity.service
    • endpoint.name
    • failure_detection.*
    • request.is_failed
    • request.is_root_span
    • service_mesh.is_proxy
    • service_mesh.is_failed
    • supportability.*

  • Fields excluded from the Processing stage

    • dt.ingest.size
    • dt.retain.size
    • duration
    • end_time
    • span.id
    • start_time
    • trace.id

Ingestion

Record maximum timestamp

If the timestamp is more than 10 minutes in the future, it's adjusted to the ingest server time plus 10 minutes.

Record minimum timestamp

Records outside of these timeframes are dropped.

Ingest API

Timestamp value

Numerical and string timestamp values are supported. OpenPipeline parses the timestamp as follows.

  • Numerical values
    • Up to 100_000_000_000 are parsed as SECONDS.
    • Up to 100_000_000_000_000 are parsed as MILLISECONDS.
    • Up to 9_999_999_999_999_999 are parsed as MICROSECONDS.
  • String values are parsed either as
    • UNIX epoch milliseconds or seconds
    • RFC3339 formats
    • RFC3164 formats
  • For other values that cannot be parsed, timestamp is overwritten with the ingest time.

If the record doesn't have a timestamp field, the field timestamp is set to ingest time.

Processing

Processing memory exhaustion

Processing memory is limited. Each change to a record—for example, parsing a field—decreases the available processing memory. Once the available processing memory is exhausted, the record is dropped. This is reported in metric dt.sfm.openpipeline.not_stored.records with dimension reason set to buffer_overflow.

Size of record after processing

The maximum size of a record after processing is 16 MB.

Size of extracted log attributes

Log attributes can be up to 32 KB in size. When log attributes are added to the event template, the size of each attribute is truncated to 4,096 bytes.

Number of extractions for a single record

You can extract data on a single record in a maximum of five different pipelines (dt. open pipeline.pipelines). Once the threshold is exceeded, data extraction is no longer performed on the single record. The record continues to be processed and persisted.

Schema validation for logs

A processed log is persisted if the following conditions are satisfied.

FieldExistsAccepted TypesValue Constraints
timestampYesString, NumericalWithin the ingestion range
contentYesStringNot evaluated

If the schema is not valid the log is dropped.

Smartscape node processor

The Smartscape ID calculation supports string only. The ID components must be of type string.

Pre-process records to convert the values you need to the string data type.

Configuration

Allowed characters in the endpoint path

The endpoint path is a unique name starting with a literal that defines the endpoint. It's case-insensitive and supports alphanumeric characters and dot (.). For example: Endpoint.1.

Endpoint path doesn't support:

  • Dot (.) as the last character
  • Whitespaces
  • Consecutive dots (..)
  • Null or empty input

Pipeline groups

  • The maximum number of composition pipelines per pipeline group is 10.
  • The maximum number of member pipelines per pipeline group is 100.
  • The pipeline role is permanent. Converting roles—from member to composition, or composition to member—isn't supported.
Related tags
Dynatrace Platform