Try it free

OpenPipeline self-monitoring metrics

  • Latest Dynatrace
  • Reference
  • 4-min read
  • Published Jun 30, 2026

You can use self-monitoring (SFM) metrics to track how data flows in OpenPipeline through the pipeline stages, from ingestion to storage and forwarding. Use these metrics to detect unexpected drops, identify bottlenecks, and correlate configuration changes with changes in data flow.

The OpenPipeline usage overview ready-made dashboard provides a preconfigured view of these metrics.

Count metrics

MetricDescriptionDimensions

dt.sfm.openpipeline.ingest_sources_in.records

Measured at the point where OpenPipeline takes over the data. Counts records entering into an ingest source, per data type and source path.

aws.region, aws.resource.type, azure.resource.type, cloud.provider, configuration, dt.openpipeline.source, gcp.asset.type, maintenance.is_under_maintenance

dt.sfm.openpipeline.routing.records

Measured after routing. Counts records processed through routing rules, per data type, route, and target pipeline.

aws.region, aws.resource.type, azure.resource.type, cloud.provider, configuration, dt.openpipeline.source, gcp.asset.type, maintenance.is_under_maintenance, pipeline_group_id, pipeline_id, route_name

dt.sfm.openpipeline.pipelines_out.records

Measured at the end of OpenPipeline. Counts records leaving a pipeline after processing, per data type and storage bucket.

aws.region, aws.resource.type, azure.resource.type, bucket_name, cloud.provider, configuration, dt.openpipeline.source, gcp.asset.type, maintenance.is_under_maintenance, pipeline_group_id, pipeline_id, route_name

dt.sfm.openpipeline.not_stored.records

Records that were not stored—dropped, not persisted, or invalid. Can occur during the ingest phase (pre-processing) or within the pipeline (processing). Check pipeline_id to determine where: present means a pipeline, absent means an ingest source.

configuration, dt.openpipeline.source, pipeline_id, reason

dt.sfm.openpipeline.forwarding.successful_records

Records successfully forwarded to an external destination. For setup, see Forwarding data to cloud storage via OpenPipeline.

configuration, dt.openpipeline.source, forwarding.destination, forwarding.id, forwarding.name

dt.sfm.openpipeline.forwarding.failed_records

Records that failed to be forwarded to an external destination. For setup, see Forwarding data to cloud storage via OpenPipeline.

configuration, dt.openpipeline.source, forwarding.destination, forwarding.id, forwarding.name, reason

Dimensions

DimensionDescriptionValue

aws.region

The cloud datacenter region of the AWS monitored entity. See Global field reference.

For example, us-east-1

aws.resource.type

The cloud resource kind of the AWS monitored entity. See Global field reference.

For example, AWS::EC2::Instance

azure.resource.type

The cloud resource kind of the Azure monitored entity. See Global field reference.

For example, MICROSOFT.SQL/SERVERS/DATABASES

bucket_name

The Grail bucket assigned for storage. Dynatrace default buckets are prefixed default_*. When <NoBucket>, no bucket was assigned. See How to organize your data stored in Grail.

For example, my_logs_bucket

cloud.provider

The cloud provider of the monitored entity's infrastructure.

The dimension can hold these values
  • aws—Amazon Web Services
  • azure—Microsoft Azure
  • gcp—Google Cloud Platform
  • null—No cloud provider is associated with the record.

configuration

The configuration scope of the pipeline.

The dimension can hold these values
  • bizevents—Business events
  • davis.events—Davis AI events
  • davis.problems—Davis AI problems
  • events—Generic events
  • events.sdlc—Software delivery lifecycle events
  • logs—Logs
  • metrics—Metrics
  • security.events—Security events
  • spans—Distributed traces / spans
  • system.events—System events
  • user.events—User events
  • usersessions—RUM user sessions

dt.openpipeline.source

The ingest API endpoint path that received the records. See Ingest sources in OpenPipeline.

For example, /api/v2/logs/ingest

forwarding.destination

The type of cloud storage destination.

The dimension can hold these values
  • AWS_S3—Amazon S3
  • GCP—Google Cloud Storage
  • AZURE—Azure Blob Storage

forwarding.id

Dynatrace-assigned unique opaque ID of the forwarding configuration.

For example, vu9U3hXa3q0AAAABAClidWlsdGluOm9wZW5waXBlbGluZS5sb2dzLmRhdGEtZm9yd2FyZGluZwAGdGVuYW50AAZ0ZW5hbnQAJDE2ZWM0ZTA3LTIyZmEtMzAzZi04NWMzLTNiZmZkYTA1Y2M1Y77vVN4V2t6t

forwarding.name

User-defined display name of the forwarding configuration.

For example, My forwarding configuration

gcp.asset.type

The cloud resource kind of the GCP monitored entity. See Smartscape - Google Cloud Platform.

For example, compute.googleapis.com/Instance

maintenance.is_under_maintenance

Whether the entity is currently under a maintenance window.

The dimension can hold these values
  • true—The entity is under a maintenance window.
  • false—The entity is not under a maintenance window.

pipeline_group_id

Dynatrace-assigned opaque ID of the pipeline group that the pipeline belongs to.

For example, vu9U3hXa3q0AAAABAClidWlsdGluOm9wZW5waXBlbGluZS5sb2dzLnBpcGVsaW5lLWdyb3VwcwAGdGVuYW50AAZ0ZW5hbnQAJDczOTc1MTgzLWFiOWQtMzg2Ni1iYWIxLWRmMTE1NjdmZjRjZr7vVN4V2t6t

pipeline_id

The ID of the pipeline. Predefined for built-in pipelines (for example, default and extension.mysql-logs) or user-defined. For not_stored.records, absent if the drop occurred during ingest (pre-processing).

For example, my-pipeline

reason (not stored)

The reason a record was not stored. Applies to not_stored.records.

The dimension can hold these values
  • not_persisted—A No storage assignment processor in the Storage stage explicitly matched the record, preventing it from being assigned to a Grail bucket.
  • intentionally_dropped—A Drop record processor in the Processing stage explicitly discarded the record.
  • not_valid—The record failed schema validation.
  • buffer_overflow—Processing memory was exhausted. The field modifications consumed all the available processing memory and the record was dropped.
  • processing_time_exceeded—The record was dropped because processing exceeded the allowed time limit.
  • nesting_level_too_deep—A DQL transformation caused the record's field nesting to exceed the allowed depth limit. See limit.
  • internal_error—An unexpected internal error occurred during processing.

reason (forwarding failure)

The reason a record failed to be forwarded. Applies to forwarding.failed_records.

The dimension can hold these values
  • unauthorized—The cloud storage destination returned an authentication or permission failure.
  • bucket_not_found—The target storage bucket can't be found or doesn't exist.
  • resource_unavailable—The cloud storage destination is unreachable.
  • target_configuration_missing—The target cloud storage destination in the forwarding configuration is misconfigured or missing.
  • other—Unclassified failure.

route_name

The routing rule that matched the record. Predefined for built-in pipelines (for example, default) or user-defined.

For example, my-route

Sample DQL queries

Run these queries in Notebooks Notebooks or Dashboards Dashboards.

Incoming records by configuration

timeseries records = sum(dt.sfm.openpipeline.ingest_sources_in.records),
by: { configuration }

Not-stored records split by reason

timeseries records = sum(dt.sfm.openpipeline.not_stored.records),
by: { configuration, reason }

All count-based metrics combined

timeseries {
  ingested   = sum(dt.sfm.openpipeline.ingest_sources_in.records),
  routed     = sum(dt.sfm.openpipeline.routing.records),
  out        = sum(dt.sfm.openpipeline.pipelines_out.records),
  not_stored = sum(dt.sfm.openpipeline.not_stored.records)
},
by: { configuration }

Share of not-stored records per pipeline

timeseries {
  routed     = sum(dt.sfm.openpipeline.routing.records),
  not_stored = sum(dt.sfm.openpipeline.not_stored.records)
},
by: { pipeline_id }, union: true,
filter: { isNotNull(pipeline_id) }
| fieldsAdd not_stored_pct = if(isNull(not_stored[]), 0, else: (not_stored[] / routed[] * 100))

Forwarding — successful vs. failed

timeseries {
  successful = sum(dt.sfm.openpipeline.forwarding.successful_records),
  failed     = sum(dt.sfm.openpipeline.forwarding.failed_records)
},
by: { forwarding.name }
Related tags
Dynatrace Platform