Try it free

Upgrade from classic pipeline to OpenPipeline

  • Latest Dynatrace
  • Upgrade guide
  • 13-min read

OpenPipeline handles ingestion, routing, and processing of all record types—from logs and business events to spans, Smartscape events, and more—with Grail storage, DQL-based processing, and object-level access control. OpenPipeline is the Latest Dynatrace solution to processing and replaces the classic pipeline for logs and business events.

This article walks you through the principles to manually convert your classic processing rules to OpenPipeline, permission management, access control for your teams, and routing so that teams can get started with processing in OpenPipeline independently.

Why upgrade?

  • Unified processing model: Manage processing for multiple signal types in OpenPipeline, including logs, business events, security events, spans, and more.
  • Scalable data handling: OpenPipeline handles high throughput at scale, supporting additional ingest sources and increased data volume.
  • Grail data flow: Processing is Grail-based only, providing a consistent approach from ingest through DQL processing to storage.
  • Pipeline groups: Enforce global processing while enabling teams to process their data independently and safely. For example, use pipeline groups for masking and permissions.
  • Granular ownership: With OpenPipeline scoped ownership and access to pipelines, teams can manage day‑to‑day processing, and administrators can retain control.

What is new?

  • Smartscape integration
  • Data forwarding
  • Cost and product allocation
  • Data extraction
  • Higher limits and flexibility

What will you do?

You'll identify the data sets processed by the classic processing rules and the users responsible for this data. You'll then build a pipeline group with a base pipeline translating the classic processing rules and an empty catch-all member pipeline to route all records. Once the catch-all member pipeline route is active, data starts being processed by OpenPipeline instead of the classic pipeline.

The classic pipeline remains active as a fallback until you can validate the new configuration. Finally, you can turn off classic processing rules.

Before you begin

Prerequisites

  • Dynatrace version 1.295+

  • Dynatrace SaaS environment powered by Grail and AppEngine

  • DPS license with log or business event capabilities

  • Permissions:

    • openpipeline:configurations:write
    • settings:objects:admin

  • Extensions are upgraded to their latest major version. Older versions may reference classic entity fields that are no longer supported in Latest Dynatrace. For more information, see Manage Extensions.

Prior knowledge

  • Familiarity with classic processing rules for log or business event
  • Basic understanding of DQL
  • Knowledge of pipeline access control

Breaking changes

Some behaviors differ between the classic pipeline and OpenPipeline and can break downstream consumers (DQL queries, dashboards, alerts, automations). The following table summarizes the key technical differences of processing logs via the log classic pipeline and OpenPipeline.

Technical pointLog classic pipelineOpenPipelineRequired action

Data type support

String

String, Number, and Boolean

Review queries that relied on implicit string coercion

Content field limit

512 kB

10 MB

No action required

Field name case sensitivity

Case-insensitive

Case-sensitive1

Update DQL queries and downstream consumers to use OpenPipeline field casing before rerouting data

Query language

LQL, DQL

DQL

Translate LQL processing statements to DQL; see Conversion to DQL for Logs and the classic command reference

Connect log data to traces

Built-in rules

Automatic2

No action required

Technology parsers

Built-in rules

Technology bundles

Add a Technology bundle in the Processing stage for each relevant built-in technology rule of the classic pipeline.

Metric dimension naming

Not supported

Supported

No action required. Existing definitions keep working; new metrics can use dimension names.

Metric-key log prefix

Mandatory

Optional

No action required. Existing definitions keep working; new metrics can drop the prefix.

Classic references on telemetry

Classic entities and classic metrics are extracted from telemetry data. Telemetry data is enriched with classic-entity fields (dt.entity.host, dt.entity.process_group_instance, dt.entity.process_group, dt.entity.service, dt.entity.cloud_application, dt.entity.cloud_application.instance, dt.entity.cloud_application_namespace, dt.entity.kubernetes_cluster, dt.entity.container_group_instance, dt.entity.container_group, dt.source_entity, dt.source_entity.type)

Classic entities, classic metrics, and classic-entity fields aren't applicable to Latest Dynatrace.

Review and update DQL queries, dashboards, alerts, and automations based on classic entities, classic metrics, or classic-entity fields.

1

When you ingest logs via Log Monitoring API v2 - POST ingest logs, field names are automatically converted to lowercase after data is routed to the Classic pipeline. While both pipelines run side by side, two records of the same type can end up with inconsistent casing depending on which pipeline they hit, silently breaking case-sensitive queries.

2

The enrichment runs automatically and is no longer rule-based, so it isn't visible as a processor in your pipeline.

New concepts

Pipeline

Collection of processors executed in an ordered sequence of stages to structure, separate, and store data.

Processor

Pre-formatted processing instruction that focuses either on modifying or extracting data. It contains a configurable matcher and processing definition.

Pipeline group

Set of team-managed pipelines to which a shared configuration applies. The shared configuration can restrict or mandate processing, enabling centralized processing across multiple pipelines.

Routing

Directing data to a pipeline, either based on matching conditions (dynamic) or by explicit pipeline selection (static).

How to upgrade

The steps below use Dynatrace web UI. The Settings API is available as well. To see an end-to-end example with JSON payloads, see Configure multi-cloud ingest governance with pipeline groups.

1. Check your migration readiness

The OpenPipeline usage overview ready-made dashboard shows the current state of your migration. It tracks how much of your log and business event volume still flows through the classic pipeline versus OpenPipeline, and lists each record type with its migration status.

  1. Go to Dashboards Dashboards > OpenPipeline usage overview.

  2. Verify that the percentage in Ingest via OpenPipeline vs. classic pipeline is above 0%.

    When the percentage is above 0%, your data is still reaching the classic pipeline. When you go to Settings Settings > Process and contextualize > OpenPipeline and select your configuration scope (Logs or Business events), you still see the Classic route in Dynamic routing.

2. Identify the data sets currently processed by classic pipelines

  1. Go to Notebooks Notebooks or Logs Logs and filter on the dt.openpipeline.pipelines field to identify the records routed to the classic pipeline.

    fetch logs
    | filter in(dt.openpipeline.pipelines, "logs:default")

  2. Review the corresponding classic processing rules and their context:

    1. Go to Settings Settings > Process and contextualize > OpenPipeline and select your configuration scope (Logs or Business events).
    2. Go to Pipelines > Classic pipeline to view your classic processing rules.
    3. Work on one data stream at a time to reduce risk and simplify validation. For each rule, perform the following actions:
      1. Learn the matcher expressions and processor definitions. If sample data is used, export it for reuse during testing.
      2. Identify downstream consumers such as dashboards, alerts, metrics, and automations to understand potential impact.
      3. Identify the users or teams responsible for that data.

2. Create the pipeline group and base pipeline

  1. Create an empty base pipeline.

    1. In OpenPipeline, go to Pipelines > Pipeline.
    2. Set the role to Base pipeline, enter a name, and select Save.

  2. Create the pipeline group.

    1. Go to Pipeline groups > Add group.
    2. Enter a name for the new group.
    3. Add your base pipeline to the group and enable all stages.
    4. Position the member pipeline placeholder below your base pipeline.
    5. Select Save.

The base pipeline and the pipeline group are listed in the respective tables.

3. Convert the classic processing rules to the base pipeline

Translate every classic processing rule into a processor on the base pipeline. These processors will apply to all records routed to the member pipelines assigned to the group.

Use the OpenPipeline API - GET translate classic pipeline to generate an OpenPipeline configuration from your existing classic pipeline definition. The API returns a pipeline JSON payload that you can use as a starting point, reducing manual translation effort.

OpenPipeline organizes processors into stages, including:

  • Processing—Parse, transform, filter, and mask records.
  • Metric extraction—Extract metrics from records.
  • Data extraction—Extract and re-ingest a new record (business event or Davis event).
  • Bucket assignment—Assign records to buckets.

For the complete list of stages and the processors available, see Processing in OpenPipeline.

The configuration workflow is

  1. In the base pipeline, open the stage that matches what the classic rule does.
  2. Select to choose the processor.
  3. Define the processor.
    • You can reuse the Matcher and the Sample data from the classic processing rule.
    • Account for breaking changes.
  4. Test the configuration using sample data.
  5. Once you're satisfied with the result, select Save.

No data is affected at this stage. The base pipeline is inactive until the catch-all member pipeline route is active.

4. Create the catch-all member pipeline and route data to it

When in a group, member pipelines support routing and inherit the base pipeline configuration. In this case, the catch-all member pipeline stays empty—its goal is to receive all the records to apply the base pipeline configuration to.

  1. Create the catch-all member pipeline.

    1. Go to Pipelines > Pipeline.
    2. Set the role to Member pipeline, enter a name, and select Save. Leave the stages empty.

  2. Add the catch-all member pipeline to the pipeline group.

    1. Go to Pipeline groups and select the pipeline group you created.
    2. Scroll to Member pipelines and assign the new pipeline to the group.

  3. Create the associated route.

    1. Go to Dynamic routing > Dynamic route.
    2. Enter true as a matching condition and choose the catch-all member pipeline as target.
    3. Select Add. The new route is added to the table and set to active by default.
    4. Position the route above the default route.
    5. Select Save.

The new route is active.

  • Data that matches the condition is processed according to the base pipeline.
  • Data that doesn't match falls back to the default route and continues to be processed by the classic pipeline until you turn off the rules.

Keep classic rules enabled until all data is reliably routed and processed by OpenPipeline. Compare OpenPipeline output against the classic processing rule output until the results match. If they don't, identify translation gaps and fix them in the base pipeline.

5. Turn off the corresponding classic processing rules

After you have validated your OpenPipeline configuration, turn off the classic processing rules you converted.

  1. Go to Dashboards Dashboards > OpenPipeline usage overview and verify that the percentage in Ingest via OpenPipeline vs. classic pipeline is 0%.

    The dashboard shows the percentage of records migrated based on routing. Zero classic pipeline usage does not guarantee that all data is fully converted, only that it was not routed via the Classic route. Before turning off a classic processing rule, ensure that data that arrives rarely or does not match any OpenPipeline route is accounted for, including sensitive data, masking, and filtering.

  2. If the remaining classic pipeline usage comes from processing rules your environment no longer needs, you can turn off those rules even if usage is above 0%.

  3. To turn off the classic processing rules:

    1. Go to Settings Settings > Process and contextualize > OpenPipeline and select your configuration scope (Logs or Business events).
    2. Go to Pipelines > Classic pipeline to view your classic processing rules.
    3. Turn off the classic processing rules.

6. Set up access control for your teams

The first user to create or edit a pipeline becomes its owner. By default, only the owner and administrators can see and manage it. For teams to get started with processing in OpenPipeline independently, as administrator, you need to set up IAM policies and access control for them.

  1. Grant environment-level permissions.

    1. Go to Account Management > Identity and Access Management.

    2. Create a policy to grant read or write access to the relevant pipeline schemas for a user or user group. For example, to grant write access to log and business event pipelines:

      ALLOW settings:objects:write WHERE settings:schemaId IN ("builtin:openpipeline.logs.pipelines", "builtin:openpipeline.bizevents.pipelines")

      For more policy patterns, see Owner-based access control in OpenPipeline.

  2. Review and distribute object-level access.

    Once permissions are in place, review pipeline ownership and share access with the teams responsible for each pipeline. As administrator, you can transfer ownership or grant accessor rights to any pipeline. To learn how to access, transfer ownership, or make a pipeline public, see Set access control in OpenPipeline.

Upgrade complete

The classic pipeline no longer processes your data. The pipeline group in OpenPipeline contains a base pipeline with the equivalent processing logic of your classic pipeline and a member pipeline to route all records. Your teams can access the pipelines and manage data-processing independently.

Next steps

The upgrade establishes parity with your classic pipeline setup. You can refine, split, and validate iteratively.

  • Refine processing.

    1. Inspect the OpenPipeline output in Notebooks Notebooks to identify gaps or opportunities.
    2. Add or update processors on the base pipeline to address what you observe.
    3. Verify the result in Notebooks Notebooks.
    4. Check the OpenPipeline usage overview dashboard to confirm coverage hasn't dropped.

  • Split processing by teams or services.

    You can add member pipelines that perform specific processing, split by team or by service, and hand ownership to teams.

    1. Review which stages are enabled for the base pipeline and the member pipeline placeholder. New member pipelines inherit the same stage configuration, so confirm the enabled stages match what each team needs.
    2. Create member pipelines for a specific team or service.
    3. Create the associated route and position it above the catch-all one.
    4. Assign the new member pipelines to the pipeline group.
    5. Manage the pipeline access and ownership.

Repeat for each team or service. The base pipeline converges to only the rules that are truly generic and shared across all data; each dedicated member pipeline owns its specific processing.

Learn more

  • Data flow in OpenPipeline: The end‑to‑end path data follows from ingest through storage.

  • Processing in OpenPipeline: Pipelines, stages, and processors used to transform data.

  • Owner-based access control in OpenPipeline: Policies and scopes that manage pipeline access level and ownership.

  • OpenPipeline pipeline groups: Group setup for shared and enforced pipeline stages.

  • Configure a processing pipeline: Step‑by‑step pipeline configuration guidance.

  • OpenPipeline processing examples: Examples of OpenPipeline processor configuration that can be compared with the log processing examples to clarify conceptual differences.

    Example: Rename attributes

    Classic pipeline

    USING(INOUT to_be_renamed, content)
    | FIELDS_RENAME(better_name: to_be_renamed)

    OpenPipeline

    Rename fields processor: Enter the field name that you want to be renamed and the new name.

    Example processor that renames fields in OpenPipeline
    Example processor that renames fields in OpenPipeline

FAQ

Can classic pipelines and OpenPipeline run side by side?

Yes.

Data is processed according to the first matching route. As long as the classic processing rules are in place in your environment, the classic pipeline is accounted for in OpenPipeline and is the default processing mechanism. When you create new pipelines and associated routes, position the new route above the default route so that OpenPipeline processes data accordingly. If some data doesn't match the new route condition, it's still routed by the default route to the classic pipeline.

Related topics

  • Log processing
  • Business event processing
Related tags
Log AnalyticsBusiness Observability