Primary Grail fields and tags enrichment through OneAgent

  • Latest Dynatrace
  • Explanation
  • 10-min read

OneAgent version 1.331+

OneAgent can enrich telemetry data at the source with custom metadata before sending metrics, spans, logs, events, and entities. The resulting data is immediately usable for platform features like segments, pipeline routing, bucket assignment, and cost allocation.

Why OneAgent-based enrichment?

Enriching primary Grail fields and tags at the OneAgent level empowers teams to take ownership of enrichment at deployment time. By enriching at the source, data is consistently enriched across all signals and Smartscape nodes from the very beginning.

Host-level enrichment of primary Grail fields and tags automatically propagates to Smartscape nodes, including hosts, processes, containers, networks, disks, and their associated signals. This approach eliminates the need for additional configuration in Dynatrace and allows enrichment responsibility to be delegated to the teams managing deployments. In Latest Dynatrace, primary tags and primary fields are the foundation for OpenPipeline routing, bucket assignment, Grail permissions, cost allocation, data segmentation, and alerting.

Platform CapabilityPrimary Grail tags and fields use-cases

Data Routing and Processing

Route data to specific pipelines based on primary_tags.* or primary fields

Bucket Assignment

Assign target retention bucket for storing data based on primary_tags.* or primary fields

Grail Permissions

Prepare the foundation for data access control by deriving security context

Cost Allocation

Track costs per team/product with dt.cost.costcenter and dt.cost.product

Segments

Define and use segments based on primary tags and fields to structure and easily filter your data across apps in Dynatrace

Alerting

Create targeted alerts and notifications based on primary tags and fields

Example

A platform team needs to separate production checkout service data into a high-retention bucket, restrict access to PCI-compliant users, and track costs to the payments cost center. To have these signals immediately queryable in Grail, you can enrich your data through the following tags and field definitions:

primary_tags.environment=production
primary_tags.team=bravo
primary_tags.application=payment-processing
dt.security_context=pci-compliant
dt.cost.costcenter=payments

Data enrichment with primary Grail fields and tags

Primary Grail fields

Primary Grail fields defined in the semantic dictionary are automatically enriched by OneAgent. Most primary Grail fields are determined by OneAgent and can't be modified.

You can define the following primary Grail fields at the OneAgent level for security and cost allocation:

FieldPurpose

dt.security_context

Data access control

dt.cost.costcenter

Cost allocation by cost center

dt.cost.product

Cost allocation by product

Primary Grail tags

Primary Grail tags are customer-defined attributes that are automatically added to all ingested telemetry. They are identified by the reserved primary_tags. prefix and enable you to organize your data based on your most important organizational attributes. Examples include:

Field nameExample values

primary_tags.team

alpha, frontiers, hearts, bravo

primary_tags.environment

production, staging, playground, dev

primary_tags.country

US, MX, CA, UK

primary_tags.application

fulfillment, warehouse, payment, mobile

primary_tags.appid

8712, E874, 0854, 7613

primary_tags.cmdbid

CI0001234, APP045678, SRV009812

primary_tags.business_unit

ecommerce, finance, HR, research

primary_tags.region

EMEA, NORAM, APAC, LATAM

Set up primary Grail fields and tags for hosts and processes

You can configure primary Grail tags and fields in the following ways:

Option 1: Define fields and tags at OneAgent deployment

You can set primary Grail fields and tags at installation time by passing --set-host-tag parameters directly to the OneAgent installer. OneAgent will enrich all telemetry data from that host with the provided primary tags and fields.

Linux example
Dynatrace-OneAgent-Linux.sh --set-host-tag="primary_tags.environment=production" --set-host-tag="dt.security_context=confidential"
Windows example
.\Dynatrace-OneAgent-Windows.exe --set-host-tag="primary_tags.environment=production" --set-host-tag="dt.security_context=confidential"

For more information on customizing OneAgent installations, see Customize OneAgent installation on Linux or Customize OneAgent installation on Windows.

Monitoring modes: FullStack, Infrastructure, Discovery.

Option 2: Define fields and tags for existing OneAgent installations using oneagentctl

Use oneagentctl to manage primary Grail fields and tags programmatically. OneAgent will enrich all telemetry data from that host with the provided primary tags and fields. For command details, see OneAgent configuration via command-line interface.

Set Primary Grail tags
# Set fields for cost allocation and security
oneagentctl --set-host-tag="dt.cost.costcenter=12345"
oneagentctl --set-host-tag="dt.cost.product=checkout-service"
oneagentctl --set-host-tag="dt.security_context=confidential"
# Set primary tags
oneagentctl --set-host-tag="primary_tags.environment=production"
oneagentctl --set-host-tag="primary_tags.team=bravo"
oneagentctl --set-host-tag="primary_tags.business_unit=finance"

Monitoring modes: FullStack, Infrastructure, Discovery.

Option 3: Define fields and tags at process level using DT_TAGS

When you use the DT_TAGS environment variable at the process level, the OneAgent enriches data for this process. Any defined primary tag or field in this environment variable is enriched on all telemetry data for that process and overrides their equivalents on the host level.

The process-level environment variable takes precedence for process-scoped signals: when you define the same primary tag or field using DT_TAGS and at the host-level, the one that is closer to the monitored entity is taken and overrides the host-level value for the same key.

Example for process-level primary tags
DT_TAGS="primary_tags.environment=production primary_tags.team=bravo primary_tags.application=payment-processing" java -jar checkout-service.jar

For serverless and managed container platforms, DT_TAGS is the recommended enrichment method.

Monitoring modes: FullStack, Infrastructure, Discovery, and Application-only scenarios.

Precedence, overrides, and limits

When the same tag key or field is set at both the host and process levels, the process-level definition (DT_TAGS) takes precedence and overrides the host-level one for that process. Host-level tags and fields apply to all observability data from a host unless a process explicitly sets the same key using DT_TAGS.

Example

ResourceEnrichment levelEnrichment valuesEnrichment result

Host

oneagentctl or install parameter

primary_tags.business_unit=ecommerce, primary_tags.environment=dev

primary_tags.business_unit=ecommerce, primary_tags.environment=dev

.NET Process

DT_TAGS (process level)

primary_tags.business_unit=finance, primary_tags.team=alpha

primary_tags.business_unit=finance, primary_tags.team=alpha, primary_tags.environment=dev

Java Process

DT_TAGS (process level)

primary_tags.business_unit=payment, primary_tags.team=hearts, primary_tags.environment=hardening

primary_tags.business_unit=payment, primary_tags.team=hearts, primary_tags.environment=hardening

Any other process

primary_tags.business_unit=ecommerce, primary_tags.environment=dev

Tag limits

You can define up to 20 tags for host- or process-level enrichment. When more than 20 tags are defined, the first 20 take precedence. Tags defined at the process level override the host-level tags.

If you exceed the limit of 20, OneAgent doesn't emit any warning and does not enrich data with the excess tags.

ScenarioHost tagsProcess tagsResult

Combined limit exceeded

20 defined

10 new (unique)

First 10 from process + first 10 from host = 20

Within limit

10 defined

10 new (unique)

All 20 tags applied

Process overwrites host

10 defined

10 (same keys as host)

10 process tags (overwrite host values)

Query enriched data in Grail

Primary Grail fields and tags appear as top-level fields in the data and can be queried with DQL as follows.

Find all spans from team bravo in production
fetch spans
| filter primary_tags.team == "bravo" AND primary_tags.environment == "production"
Cost allocation query
fetch bizevents
| filter dt.cost.costcenter == "payments"
| summarize sum(value), by: {dt.cost.product}
Security context filtering
fetch logs
| filter dt.security_context == "confidential"
| filter primary_tags.application == "payment-processing"

Limitations

The following are not covered by OneAgent primary grail fields and tags:

  • Dynatrace Classic platform.
  • Mainframe deployments.
  • Mapping of existing tags to all signals without primary_tags.* prefix.
Related tags
Application Observability