Preview
These commands are available only to customers enrolled in the Query and import archived data with DQL preview. For eligibility, conditions, and enrollment instructions, see the Preview program.
The commands on this page and their parameters are subject to change before general availability. Avoid relying on the current syntax in production pipelines or automated scripts.
Starting with general availability, consumption will be metered per query on the volume of data scanned, including list and load operations. During the preview, queries can still incur consumption on your existing rate card items when they include data from Grail.
The commands on this page let you list and parse data stored in an external datastore, such as an AWS S3 bucket, directly from DQL. These commands require an external datastore to be set up and access to be granted by an administrator. If you don't have access yet, ask your Dynatrace administrator to follow the instructions in Set up external datastores.
Run the following DQL query to check which external datastores you can access:
fetch dt.system.external_datastores
Lists external resources (objects, blobs, or files) in an external datastore that match the given pattern. In addition to the name and exact path, listing provides metadata such as the last-modified time and file size. This metadata can be used directly in queries or for query optimization, for instance to enable partitioned access.
Listing can become an expensive and time-consuming operation when used carelessly. This typically happens when listing files with a pattern that causes the remote system to scan a large number of files. Limit the pattern as much as possible, for example by using time patterns to only scan a relevant subset of files, use scanLimitGBytes to cap the billable scanned bytes, and pipe into limit to cap the number of returned records.
previewExternalList uri [, mode:] [, includeObjects:] [, includeDirectories:] [, timezone:] [, locale:] [, ignoreErrors:] [, scanLimitGBytes:]
| Parameter | Type | Description | Required |
|---|---|---|---|
uri | string | Cloud URI in the form | Required |
mode | string | Switches between listing all objects matching the naming pattern and grouping them by "directories" (common prefixes). Use | Optional |
includeObjects | boolean | Include objects (files) matching the pattern in the results. Default: | Optional |
includeDirectories | boolean | Include directory entries—common prefixes formed by the delimiter ( | Optional |
timezone | string | Timezone used for evaluating time patterns in the | Optional |
locale | string | Locale used for evaluating time patterns in the | Optional |
ignoreErrors | boolean | When set to | Optional |
scanLimitGBytes | long | The maximum amount of data processed by the listing operation. A partial result is returned if this limit is exceeded. Use the limit command in the pipeline to cap the number of returned records instead. | Optional |
The mode parameter is mutually exclusive with includeObjects and includeDirectories. The two modes map to the following parameter combinations:
mode:"flat" is equivalent to includeObjects:true, includeDirectories:falsemode:"browse" is equivalent to includeObjects:true, includeDirectories:trueFor mode:"browse", the URI path up to and including the last / must be a literal string with no wildcards—this prefix selects a single directory level to list. Wildcards (*, **, ?) are only permitted in the final path segment after the last /, where they filter the entries returned for that directory.
Each result record contains the metadata of one listed file or folder matching the URI pattern:
| Field | Description |
|---|---|
uri | Path to the object. |
file_name | Part of the path after the last slash ( |
offset | Offset in bytes. Used to slide the start position when parsing the content. Set to |
length | The number of bytes from the content to parse. Set to the size of the object for parsing the entire object. |
last_modified | Timestamp the object was last modified. |
path_time | Timestamp evaluated from the time patterns in the |
etag | Used internally for optimizations. Don't modify. |
is_object |
|
compression | The compression algorithm to use. For more information, see Decompression. |
Use mode:"browse" to explore the storage structure one level at a time. Results are grouped by common prefix, giving you an overview of what directories exist at the root of the datastore.
previewExternalList "cloud://my-datastore/*", mode:"browse"
Use the ** globstar to list every file nested under the events/ prefix, regardless of how many subdirectory levels deep they are.
previewExternalList "cloud://my-datastore/events/**"
Use a time pattern in the URI to extract a timestamp from each object key into the path_time field, then filter to only objects from the last 24 hours.
previewExternalList "cloud://my-datastore/logs/<yyyyMMdd>_<HHmmss>.<SSS>*"| filter path_time > now()-24h
Use timestamp() comparisons on path_time to scope a listing to a fixed historical window. This is useful when now() offsets aren't precise enough, for example when you need exactly a four-hour slice from a past date.
previewExternalList "cloud://my-datastore/logs/<yyyy>/<MM>/<dd>/<HH>/*"| filter path_time >= timestamp(2026,5,18,0,0,0) and path_time < timestamp(2026,5,18,4,0,0)
You need the storage:external-data:list permission, scoped to the datastore you want to list. For details, see Grant users access to an external datastore.
Loads the external data from the objects identified by previewExternalList and converts it into a stream of structured records. The command operates on text-based data: it reads object content as a UTF-8 text stream and parses it using a DPL pattern. Text-based formats such as JSON, CSV, and plain logs work; binary formats such as Parquet or Avro are not supported.
For each object returned by previewExternalList, the command downloads it and applies the DPL pattern to its content, emitting one output record per match—similar to how parseAll finds all pattern matches in a string.
previewExternalParse pattern [, skipRecords:] [, autoExtract:] [, scanLimitGBytes:]
| Parameter | Type | Description | Required |
|---|---|---|---|
pattern | string (DPL pattern) | DPL pattern for extracting and transforming the input data. | Required |
skipRecords | long | The number of initial records to discard. Can be used to skip header rows. Default: | Optional |
autoExtract | boolean | Set to | Optional |
scanLimitGBytes | long | The maximum amount of uncompressed data processed by the command. A partial result is returned if this limit is exceeded. | Optional |
A stream of records with the results of the parsing. The fields produced by previewExternalList (uri, file_name, offset, length, last_modified, path_time, etag, is_object, compression) are carried through, prefixed with dt. (for example dt.file_name), and can be selected with the field selection commands. Fields extracted from the DPL pattern, including any named path captures, are added as top-level fields.
The following table lists the supported compression formats. In general, the compression format is selected based on the filename extension. If it is not automatically identified, you can set the compression field in the input stream to the respective value in the table:
| Value | Algorithm | Recognized file extension | Tools producing compressed content |
|---|---|---|---|
| Gzip |
|
|
| Bzip2 |
|
|
| LZ4 |
|
|
| zlib |
|
|
| XZ |
|
|
| Zstandard |
|
|
| DEFLATE |
| raw DEFLATE entries in zips |
| DEFLATE64 |
| PKZIP, WinZip |
| No compression (overrides file extension recognition) | — | — |
Only streaming compression formats are supported. Archive formats such as zip, 7z (sevenz), rar, tar, or tgz/tar.gz are not supported.
The input stream is currently always read as UTF-8. Configuring a different charset isn't supported yet.
Use DATA{0,256000}:raw to read the first 256 KB into a single field named raw. This is useful for inspecting the content before writing a more specific DPL pattern. Use the DPL Architect to design your DPL pattern.
previewExternalList "cloud://my-datastore/path/to/file.log"| previewExternalParse "DATA{0,256000}:raw"
Use LD:line to read each line into a single field named line. This is useful for inspecting if the format of the content varies.
previewExternalList "cloud://my-datastore/path/to/file.log"| previewExternalParse "LD:line"
List compressed JSON files and pipe them into previewExternalParse to extract their contents. autoExtract:true promotes the root-level JSON keys to top-level fields in the result, making them immediately available for filtering and aggregation.
previewExternalList "cloud://forwarded-data/logs/<yyyyMMdd>_<HHmmss>.<SSS>*.json.gz"| previewExternalParse "JSON:json", autoExtract:true
Parse each line as raw text and filter by content. This is useful for quickly scanning many files before writing a more specific DPL pattern.
previewExternalList "cloud://my-datastore/logs/2026/05/*"| previewExternalParse "LD:line"| filter matchesPhrase(line, "ERROR")| limit 100
Named captures in the URI pattern, such as {account_id:*} or {region:*}, are available as regular fields both in the previewExternalList output and after previewExternalParse prefixed with dt.. Use them to group or filter parsed content by the storage path structure.
previewExternalList "cloud://my-datastore/AWSLogs/{account_id:*}/logs/{region:*}/<yyyy>/<MM>/<dd>/*"| previewExternalParse "LD:line"| summarize records = count(), by: {dt.account_id, dt.region}
You need the storage:external-data:load permission, scoped to the datastore you want to read from. For details, see Grant users access to an external datastore.
A URI pattern is a template that selects many cloud objects at once. The path after cloud://storeName/ can contain time tokens, wildcards, and named captures or alternation. The following example uses all of them:
cloud://my-datastore/logs/<yyyy>/<MM>/<dd>/{app:'payment-api'|'auth-service'}/*.log{'.gz'|''}
<yyyy>/<MM>/<dd> — time tokens extract date parts and assemble them into path_time. Listing is automatically narrowed to the partitions in the query's time range.{app:'payment-api'|'auth-service'} — a named capture with alternation: matches one of the two listed app names and captures which one matched as the column app.* — a wildcard: matches any characters except / within one segment.{'.gz'|''} — an anonymous optional alternation: matches .gz or nothing; no column is created.Any character in a pattern matches itself, except for the special pattern characters *, ?, {, }, <, and > described below. To match a special pattern character literally, wrap it in a single-branch capture, for example {'*'} for a literal asterisk.
File patterns used with previewExternalList support common globbing syntax for matching a pattern to individual objects, blobs, or files.
| Wildcard | Description | Example | Matches |
|---|---|---|---|
| Matches any characters except |
| All files in the directory |
| Matches zero or more complete path segments when enclosed between delimiters: |
|
|
| Matches any characters including |
| Any characters between |
| Matches any single character, except |
| Logs from early January 2016. |
A run of three or more consecutive * (***) is a parse error, and ** is only valid outside {...} captures — using ** inside a capture is also a parse error.
** disables prefix optimization from that point forward: the engine recursively lists all objects under the preceding prefix. Only use it on subtrees with fewer than roughly 100,000 objects—beyond that, listing overhead exceeds several seconds. Prefer explicit time placeholders, such as <yyyy>/<MM>/<dd>, or named captures over ** wherever possible.
A named capture, {name:pattern}, extracts part of the path into an additional field in the previewExternalList result.
| Form | Description | Example |
|---|---|---|
| Captures one path segment. |
|
| Alternation. Matches one of the listed values and captures which one matched. |
|
| Numeric range, zero-padded to the width of the start value. A start value without leading zeros, such as |
|
The capture name must match [_a-zA-Z][_a-zA-Z0-9.]*; for other characters, quote the whole name with backticks, for example {`log-group`:*}.
An alternation or capture without a name, such as {access|error}, matches but doesn't add a field.
Unquoted values in an alternation may only contain letters, digits, _, ., *, and ?. Any other character, such as -, must be wrapped in single quotes. */? inside quotes are literal, not wildcards:
{app:'payment-api'|'auth-service'}
Quoted and unquoted tokens can be mixed within a single value. Use single quotes to protect literal characters and leave wildcards outside the quotes so they remain active:
{region:'us-'????'-'?}
'us-' is a literal prefix, ???? matches exactly four characters, '-' is a literal hyphen, and the trailing ? matches one character — so this matches us-east-1 or us-west-2 but not eu-west-1.
Add an empty alternative ('') to make a value optional:
{compress:'.gz'|'.zst'|''}
This matches .gz, .zst, or nothing. A path segment can't consist only of optional ('') alternatives—it must be guaranteed to match at least one character, so pair it with a literal, such as a fixed .log suffix alongside the optional compression extension.
URI patterns also support a time pattern syntax for matching paths and extracting time information into the path_time field returned by previewExternalList.
| Time pattern | Description |
|---|---|
| Variable-length year (1–4 digits) |
| 2-digit year, interpreted as |
| 4-digit year |
| Variable-length month in year (1–12) |
| 2-digit month in year (01–12) |
| Variable-length day in month (1–31) |
| 2-digit day in month (01–31) |
| Variable-length day in year (1–366) |
| 3-digit day in year (001–366) |
| Variable-length hour of a zero-based 24-hour clock (0–23) |
| 2-digit hour of a zero-based 24-hour clock (00–23) |
| Variable-length minute in hour (0–59) |
| 2-digit minute in hour (00–59) |
| Variable-length second in minute (0–59) |
| 2-digit second in minute (00–59) |
| Variable-length millisecond in second (0–999) |
| 3-digit millisecond in second (000–999) |
A single letter is flexible width. Repeated letters are fixed-width and zero-padded, accepting only the respective number of digits. Multiple tokens can be combined in one placeholder using the separators -, _, ., or :, for example <yyyy-MM-dd>. Fixed-width tokens can also be packed with no separator, such as <yyyyMMdd>, but flexible single-letter tokens can't.
Each calendar field (year, month, day, hour, minute, second, millisecond) may only be supplied once across the whole pattern; supplying it twice, for example both <yyyy> and <yy> in the same pattern, is a parse error. Any field the pattern doesn't include defaults to its minimum value (year → 1970, month → 1, day → 1, all time components → 0).
The timezone specified in the timezone parameter of previewExternalList is used to convert the parsed timestamp.
Example: logs/auth/<yyyy>/<MM>/<dd>/*.log
Where a time or capture segment resolves to concrete values, for example through a query filter, listing narrows down to matching prefixes instead of scanning everything. This narrowing is limited to about 1,000 prefixes—a very wide alternation or numeric range, such as {0..100000}, falls back to listing the coarser prefix.
Also avoid combining several wildcards within a single segment (for example *-*-*-*), since matching is greedy with backtracking and the cost grows with each additional * in that segment.