Selection and modification commands

fields

Keeps only the specified fields.

The fields command also ensures that the order of the fields in the output of the command aligns with the order of the fields specified as parameters of the command. Notably, the fields command creates a new field if a specified field is not present in the result generated by the previous pipeline stage.

Syntax

fields fieldName [= Expression] [, ...]

Basic examples

Example 1: Keep only selected fields and order them

The following query uses the fields command to keep only the fields event, k8s.pod.name, and status in the query result.

data record(event = "failed to find product",
            status = "NONE",
            dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
            k8s.namespace.name = "unguard",
            k8s.pod.name = "unguard-user-simulator-28334535-tqqxt")
| fields event, k8s.pod.name, status

Query result:

Note that the order of the fields in the query result matches the order of the parameters of the fields command.

Example 2: Rename existing fields and add new fields using expressions

The following query uses the fields command to rename the field dt.entity.process_group and create two new fields based on the given expressions. The fields command removes all other fields from the query result.

data record(event = "failed to find product",
            dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
            k8s.namespace.name = "unguard",
            k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",
            status = "NONE")
| fields concat(status, ": ", event),
         process_group = dt.entity.process_group,
         uid = upper(k8s.pod.uid)

Query result:

Note that the expression concat(status, ": ", event) is also used as a field name in the query result since the fields command doesn't define an alias for this field.

Practical example

Example: Fetch selected fields from logs

The following query uses the fetch command to load the data from logs. By default, the query result includes all fields of the fetched data object. The fields command in the query below keeps only the fields listed as its parameters in the query result. In this example, the fields command also converts the loglevel field to lowercase by the DQL lower function and renames it to severity.

fetch logs
| fields timestamp, severity = lower(loglevel), content

fieldsAdd

Evaluates an expression and appends or replaces a field.

Syntax

fieldsAdd fieldName [= Expression] [, ...]

Basic examples

Example 1: Add a field with an expression

The following query adds the new field uid based on the expression upper(k8s.pod.uid).

data record(
  event = "failed to find product",
  k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",
  status = "NONE"
)
| fieldsAdd uid = upper(k8s.pod.uid)

Query result:

Example 2: Add a field conditionally

The following query adds the new field severity based on evaluating the expression with the if function.

data record(
  event = "failed to find product",
  dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
  status = "NONE"
)
| fieldsAdd severity = if(status == "NONE", "INFO", else: status)

Query result:

Example 3: Add a field based on alternative conditions

The following query adds a new field based on an alternative condition. For every alternative condition, you need a separate fieldsAdd command. In the first fieldsAdd command, you don't need to set the else parameter in the if expression. For every subsequent fieldsAdd command, you must set the added field in the else parameter.

data record(type = "event", count = 4, month = 3),
     record(type = "event", count = 3, month = 5),
     record(type = "event", count = 6, month = 9)
| fieldsAdd quarter = if(month >= 1 and month <=3, "Q1")
| fieldsAdd quarter = if(month >= 4 and month <= 6, "Q2", else:quarter)
| fieldsAdd quarter = if(month >= 7 and month <= 9, "Q3", else: quarter)
| fieldsAdd quarter = if(month >= 10 and month <= 12, "Q4", else:quarter)

Query result:

Practical example

Example: Add a field for logs

The following example adds the new field severity based on the expression lower(loglevel).

fetch logs
| fieldsAdd severity = lower(loglevel)

fieldsKeep

Keeps the selected fields. Contrary to the fields command, no new field is created in case it is not present on the record.

Syntax

fieldsKeep fieldNameOrPattern [,...]

Basic examples

Example: Keep fields by field name

The following example keeps the fields event and status in the query result.

data record(
  event = "failed to find product",
  dt.entity.host = "HOST-6DF6DE092963F",
  dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
  k8s.namespace.name = "unguard",
  k8s.pod.name = "unguard-user-simulator-28334535-tqqxt",
  k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",
  status = "NONE"
)
| fieldsKeep event, status

Query result:

Working with string patterns

String patterns are a powerful tool for referencing more than one field. Here's how to quickly reference a list of fields matching a certain string pattern expression:

Example: Keep fields by pattern

The following example keeps the fields where the field name starts with dt.entity. in the query result.

data record(
  event = "failed to find product",
  dt.entity.host = "HOST-6DF6DE092963F",
  dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
  k8s.namespace.name = "unguard",
  k8s.pod.name = "unguard-user-simulator-28334535-tqqxt",
  k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",
  status = "NONE"
)
| fieldsKeep event, "dt.entity.*"

Query result:

Practical example

Example: Keep selected fields for logs

The following example fetches logs and keeps the timestamp, content, and all fields whose name starts with dt.entity.

fetch logs
| fieldsKeep timestamp, content, "dt.entity.*"

fieldsRemove

Removes fields from the result.

Syntax

| fieldsRemove fieldNameOrPattern [,...]

Basic example

Example: Remove fields by name

The following example removes the fields k8s.namespace.name, k8s.pod.name, and k8s.pod.uid from the query result.

data record(
  event = "failed to find product",
  dt.entity.host = "HOST-6DF6DE092963F",
  dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
  k8s.namespace.name = "unguard",
  k8s.pod.name = "unguard-user-simulator-28334535-tqqxt",
  k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",
  status = "NONE"
)
| fieldsRemove k8s.namespace.name, k8s.pod.name, k8s.pod.uid

Query result:

Practical examples

Example: Remove field from logs

The following example uses the fieldsAdd command to generate a new field called severity. The fieldsRemove command removes the original loglevel field because it is duplicate information.

fetch logs
| fieldsAdd severity = lower(loglevel)
| fieldsRemove loglevel

Working with string patterns

String patterns are a powerful tool for referencing more than one field. Here's how to quickly reference a list of fields matching a certain string pattern expression:

Example: Remove log fields by pattern

The following example uses the fieldsAdd command to generate a new field called severity. The fieldsRemove command removes the field loglevel and all fields where the name starts with k8s.cluster..

fetch logs
| fieldsAdd severity = lower(loglevel)
| fieldsRemove loglevel, 'k8s.cluster.*'

fieldsRename

Renames a field.

Syntax

fieldsRename fieldName = originalName, [,...]

Basic example

Example: Rename selected fields

The following example renames the fields dt.entity.host and dt.entity.process_group.

data record(
  event = "failed to find product",
  dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",
  dt.entity.host = "HOST-6DF6DE092963F",
  status = "NONE"
)
| fieldsRename host = dt.entity.host, `Process Group` = dt.entity.process_group

Query result:

Practical example

Example: Rename log fields

The following example queries logs and renames the fields loglevel, log.source, and content.

fetch logs
| fieldsRename severity = loglevel, source = log.source, logmessage = content