Selection and modification commands
fields
Keeps only the specified fields.
The fields
command also ensures that the order of the fields in the output of the command aligns with the order of the fields specified as parameters of the command. Notably, the fields
command creates a new field if a specified field is not present in the result generated by the previous pipeline stage.
Syntax
fields fieldName [= Expression] [, ...]
Basic examples
Example 1: Keep only selected fields and order them
The following query uses the fields
command to keep only the fields event
, k8s.pod.name
, and status
in the query result.
data record(event = "failed to find product",status = "NONE",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",k8s.namespace.name = "unguard",k8s.pod.name = "unguard-user-simulator-28334535-tqqxt")| fields event, k8s.pod.name, status
Query result:
failed to find product
unguard-user-simulator-28334535-tqqxt
NONE
Note that the order of the fields in the query result matches the order of the parameters of the fields
command.
Example 2: Rename existing fields and add new fields using expressions
The following query uses the fields
command to rename the field dt.entity.process_group
and create two new fields based on the given expressions. The fields
command removes all other fields from the query result.
data record(event = "failed to find product",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",k8s.namespace.name = "unguard",k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",status = "NONE")| fields concat(status, ": ", event),process_group = dt.entity.process_group,uid = upper(k8s.pod.uid)
Query result:
NONE: failed to find product
PROCESS_GROUP-01A48F48FA09FE4A
D3E93267-E7F9-4A2D-B5DF-DC2AC226A7CE
Note that the expression concat(status, ": ", event)
is also used as a field name in the query result since the fields
command doesn't define an alias for this field.
Practical example
Example: Fetch selected fields from logs
The following query uses the fetch command to load the data from logs
. By default, the query result includes all fields of the fetched data object. The fields
command in the query below keeps only the fields listed as its parameters in the query result. In this example, the fields
command also converts the loglevel
field to lowercase by the DQL lower function and renames it to severity
.
fetch logs| fields timestamp, severity = lower(loglevel), content
fieldsAdd
Evaluates an expression and appends or replaces a field.
Syntax
fieldsAdd fieldName [= Expression] [, ...]
Basic examples
Example 1: Add a field with an expression
The following query adds the new field uid
based on the expression upper(k8s.pod.uid)
.
data record(event = "failed to find product",k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",status = "NONE")| fieldsAdd uid = upper(k8s.pod.uid)
Query result:
failed to find product
d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce
NONE
D3E93267-E7F9-4A2D-B5DF-DC2AC226A7CE
Example 2: Add a field conditionally
The following query adds the new field severity
based on evaluating the expression with the if function.
data record(event = "failed to find product",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",status = "NONE")| fieldsAdd severity = if(status == "NONE", "INFO", else: status)
Query result:
failed to find product
PROCESS_GROUP-01A48F48FA09FE4A
NONE
INFO
Example 3: Add a field based on alternative conditions
The following query adds a new field based on an alternative condition.
For every alternative condition, you need a separate fieldsAdd
command.
In the first fieldsAdd
command, you don't need to set the else parameter in the if expression.
For every subsequent fieldsAdd
command, you must set the added field in the else parameter.
data record(type = "event", count = 4, month = 3),record(type = "event", count = 3, month = 5),record(type = "event", count = 6, month = 9)| fieldsAdd quarter = if(month >= 1 and month <=3, "Q1")| fieldsAdd quarter = if(month >= 4 and month <= 6, "Q2", else:quarter)| fieldsAdd quarter = if(month >= 7 and month <= 9, "Q3", else: quarter)| fieldsAdd quarter = if(month >= 10 and month <= 12, "Q4", else:quarter)
Query result:
event
4
3
Q1
event
3
5
Q2
event
6
9
Q3
Practical example
Example: Add a field for logs
The following example adds the new field severity
based on the expression lower(loglevel)
.
fetch logs| fieldsAdd severity = lower(loglevel)
In case of identical field names, added fields override the existing fields in the processing stream. When two identical field names are specified in the DQL statement, a warning "The field <fieldName>
overrides an existing field." is returned.
fieldsKeep
Keeps the selected fields. Contrary to the fields
command, no new field is created in case it is not present on the record.
Syntax
fieldsKeep fieldNameOrPattern [,...]
Basic examples
Example: Keep fields by field name
The following example keeps the fields event
and status
in the query result.
data record(event = "failed to find product",dt.entity.host = "HOST-6DF6DE092963F",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",k8s.namespace.name = "unguard",k8s.pod.name = "unguard-user-simulator-28334535-tqqxt",k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",status = "NONE")| fieldsKeep event, status
Query result:
failed to find product
NONE
Working with string patterns
String patterns are a powerful tool for referencing more than one field. Here's how to quickly reference a list of fields matching a certain string pattern expression:
Example: Keep fields by pattern
The following example keeps the fields where the field name starts with dt.entity.
in the query result.
data record(event = "failed to find product",dt.entity.host = "HOST-6DF6DE092963F",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",k8s.namespace.name = "unguard",k8s.pod.name = "unguard-user-simulator-28334535-tqqxt",k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",status = "NONE")| fieldsKeep event, "dt.entity.*"
Query result:
failed to find product
HOST-6DF6DE092963F
PROCESS_GROUP-01A48F48FA09FE4A
Practical example
Example: Keep selected fields for logs
The following example fetches logs and keeps the timestamp
, content
, and all fields whose name starts with dt.entity
.
fetch logs| fieldsKeep timestamp, content, "dt.entity.*"
fieldsRemove
Removes fields from the result.
Syntax
| fieldsRemove fieldNameOrPattern [,...]
Basic example
Example: Remove fields by name
The following example removes the fields k8s.namespace.name
, k8s.pod.name
, and k8s.pod.uid
from the query result.
data record(event = "failed to find product",dt.entity.host = "HOST-6DF6DE092963F",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",k8s.namespace.name = "unguard",k8s.pod.name = "unguard-user-simulator-28334535-tqqxt",k8s.pod.uid = "d3e93267-e7f9-4a2d-b5df-dc2ac226a7ce",status = "NONE")| fieldsRemove k8s.namespace.name, k8s.pod.name, k8s.pod.uid
Query result:
failed to find product
HOST-6DF6DE092963F
PROCESS_GROUP-01A48F48FA09FE4A
NONE
Practical examples
Example: Remove field from logs
The following example uses the fieldsAdd
command to generate a new field called severity
. The fieldsRemove
command removes the original loglevel
field because it is duplicate information.
fetch logs| fieldsAdd severity = lower(loglevel)| fieldsRemove loglevel
Working with string patterns
String patterns are a powerful tool for referencing more than one field. Here's how to quickly reference a list of fields matching a certain string pattern expression:
Example: Remove log fields by pattern
The following example uses the fieldsAdd
command to generate a new field called severity
.
The fieldsRemove
command removes the field loglevel
and all fields where the name starts with k8s.cluster.
.
fetch logs| fieldsAdd severity = lower(loglevel)| fieldsRemove loglevel, 'k8s.cluster.*'
fieldsRename
Renames a field.
Syntax
fieldsRename fieldName = originalName, [,...]
Basic example
Example: Rename selected fields
The following example renames the fields dt.entity.host
and dt.entity.process_group
.
data record(event = "failed to find product",dt.entity.process_group = "PROCESS_GROUP-01A48F48FA09FE4A",dt.entity.host = "HOST-6DF6DE092963F",status = "NONE")| fieldsRename host = dt.entity.host, `Process Group` = dt.entity.process_group
Query result:
failed to find product
PROCESS_GROUP-01A48F48FA09FE4A
HOST-6DF6DE092963F
NONE
Practical example
Example: Rename log fields
The following example queries logs and renames the fields loglevel
, log.source
, and content
.
fetch logs| fieldsRename severity = loglevel, source = log.source, logmessage = content
In case of identical field names, renamed fields override the existing fields in the processing stream. If two identical field names are specified in the DQL statement, the warning "The field <fieldName>
overrides an existing field." is returned.