Filtering commands
dedup
Removes duplicates from a list of records.
You can use multiple field names or expressions for deduplication. The output of the dedup
command will have the same number of records as there are unique combinations in the given fields or expressions.
The original order of the records is not preserved. Therefore, by default the sequence of records that are chosen during deduplication is random. If you want to pick a particular record out of the duplicates, you can use the sort parameter.
The count of records following the dedup
command will be identical to the count of records following the summarize
command. The summarize
command groups together records with the same values for a given field and aggregates them, whereas dedup
removes duplicate values from a list of records.
Syntax
dedup expr, [expr...], [sort:expr [asc|desc], [....expr [asc|desc]]]
Basic examples
Example 1: Show all locations
In the following example, the input for the dedup
command are records that show the number of bookings and their locations historically. The dedup
command removes duplicate records based on the field location and keeps a random record per unique location.
data record(timestamp = 1, location = "Vienna", bookings = 1254),record(timestamp = 1, location = "London", bookings = 4564),record(timestamp = 2, location = "Vienna", bookings = 1457),record(timestamp = 2, location = "London", bookings = 8741),record(timestamp = 3, location = "Vienna", bookings = 1654)| dedup location
Query result:
1
London
4,564
1
Vienna
1,254
Example 2: Bookings per location
In the following example, the input for the dedup
command are records that show the number of bookings and their locations historically. The dedup
command removes duplicate records based on the field location and for each location keeps the record with the latest timestamp.
data record(timestamp = 1, location = "Vienna", bookings = 1254),record(timestamp = 1, location = "London", bookings = 4564),record(timestamp = 2, location = "Vienna", bookings = 1457),record(timestamp = 2, location = "London", bookings = 8741),record(timestamp = 3, location = "Vienna", bookings = 1654)| dedup location, sort: { timestamp desc }
Query result:
2
London
8,741
3
Vienna
1,654
Example 3: Bookings over time
In the following example, the input for the dedup
command are records that show the number of bookings and their locations historically. The dedup
command removes duplicate records based on the fields timestamp and location and per unique combination keeps the record with the largest value in the bookings field.
data record(timestamp = 1, location = "Vienna", bookings = 1254),record(timestamp = 1, location = "London", bookings = 4553),record(timestamp = 1, location = "London", bookings = 4564),record(timestamp = 2, location = "Vienna", bookings = 1451),record(timestamp = 2, location = "Vienna", bookings = 1457),record(timestamp = 2, location = "London", bookings = 8741)| dedup { timestamp, location }, sort: { bookings desc }
Query result:
1
London
4,564
1
Vienna
1,254
2
London
8,741
2
Vienna
1,457
Practical example
Example: List currently open vulnerabilities
The following example shows the currently open vulnerabilities. Because vulnerabilities are stored as snapshot, the dedup
command is used to retrieve the most recent snapshot of a vulnerability.
fetch events| filter event.provider == "Dynatrace"| filter event.type == "VULNERABILITY_STATE_REPORT_EVENT"| filter event.level== "VULNERABILITY"| filter vulnerability.resolution.status == "OPEN"| filter vulnerability.mute.status != "MUTED"| dedup vulnerability.display_id, sort: { timestamp desc }
filter
Reduces the number of records in a list by keeping only those records that match the specified condition.
Syntax
filter condition [, input]
Basic examples
Example 1: Filter on one condition
The following query uses the filter
command to filter on the field event
.
data record(event = "search failed", prodId = 5, qty = 6, host = "A"),record(event = "product found", prodId = 3, qty = 3, host = "A"),record(event = "search failed", prodId = 4, qty = 7, host = "B"),record(event = "search failed", prodId = 2, qty = 7, host = "C")| filter event == "search failed"
Query result:
search failed
5
6
A
search failed
4
7
B
search failed
2
7
C
Example 2: Filter on multiple conditions
The following query uses the filter
command to filter records based on evaluating multiple conditions.
Logical operators connect the conditions.
data record(event = "search failed", prodId = 5, qty = 6, host = "A"),record(event = "product found", prodId = 3, qty = 3, host = "A"),record(event = "search failed", prodId = 4, qty = 7, host = "B"),record(event = "search failed", prodId = 2, qty = 7, host = "C")| filter in(host, array("B", "C")) or prodId == 3
Query result:
product found
3
3
A
search failed
4
7
B
search failed
2
7
C
Practical example
Example: Filter for critical logs
The following example uses the fetch command to load data from logs.
The filter
command keeps only error records containing the string failed
in the content
field.
fetch logs| filter loglevel == "ERROR" and matchesPhrase(content, "failed")
filterOut
Removes records that match a specific condition.
Syntax
filterOut condition [, input]
Basic examples
Example 1: Filter out by one condition
The following query uses the filterOut
command to remove records based on the field event
.
data record(event = "search failed", prodId = 5, qty = 6, host = "A"),record(event = "product found", prodId = 3, qty = 3, host = "A"),record(event = "search failed", prodId = 4, qty = 7, host = "B"),record(event = "search failed", prodId = 2, qty = 7, host = "C")| filterOut event == "product found"
Query result:
search failed
5
6
A
search failed
4
7
B
search failed
2
7
C
Example 2: Filter out by multiple conditions
The following query uses the filterOut
command to remove records based on evaluating multiple conditions.
Logical operators connect the conditions.
data record(event = "search failed", prodId = 5, qty = 6, host = "A"),record(event = "product found", prodId = 3, qty = 3, host = "A"),record(event = "search failed", prodId = 4, qty = 7, host = "B"),record(event = "search failed", prodId = 2, qty = 7, host = "C")| filterOut host == "A" and qty > 3 or prodId == 4
Query result:
product found
3
3
A
search failed
2
7
C
Practical example
Example: Filter out informational logs
The following example uses the fetch command to load data from logs.
The filterOut
command removes records with a loglevel
value of NONE
or INFO
.
fetch logs| filterOut loglevel == "NONE" or loglevel == "INFO"