Parses a JSON string and extracts the root-level keys of the JSON object to root-level record fields.
jsonExtract input [, conflicts]
| Parameter | Type | Description | Required |
|---|---|---|---|
| string | A string expression containing the JSON object to extract fields from. | Required |
| field identifier | The name of the field where to store colliding fields. If not specified, collisions are discarded. | Optional |
Existing fields in the record are never removed or overwritten. If root-level fields with the same name already exist (that is, if collisions occur), the respective JSON keys can be forwarded into a dedicated field defined by the conflicts parameter.
conflicts parameter, the colliding JSON keys are collected into a dedicated record stored under the specified field name. If no collisions occur, the field contains an empty record.conflicts parameter, the colliding JSON keys are discarded. No additional field is created.If the JSON object contains duplicate keys, only one value is kept. There is no guarantee as to which key/value pair is selected.
The data types of the extracted fields (and array elements or nested fields) correspond to the JSON data type of the respective keys:
| JSON type | DQL type |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
When using the jsonExtract command in DQL queries, all fields in the input stream must be known at query planning time. If this is not the case, you can insert the fields command before the jsonExtract command to specify all the fields that should be retained.
data record(timestamp = now(),content = """{"loggerName":"Log4JGenerator", "threadId":33, "loglevel":"ERROR"}""",loglevel = "WARN",log.source = "/var/log/messages")| jsonExtract content, conflicts: collisions
Query result:
| timestamp | content | loglevel | log.source | loggerName | threadId | collisions |
|---|---|---|---|---|---|---|
|
|
|
|
|
| loglevel: |
The following example extracts JSON fields from the content field of log records fetched from Grail. Since log records don't have a fixed schema, the fields command is used to guarantee that all fields in the input stream to the jsonExtract command are known.
fetch logs| filter startsWith(content, "{")| fields timestamp, content| jsonExtract content
Parses a record field and puts the result(s) into one or more fields as specified in the pattern.
The parse command works in combination with the Dynatrace Pattern Language for parsing strings.
parse expression, pattern [, preserveFieldsOnFailure] [, parsingPrerequisite]
| Parameter | Type | Description | Required |
|---|---|---|---|
expression | string | A field or string expression to parse. | Required |
pattern | The parse pattern. | Required | |
preserveFieldsOnFailure | boolean | Determines if field values should be preserved if parsing fails. When used in OpenPipeline, the value is | Optional |
parsingPrerequisite | boolean | Determines if record should be parsed. | Optional |
The following example parses the content field, which shows the content of a log line.
The parse command adds the parsed fields to the set of fields of the record.
data record(content="117.16.75.9--[14/Mar/2016:23:34:25 +0200] GET//setup.php HTTP/1.1 404 474")| parse content, "IPV4:ip LD HTTPDATE:time ']' LD:text"
Query result:
| content | ip | time | text |
|---|---|---|---|
|
|
|
|
The following example parses the content field preserving existing fields specified in the pattern if parsing fails for the record.
data record(content = "1,alice,192.168.1.1"),record(content = "2,,10.6.24.18", username = "bob"),record(content = "3,mallory,192.168.1.3")| parse content, "( INT:sequence LD:username IPADDR:ip)(fs=',')",preserveFieldsOnFailure: true
Query result:
| content | sequence | username | ip |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
The following example conditionally parses the content field. The parsingPrerequisite parameter of the parse command determines which records to parse.
data record(content = "2016-03-14 23:37:07;www.example.com (192.168.0.1)"),record(content = "2016-03-14 23:37:06;www.example.com")| parse content, "TIMESTAMP ';'LD ( '(' IPADDR:server ')' )",parsingPrerequisite: contains(content,"(")
Query result:
| content | server |
|---|---|
|
|
|
|
In the following example, the parse command extracts all the relevant fields from Apache access logs.
fetch logs| filter dt.entity.process_group == "PROCESS_GROUP-628E1D4CAD1B41B9"| fieldsKeep content| parse content, """(IPADDR:'http.client_ip' | [! \n]+):host' ' ('-' | NSPACE:ident)' ' ('-' | (DATA{1,8096}:auth >>(' [' HTTPDATE)))' ' '[' HTTPDATE:event_time ']'' ' (('\"' [A-Z-_]+:'http.method' ' ' LD{0,8096}:uri ' ' LD{3,10}:'http.flavor' '\"')| DQS:invalid_request)' ' LONG:'http.status_code'' ' (LONG:'http.response.content_length' | '-')(' ' DQS:referer (' ' DQS:user_agent)?)?"""| summarize count = count(), by: { http.status_code }
In case of identical names, fields added by the parse command override the existing fields. When two identical field names are specified in the DQL statement, a warning "The field <fieldName> overrides an existing field." is returned.