JSON log pre-processing detects escape characters in JSON strings and converts them into structured JSON objects for further processing and deeper analysis. You can then query the unescaped JSON field using the jsonField and jsonPath DQL functions for precise extraction and filtering log attributes.
JSON log pre-processing benefits:
Many log forwarders wrap the original log message as JSON strings within the content field with escape characters.
JSON log pre-processing performs the following steps.
Detects and unescapes escape characters in the JSON string.
Converts the JSON strings into structured JSON objects. The conversion happens during log pre-processing, making results available for further processing in custom pipelines.
{"content": {"loglevel": "ERROR","event": "{\\\"type\\\":\\\"db_error\\\",\\\"code\\\":\\\"CONN_FAIL\\\"}"},"source": "fluentbit","host.name": "app-server-01"}
{"content": {"loglevel": "ERROR","event": {"type": "db_error","code": "CONN_FAIL"}},"source": "fluentbit","host.name": "app-server-01"}
You can query the unescaped JSON field for precise extraction and filtering log attributes using the following DQL functions.
jsonField function for extracting the value by its actual name.
This is an example of extracting loglevel using jsonField.
fetch logs| fieldsAdd logLevel = jsonField(content, "loglevel")| filter logLevel == "ERROR"
jsonPath function for extracting value by a JSONPath expression.
This is an example of extracting eventType using jsonPath.
fetch logs| fieldsAdd eventType = jsonPath(content, "$.event.type")| filter eventType == "db_error"
Unescaping—for example, removing a forward slash—is skipped when the JSON is invalid. The original content stays.