Log processing examples

Depending on the rules that you create, you can customize incoming log data according to your needs. See below for example data processing scenarios.

Example 1: Fix unrecognized timestamp and loglevel

You can fix unrecognized timestamp and loglevel visible in the log viewer based on matched log source. For this example, let's assume that you see a stored event in the log viewer from the log.source application that is set to /var/log/myapp/application.log.#

You notice a couple of things you want to fix:

So you want to transform your log data to contain the proper values in the timestamp and logLevel fields, and you want to add a new thread.name attribute containing a properly extracted value.

To create a processing rule

As the new log data is ingested, you'll be able to see processed log data in the log viewer.

Example 2: Create billed duration metric for AWS service using log data

In this example, you want to monitor the actual billed duration from your AWS services. You want to use the cloud.provider attribute with the aws value in your log data. In the log viewer, you see a log record containing the following line:

REPORT RequestId: 000d000-0e00-0d0b-a00e-aec0aa0000bc Duration: 5033.50 ms Billed Duration: 5034 ms Memory Size: 1024 MB Max Memory Used: 80 MB Init Duration: 488.08 ms

Additionally, that log record contains the cloud.provider attribute with the aws value.

Example 3: Parse out specific fields from JSON content

In this example, you see a log line that has the following JSON structure:

{"intField": 13, "stringField": "someValue", "nested": {"nestedStringField1": "someNestedValue1", "nestedStringField2": "someNestedValue2"} }

The sample log would look like this:

{
  "content": "{\"intField\": 13, \"stringField\": \"someValue\", \"nested\": {\"nestedStringField1\": \"someNestedValue1\", \"nestedStringField2\": \"someNestedValue2\"} }"
}

Example 4: Parse out attributes from different formats

You can parse out attributes from different formats within a single pattern expression.

In this example, one or more applications is logging a user identifier that you want to extract as a standalone log attribute. The log format is not consistent because it includes various schemes to log the user ID:

With the optional modifier (question ?) and Alternative Groups, you can cover all such cases with a single pattern expression:

PARSE(content, "
  LD //matches any text within a single line
  ('user'| 'User') //user or User literal
  SPACE? //optional space
  ('id'|'Id'|'ID') //matches any of these
  SPACE? //optional space
  PUNCT? //optional punctuation
  SPACE? //optional space
  INT:my.user.id
")

Using such a rule, you can parse out the user identifier from many different notations. For example:

03/22 08:52:51 INFO user ID=1234567 Call = 0319 Result = 0
03/22 08:52:51 INFO UserId = 1234567 Call = 0319 Result = 0
03/22 08:52:51 INFO user id=1234567 Call = 0319 Result = 0
03/22 08:52:51 INFO user ID:1234567 Call = 0319 Result = 0
03/22 08:52:51 INFO User ID: 1234567 Call = 0319 Result = 0
03/22 08:52:51 INFO userid: 1234567 Call = 0319 Result = 0

Example 5: Multiple PARSE commands within a single processing rule

You can handle various formats or perform additional parsing on already parsed-out attributes with multiple PARSE commands connected with pipes (|).

For example, with the following log:

{
  "content": "{\"intField\": 13, \"message\": \"Error occurred for user 12345: Missing permissions\", \"nested\": {\"nestedStringField1\": \"someNestedValue1\", \"nestedStringField2\": \"someNestedValue2\"} }"
}

First, you can parse out the message field, the user ID, and the error message.

PARSE(content, "JSON{STRING:message}(flat=true)") | PARSE(message, "LD 'user ' INT:user.id ': ' LD:error.message")

The result is:

{
  "content": "{\"intField\": 13, \"message\": \"Error occurred for user 12345: Missing permissions\", \"nested\": {\"nestedStringField1\": \"someNestedValue1\", \"nestedStringField2\": \"someNestedValue2\"} }",
  "message": "Error occurred for user 12345: Missing permissions",
  "user.id": "12345",
  "error.message": "Missing permissions"
}

Example 6: Use specialized matchers

We provide a comprehensive list of matchers that ease pattern building.

For example, you can parse the following sample log event:

{
  "content":"2022-05-11T13:23:45Z INFO 192.168.33.1 \"GET /api/v2/logs/ingest HTTP/1.0\" 200"
}

using the specialized matchers:

PARSE(content, "ISO8601:timestamp SPACE UPPER:loglevel SPACE IPADDR:ip SPACE DQS:request SPACE INTEGER:code")

and the result is:

{
  "content": "2022-05-11T13:23:45Z INFO 192.168.33.1 \"GET /api/v2/logs/ingest HTTP/1.0\" 200",
  "timestamp": "1652275425000",
  "loglevel": "INFO",
  "ip": "192.168.33.1",
  "request": "GET /api/v2/logs/ingest HTTP/1.0",
  "code": "200"
}

Example 7: Manipulate any attribute from log (not only content)

Unless specified otherwise, the processing rule works only on the read-only content field. For it to work on different log event attributes, you need to use the USING command.

For example, the following rule declares two input attributes: writable status and read-only content. Next, it checks whether the status is WARN and the content contains the text error. If both conditions are true, the rule overwrites status with the value ERROR.

Processing rule definition:

USING(INOUT status:STRING, content)
| FIELDS_ADD(status:IF_THEN(status == 'WARN' AND content CONTAINS('error'), "ERROR"))

Log data sample:

{
  "log.source": "using",
  "timestamp": "1656011002196",
  "status": "WARN",
  "content":"Some error message"
}

Result after transformation:

{
  "log.source": "using",
  "timestamp": "1656011002196",
  "status": "ERROR",
  "content":"Some error message"
}

Example 8: Add a new attribute to the current log event structure

The FIELDS_ADD command can be used to introduce additional top-level log attributes. The following script adds two attributes: the first one stores the length and the second one stores the number of words that are in the content field.

Processing rule definition:

FIELDS_ADD(content.length: STRLEN(content), content.words: ARRAY_COUNT(SPLIT(content,"' '")))

Log data sample:

{
  "log.source": "new_attributes",
  "timestamp": "1656010654603",
  "content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis."
}

Result after transformation:

{
  "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis.",
  "timestamp": "1656010654603",
  "log.source": "new_attribute",
  "content.length": "62",
  "content.words": "9"
}

Example 9: Basic math on attributes

With all of the available functions and operators, it's easy to perform calculations.

In the following example, we parse the values of total and failed, calculate the percentage that failed, and concatenate the value with the percentage sign. Then we store it in a new attribute called failed.percentage and remove the temporary fields.

Processing rule definition:

PARSE(content,"LD 'total: ' INT:total '; failed: ' INT:failed")
| FIELDS_ADD(failed.percentage: 100.0 * failed / total + '%')
| FIELDS_REMOVE(total, failed)

Log data sample:

{
  "timestamp": "1656011338522",
  "content":"Lorem ipsum total: 1000; failed: 250"
}

Result after transformation:

{
  "content": "Lorem ipsum total: 1000; failed: 250",
  "timestamp": "1656011338522",
  "failed.percentage": "25.0%"
}

Example 10: Drop a specific attribute from log message

To drop an event attribute that is a part of the original record, we first need to declare it as a writable (INOUT option) input field with the USING command and then explicitly remove it with the FIELDS_REMOVE command so that it is not present in the output of the transformation.

In the following example, we declare redundant.attribute as an obligatory writeable attribute of STRING type and then we remove it.

Processing rule definition:

USING(INOUT redundant.attribute:STRING)
| FIELDS_REMOVE(redundant.attribute)

Log data sample:

{
  "redundant.attribute": "value",
  "timestamp": "1656011525708",
  "content":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla ac neque nisi. Nunc accumsan sollicitudin lacus."
}

Result after transformation:

{
  "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla ac neque nisi. Nunc accumsan sollicitudin lacus.",
  "timestamp": "1656011525708"
}

In this case, the definition would look like this:

USING(INOUT redundant.attribute:STRING?)
| FIELDS_REMOVE(redundant.attribute)

Example 11: Drop the whole log event

The whole log event can be dropped with a FILTER_OUT command. The event is dropped when the condition passed as the command parameter is fulfilled.

Example 12: Mask any attribute

Whenever the content or any other attribute is to be changed, it has to be declared as INOUT (writable) with the USING command. The REPLACE_PATTERN is a very powerful function that can be useful when we want to mask some part of the attribute.

Example 13: Rename attributes

With the FIELDS_RENAME command, we can rename attributes that were a part of an original log event and attributes created within the processor. Whenever we want to change any attribute from the original event, we need to declare it as INOUT (writeable).

In the following example, we rename an existing attribute. Furthermore, we parse out the field from JSON in flat mode and rename the new attribute that has been created automatically with the JSON field name.

Processing rule definition:

USING(INOUT to_be_renamed, content)
| FIELDS_RENAME(better_name: to_be_renamed)
| PARSE(content,"JSON{STRING:json_field_to_be_renamed}(flat=true)")
| FIELDS_RENAME(another_better_name: json_field_to_be_renamed)

Log data sample:

{
  "timestamp": "1656061626073",
  "content":"{\"json_field_to_be_renamed\": \"dolor sit amet\", \"field2\": \"consectetur adipiscing elit\"}",
  "to_be_renamed": "Lorem ipsum"
}

Result after transformation:

{
  "content": "{\"json_field_to_be_renamed\": \"dolor sit amet\", \"field2\": \"consectetur adipiscing elit\"}",
  "timestamp": "1656061626073",
  "better_name": "Lorem ipsum",
  "another_better_name": "dolor sit amet"
}

Example 14: Input field data types

The script in a processor definition operates with strongly typed data: the functions and operators accept only declared types of data. The type is assigned to all input fields defined in the USING command as well as to variables created while parsing or using casting functions.

Processing rule definition:

USING(number:INTEGER, avg:DOUBLE, addr:IPADDR, arr:INTEGER[],bool:BOOLEAN, ts:TIMESTAMP)
| FIELDS_ADD(multi:number*10)
| FIELDS_ADD(avgPlus1:avg+1)
| FIELDS_ADD(isIP: IS_IPV6(addr))
| FIELDS_ADD(arrAvg: ARRAY_AVG(arr))
| FIELDS_ADD(negation: NOT(bool))
| FIELDS_ADD(tsAddYear: TIME_ADD_YEAR(ts,1))

Log data sample:

{
  "content":"Lorem ipsum",
  "number":"5",
  "avg":"123.5",
  "addr":"2a00:1450:4010:c05::69",
  "arr": ["1","2"],
  "bool":"false",
  "ts":"1984-11-30 22:19:59.789 +0000"
}

Result after transformation:

{
  "content": "Lorem ipsum",
  "number": "5",
  "avg": "123.5",
  "addr": "2a00:1450:4010:c05::69",
  "arr": [
    "1",
    "2"
  ],
  "bool": "false",
  "ts": "1984-11-30 22:19:59.789 +0000",
  "tsAddYear": "1985-11-30T22:19:59.789000000 +0000",
  "negation": "true",
  "arrAvg": "1.5",
  "isIP": "true",
  "avgPlus1": "124.5",
  "multi": "50"
}