Syslog ingestion with ActiveGate

  • 8-min read

ActiveGate version 1.295+ recommended

Syslog, short for system logging protocol, is a logging mechanism that enables system administrators to oversee and control log files from various system components, such as network devices, Linux host syslog, syslog servers, or other syslog producers.

This guide shows you how to configure your Environment ActiveGate on Linux to collect syslog logs in your network and ingest them to Dynatrace.

Prerequisites

  • Environment ActiveGate version 1.295+ on Linux installed to monitor remote technologies.
  • Your network devices have the syslog enabled or you have other syslog producers configured in your network. Refer to RFC 3164 and RFC 5424 for details. Dynatrace supports a wide variety of syslog implementations, including RSysLog, Syslog-NG, NXLog, and others.
  • By default, the ingested syslog must be in the format defined by RFC 3164 and RFC 5424. If your devices produce non-standard syslog format, you need to transform it to the supported format using Dynatrace OpenPipeline processing. RFC 3164 requires receiver configuration. For details, see Edit the syslog receiver configuration below.

Hardware requirements

Syslog ingestion is performed by an ActiveGate. The syslog ingestion throughput depends on the hardware your ActiveGate is deployed on.

CPUsRAM (GB)Maximum throughput
416~1TB/day
832~2.7TB/day

Who is it for?

This guide is intended for network and Dynatrace admins who are tasked to enable the syslog log ingestion into Dynatrace.

Enable syslog ingestion

Enabling syslog log ingestion requires you to:

  • Deploy Environment ActiveGate in a place ensuring the connectivity between ActiveGate and monitored devices.
  • Enable syslog ingestion on ActiveGate.
  • optional in some cases, you'll need to adapt the default syslog receiver configuration.
  1. Deploy Environment ActiveGate.

    See instructions for Linux. Use the remote technologies monitoring purpose.

  2. Enable syslog ingestion on your ActiveGate.

    Edit the /var/lib/dynatrace/remotepluginmodule/agent/conf/extensionsuser.conf file and add the following flag:

    syslogenabled=true
  3. optional Edit the syslog receiver configuration.

    ActiveGate uses an embedded Dynatrace OpenTelemetry Collector instance and stores the receiver configuration in the /var/lib/dynatrace/remotepluginmodule/agent/conf/syslog.yaml file. The Collector is installed by default.

    Use this configuration only for syslog ingestion.

    If your syslog producers use the default ports per supported protocols, your syslog-enabled ActiveGate should receive syslog records right away.

    You need to modify the configuration if your syslog producers cast events on custom ports. Also, the receiver configuration needs to be updated if the syslog protocol is RFC 3164 (update the protocol attribute).

receivers:
syslog/udp:
udp:
listen_address: '0.0.0.0:514'
add_attributes: true
protocol: rfc5424
operators:
- type: syslog_parser
protocol: rfc5424
syslog/tcp:
tcp:
listen_address: '0.0.0.0:601'
add_attributes: true
protocol: rfc5424
operators:
- type: syslog_parser
protocol: rfc5424
# syslog/tcp_tls:
# tcp:
# listen_address: "0.0.0.0:6514"
# tls:
# cert_file: "/absolute/path/to/server.crt"
# key_file: "/absolute/path/to/server.key"
# protocol: rfc5424
# operators:
# - type: syslog_parser
# protocol: rfc5424
#DO.NOT.MODIFY
exporters:
otlphttp/syslog: ${file:syslogendpoint.yaml}
processors:
batch:
send_batch_size: 512
send_batch_max_size: 1024
transform:
log_statements:
- context: log
statements:
- set(body, attributes["message"])
attributes:
actions:
- key: net.host.name
action: delete
- key: net.peer.name
action: delete
- key: net.peer.port
action: delete
- key: net.transport
action: delete
- key: net.host.ip
action: delete
- key: dt.ingest.port
from_attribute: net.host.port
action: upsert
- key: dt.ingest.source.ip
from_attribute: net.peer.ip
action: upsert
- key: net.peer.ip
action: delete
- key: net.host.port
action: delete
- key: syslog.hostname
from_attribute: hostname
action: upsert
- key: hostname
action: delete
- key: syslog.facility
from_attribute: facility
action: upsert
- key: facility
action: delete
- key: syslog.priority
from_attribute: priority
action: upsert
- key: priority
action: delete
- key: syslog.proc_id
from_attribute: proc_id
action: upsert
- key: proc_id
action: delete
- key: syslog.version
from_attribute: version
action: upsert
- key: version
action: delete
- key: syslog.appname
from_attribute: appname
action: upsert
- key: appname
action: delete
- key: message
action: delete
service:
telemetry:
metrics:
level: none
pipelines:
logs/udp:
receivers: [syslog/udp]
processors: [transform, attributes, batch]
exporters: [otlphttp/syslog]
logs/tcp:
receivers: [syslog/tcp]
processors: [transform, attributes, batch]
exporters: [otlphttp/syslog]
# logs/tcp_tls:
# receivers: [syslog/tcp_tls]
# processors: [transform, attributes, batch]
# exporters: [otlphttp/syslog]

Note: Do not modify the exporter configuration. It's preconfigured to forward your syslogs to the Dynatrace Environment.

For more information on syslog receiver configuration, see Ingest syslog data using OpenTelemetry Collector.

  1. Verify the syslog ingestion is enabled.

    After you enable syslog ingestion, check the following log files to verify it:

    Open the newest ruxit_extensionmodule_*.log log file in the extensions log directory:

    • Linux: /var/lib/dynatrace/remotepluginmodule/log/extensions

    It should contain the following line:

    Otel syslog enabled: true
  2. Enable syslog on the devices you want to monitor.

    The way you enable syslog depends on the device and its platform, refer to specific documentation for details.

    Example Configure Rsyslog on Linux Ubuntu to forward syslog logs to a remote server.

    Add the following line to the syslog daemon configuration file (/etc/rsyslog.conf)

    • UDP
      *.* @<ActiveGate host IP>:514
    • TCP
      *.* @@<ActiveGate host IP>:601

    The *.* instructs the daemon to forward all messages to the specified ActiveGate listening on the provided port and IP address. <ActiveGate host IP> needs to point to the IP address of a syslog-enabled ActiveGate.

    For more examples, see Syslog via OpenTelemetry Collector

  3. Verify ActiveGate receives the syslog events.

    After your syslog producers start to cast log records, open the latest dynatracesourceotelcollector.*.log file in /var/lib/dynatrace/remotepluginmodule/log/extensions/datasources/otelSyslog.

    If ActiveGate receives the log records you should see entries as in the example below:

    [otelSyslog][otelSyslog][37448][err]LogRecord #3
    [otelSyslog][oteiSyslog][37448][err]ObservedTimestamp: 2024-05-06 @9:52:10.6748723 +8000 UTC
    [otelSyslog][otelSyslog][37448][err]Timestamp: 2624-05-@6 11:52:16 +90e0 UTC
    [otelSyslog][otelsyslog][37448][err]SeverityText: info
    [otelSyslog][otelSyslog][37443][err]SeverityNumber: Info(9)
    [otelSyslog][otelSyslog][37448][err]Body: Str(<30>May 6 11:52:10 SOME-HOST systemd[1]: Finished Load Kernel Module fuse.)
    [otelSyslog][otelSyslog][37448][err]Attributes:
    [otelSyslog][otelSyslog][37448][err] -> priority: Int(3)
    [otelSyslog][otelSyslog][37448][err] -> facility: Int(3)
    [otelSyslog][otelSyslog][37448][err] -> appname: Str(systemd)
    [otelSyslog][otelSyslog][37448][err] -> proc_id: Str(1)
    [otelSyslog][otelSyslog][37443][err] -> log: Map({“source": “syslog"})
    [otelSyslog][otelSyslog][37443][err] -> hostname: Str(SOME-HOST)
    [otelSyslog][otelSyslog][37443][err] -> message: Str(Finished Load Kernel Module fuse.)
    [otelSyslog][otelSyslog][37448][err]Trace ID:
    [otelSyslog][otelSyslog][37448][err]Span ID:
    [otelSyslog][otelSyslog][37443][err]Flags: 0

    For more information on troubleshooting the syslog receiver, see Collector troubleshooting.

  4. You've arrived! Now, your syslog-ingested events are enriched with the host-specific attributes and become available in Grail for Davis® AI-based data analysis, log processing, or querying via DQL.

Mask sensitive data

ActiveGate syslog ingestion supports the OpenTelemetry Transform Processor and OpenTelemetry Transformation Language (OTTL) to process your syslog data at edge, before sensitive data leaves your network.

This way, you can use it to mask or hash sensitive data in your syslog lines, so that no sensitive information is ingested into Dynatrace.

Let's assume your credit card data is visible in syslog as:

<14>2 2024-07-19T14:53:55Z example-host 0OOButHPbR 1234 - - New operation for CreditCard 1234567891011124

To mask a credit card number, add the following configuration under the processors node of the syslog.yaml file:

processors:
transform/redact_credict_cart:
log_statements:
- context: log
statements:
- replace_pattern(body, "\\d{15,16}", "REDACTED")

The replace_pattern function replaces the credit card number with the REDACTED string. The credit card number in the content is matched by the body, "\\d{15,16}" pattern.

Add custom attributes

You can also modify the default configuration if you want to group a set of various devices by configuring them to use a specific port. For example, using very generic log messages, you can enrich your syslog events cast on specific TCP ports with custom attributes using the configuration as in the example below.

receivers:
syslog/f5:
tcp:
listen_address: "0.0.0.0:54526"
protocol: rfc5424
operators:
- type: add
field: attributes.log.source
value: syslog
- type: add
field: attributes.dt.ip_addresses
value: "1xx.xx.xx.xx1"
- type: add
field: attributes.instance.name
value: "ip-1xx-xx-x-xx9.ec2.internal"
- type: add
field: attributes.device.type
value: "f5bigip"
syslog/host:
tcp:
listen_address: "0.0.0.0:54527"
protocol: rfc5424
operators:
- type: add
field: attributes.log.source
value: syslog
- type: add
field: attributes.device.type
value: "ubuntu-syslog"

You can also use:

  • delete to exclude specific attributes from ingestion.
  • upsert to insert a new attribute to your log line where the key does not already exist, or to update an attribute where the key does exist.

For example, if we can read the net.peer.port attribute, its value is used for custom.remote.port. Otherwise, the custom.report.port isn't set.

attributes:
actions:
- key: custom.remote.port
from_attribute: net.peer.port
action: upsert

For more information on attributes configuration, see Attributes Processor.

Filter data

You can filter the syslog data to drop irrelevant log lines and reduce your consumption at edge, before the data leaves your network.

For example, assume we want to ignore log lines categorized with syslog facility 21,

<21> 2024-07-19T14:53:55Z example-host 0OOButHPbR 1234 - - Spam mail

Add the following filter to the syslog.yaml file.

filter/mail:
logs:
log_record:
- attributes["syslog.facility"] == 21

The log line isn't ingested based on the 21 syslog facility.

Process logs with technology bundle parsers

Through OpenPipeline, you can use and configure technology bundles. A technology bundle is a library of parsers (processing rules), that process logs from various technologies such as Java, .NET, Microsoft IIS, etc.

Parsers help you to improve filtering, troubleshooting, metrics, alerts, and dashboards by efficiently extracting log levels and relevant attributes. You can also use technology bundles to structure logs from technologies that are not supported by Dynatrace out of the box.

Technology bundle parsers

Related tags
Log Analytics