Stream logs to Dynatrace with Fluent Bit

You can send logs to Dynatrace using Fluent Bit. Configure Fluent Bit to send logs to Dynatrace generic ingest API.

Capabilities

  • Fluent Bit is a multiplatform log processor and forwarder that allows you to collect data/logs from different sources, unify and send them to multiple destinations. It is compatible with the Docker and Kubernetes environments.
  • Dynatrace can be configured as the target log management and analytics environment for your data thanks to Fluent Bit's configurable http output.
  • You can use any of Fluent Bit input plugins to get logs and events from your application to Dynatrace.

Configuration

The Fluent Bit http output plugin allows you to forward your logs to the Dynatrace generic logs ingest endpoint.

  1. Get a Dynatrace API token with the logs.ingest (Ingest Logs) scope.
  2. Deploy Fluent Bit.
  3. To send logs into the Dynatrace generic logs ingest endpoint, configure the http output plugin through the configuration file.
  4. In your main Fluent Bit configuration file, append the Output section with the following parameters:
[OUTPUT]
    name  http
    match *
    header Content-Type application/json; charset=utf-8
    header Authorization Api-Token {your-API-token-here}
    allow_duplicated_headers false
    host  {your-environment-id}.live.dynatrace.com
    Port  443
    URI   /api/v2/logs/ingest
    Format json
    json_date_format iso8601
    json_date_key timestamp
    tls On
    tls.verify On

You can place your API token in the header or as GET variable in URI (see example below).

  • For Dynatrace SaaS, the generic logs ingest endpoint is available in your environment.
  • For Dynatrace Managed or if Environment ActiveGate is your choice for an endpoint in local environment, install ActiveGate instance: In Dynatrace Hub, select ActiveGate > Set up.
  • Generic log ingest API v2 is automatically enabled on ActiveGate.

Example

Fluent Bit is the recommended solution whenever reducing resource consumption is critical. In the below example, you need to ingest your AWS Fargate logs. FireLens enables you to set up Fluent Bit for this ingestion.

Ingest AWS Fargate logs with Fluent Bit

When creating a new task definition using the AWS Management Console, the FireLens integration section makes it easy to add a log router container. Follow the steps below to execute your ingest:

  1. In the AWS Management Console, go to the Firelens integration section.

Final log integration page in AWS Management Console

  1. Pick the built-in Fluent Bit image.
  1. Edit the container in which your app-generating logs are running.
  1. In the Storage and Logging section, select awsfirelens as the log driver.

Set AWS Firelens as a log driver

The settings for the log driver should point to the log ingest API of your SaaS tenant. You need to provide two headers for Fluent Bit: content type and authorization token. As FireLens supports only one header, you can pass the token as part of the URL. Your configuration for AWS FireLens should have the following:

Name: http
TLS: on
Format: json
Header: Content-Type application/json; charset=utf-8
Host: {your-environment-id}.live.dynatrace.com
Port: 443
URI: /api/v2/logs/ingest?api-token={your-API-token-here}
Allow_Duplicated_Headers": "false"
Json_Date_Format": "iso8601"
Json_Date_Key": "timestamp"

To avoid publishing the token in plaintext, follow the directions from AWS Secrets Manager. Once your application starts publishing logs, you can view them in the Dynatrace UI.

Refer to AWS sample repository for the task definition JSON with Dynatrace configuration.

For more configuration details, see Amazon ECS Developer Guide.