Generic log ingestion

Dynatrace automatically collects log and event data from a vast array of technologies. With generic log ingestion, you can stream log records to a system and have Dynatrace transform the stream into meaningful log messages.

LMA - Generic log ingestion API

Generic log ingestion allows you to stream log records to the system. It is available via Log Monitoring API - POST ingest logs for JSON and text format or the OTLP endpoint for OTLP binary protobuf format.

  • For Dynatrace SaaS, the generic logs ingest endpoint is available in your environment.

  • If the Environment ActiveGate is your choice for an endpoint in your local environment, install an ActiveGate instance: In Dynatrace Hub, select ActiveGate > Set up. The generic log ingest API v2 is automatically enabled on ActiveGate.

  • The endpoint is enabled by default on all of your ActiveGates.

  • ActiveGate is responsible for serving the endpoint, collecting the data, and forwarding it to Dynatrace in batches.

  • Saas endpoints:

    • https://{your-environment-id}.live.dynatrace.com/api/v2/logs/ingest
    • https://{your-environment-id}.live.dynatrace.com/api/v2/otlp/v1/logs

  • Environment ActiveGate endpoints:

    • https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/logs/ingest
    • https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/otlp/v1/logs

ActiveGate will collect and attempt to automatically transform any log data containing the following JSON elements:

  • Log content
  • Timestamp
  • Key-Values attributes

When using log processing with the custom processing pipeline (OpenPipeline), ingest supports all JSON data types for attribute values. This requires SaaS version 1.295+ when using the SaaS API endpoint or ActiveGate version 1.295+ when using the ActiveGate API endpoint. In all other cases, all ingested values are converted to the string type.

To view all predefined key-value attributes, such as the supported semantic attribute keys, check Log Monitoring API - POST ingest logs.

Log data queue

You can customize the log data queue properties by editing the custom.properties file (see Configuration properties and parameters of ActiveGate) on your ActiveGate to set the following values:

[generic_ingest]
#disk_queue_path=<custom_path> # defaults to temp folder
#disk_queue_max_size_mb=<limit> # defaults to 300 MB
503 Usable space limit reached

The log data ingestion API returns a 503 Usable space limit reached error when the ingested log data exceeds the configured queue size. Typically, this is a temporary situation that occurs only during spikes. If this error persists, increase the value of disk_queue_max_size_mb in custom.properties to allow log ingestion spikes to be queued.

Example

In this example, the API request ingests log data that will create a log event with defined log attributes content, status, service.name, and service.namespace.

The API token is passed in the Authorization header.

The response contains response code 204.

Curl

curl -X POST \
   https://environment.activegate.domain.com:9999/e/abc123a/api/v2/logs/ingest \
  -H 'Content-Type: application/json; charset=utf-8' \
  -H 'Authorization: Api-Token dt0c01.abc123.abcdefjhij1234567890' \
  -d '[
    {
    "content": "Exception: Custom error log sent via Generic Log Ingest",
    "status": "error",
    "service.name": "log-monitoring-tenant",
    "service.namespace": "dev-stage-cluster"
    }
]'

Request URL

https://environment.activegate.domain.com:9999/e/abc123a/api/v2/logs/ingest

Response content

Success

Response code

204

Related topics