Log ingestion
Log ingestion is the process of collecting log data from various sources within an infrastructure. The logs are stored in the Grail data lakehouse for analysis, automation, and monitoring. Dynatrace simplifies this process with OneAgent, which automatically discovers logs and offers central management options. In serverless environments or where OneAgent installation isn't possible, the Logs Ingestion API can be used.
Find below an overview of Dynatrace's log ingest strategies.
OneAgent
recommended
Automatically ingest log data from a wide variety of sources.
Log ingestion API
Configure Log ingest API integration for your use cases.
Dynatrace Extensions
Extend log observability with Dynatrace Extensions.
Syslog ingestion via ActiveGate
Preview
Ingest syslog logs.
Ingest via OneAgent
recommended
OneAgent is a recommeded, powerful tool that automatically finds log sources from a wide range of technologies on many different platforms, container orchestartion and operating systems. Refer to OneAgent platform and capability support matrix to see the supported operating systems.
We recommend using OneAgent for logs, as it provides the following advantages:
- Simplified instrumentation for hosts, processes, and Kubernetes clusters.
- Seamless installation on hosts, and Operator for Kubernetes ensures a first-class experience with built-in logs observability.
- Out-of-the-box log enrichment with contextual information such as topology and Kubernetes metadata.
- One-click opt-in for trace context inclusion in logs, enhancing traceability.
- Automatic detection of critical logs coupled with flexible custom log source configuration, ensuring comprehensive observability.
- Advanced log management capabilities at scale, offering configurations for log formats, sensitive data masking, and capture and processing filtering.
OneAgent for host logs
OneAgent simplifies log management by automatically decorating logs based on infrastructure and log source context, and enabling one-click trace enrichment for enhanced troubleshooting. Installation and central log ingestion rules setup in Dynatrace are all it takes to start monitoring logs. OneAgent also offers advanced features for scalable log management, including filtering, masking sensitive data, custom log source definition, log rotation pattern detection, and centralized configuration for easier lifecycle management. Learn more by accessing the Log ingestion via OneAgent.
"timestamp": "2024-05-23T15:46:23.000000000+02:00","content": "2024-05-23 15:46:23 WebLaunche ERROR [HeadlessVisitRunnable] DriverEntry shutDown. [com.dynatrace.diagnostics.uemload.headless.DriverEntry@647129f3 useCnt: [4] drv: [ChromeDriver: chrome on LINUX (01b4aedd5176375e9712d60df153d6a2) http://localhost:17828] proxy: [org.littleshoot.proxy.impl.DefaultHttpProxyServer@4598e617 /127.0.0.1:45875] chrome_driver: [http://localhost:17828] debug port: [33787] ip: [91.172.93.134] healthy: [true]]","dt.entity.host": "HOST-9A17CDBA8FF4FCBB","dt.source_entity": "HOST-9A17CDBA8FF4FCBB","event.type": "LOG","host.name": "demodev-master","log.source": "/home/labuser/.dynaTrace/easyTravel 2.0.0/easyTravel/log/WebLauncher.log","loglevel": "ERROR","process.technology": ["Apache Tomcat","Java"],"status": "ERROR","date_ingested": "2024-05-22T22:14:42.079000000Z"
Kubernetes logs via OneAgent
Dynatrace Log Monitoring enables the collection of logs from Kubernetes container orchestration systems through OneAgent. Kubernetes logs ingestion via OneAgent includes out-of-the-box sensitive data masking, entity linking and preservation of Kubernetes metadata. You can centrally configure OneAgent ingestion rules across your entire Kubernetes environment. By applying centralized filtering rules, you can ensure that only logs relevant to your use case are ingested, reducing maintenance efforts.
Read more about configuring log ingest from Kubernetes by accessing the Log Monitoring in Kubernetes page.
{"timestamp": "2024-05-23T15:55:23.000000000+02:00","content": "2024/05/23 13:55:23 Failed to export to Stackdriver: rpc error: code = PermissionDenied desc = The caller does not have permission","dt.entity.cloud_application": "CLOUD_APPLICATION-63AACD91ADBAB15F","dt.entity.cloud_application_instance": "CLOUD_APPLICATION_INSTANCE-F731124830922265","dt.entity.cloud_application_namespace": "CLOUD_APPLICATION_NAMESPACE-0A4EA744229201C9","dt.entity.container_group": "CONTAINER_GROUP-4F1B012F9B098D9F","dt.entity.container_group_instance": "CONTAINER_GROUP_INSTANCE-D8EF90CDA84B35F2","dt.entity.gcp_zone": "GCP_ZONE-4E0474C4AFCCC79A","dt.entity.host": "HOST-C4E8984646B39EBE","dt.entity.kubernetes_cluster": "KUBERNETES_CLUSTER-324E5954D86018E3","dt.entity.kubernetes_node": "KUBERNETES_NODE-4B5BC37280D9BFD6","dt.entity.process_group": "PROCESS_GROUP-B6AA568F4AD316D7","dt.entity.process_group_instance": "PROCESS_GROUP_INSTANCE-8E2A55B6CF37CF42","dt.kubernetes.cluster.name": "gke","dt.kubernetes.node.system_uuid": "592f7b67-a340-e136-a9a2-488969f9fe34","dt.process.name": "server frontend-*","dt.source_entity": "PROCESS_GROUP_INSTANCE-8E2A55B6CF37CF42","event.type": "LOG","gcp.instance.id": "7994835647533846587","gcp.project.id": "dynatrace-demoability","gcp.region": "us-central1","host.name": "gke-keptn-demo1-e2-custom-4-8192-08f6a08a-1xvo.c.dynatrace-demoability.internal","k8s.container.name": "server","k8s.deployment.name": "frontend-*","k8s.namespace.name": "online-boutique","k8s.pod.name": "frontend-7cc5676659-j2n5l","k8s.pod.uid": "776226ff-4a33-4ea5-807e-2c930759d6eb","log.source": "Container Output","loglevel": "ERROR","process.technology": ["C-Library","Containerd","Go"],"status": "ERROR","OperatorVersion": "v1.1.0","gcp.zone": "us-central1-c","k8s.cluster.uid": "74d7702f-11bf-445f-8fbc-2998804007ab","k8s.node.name": "gke-keptn-demo1-e2-custom-4-8192-08f6a08a-1xvo","log.iostream": "stderr"},
Ingest via Log Ingestion API
When unable to install OneAgent, use the Log Ingestion API. For example, in serverless environments like AWS Fargate, where logging relies on a built-in log router such as Fluent Bit, which can be easily integrated with the Dynatrace Log Ingestion API. The Log ingest API allows you to stream log records to the Grail data lakehouse, and have Dynatrace transform the stream into meaningful log messages. You can configure Log ingest API integration for the vast variety of use cases, and you can include custom integrations. You can use our supported integrations for clouds or log shippers and for your custom use cases.
You can configure Log Ingestion API integration for the following log shippers: OpenTelemerty Collector, Fluentbit, Fluentd, Logstash, and any other solution that integrates with REST API.
Learn more by accessing the Log Ingestion API page. JSON, text and OTLP formats are supported.
Cloud platforms monitoring
Cloud log forwarding allows the streaming of log data from various cloud platforms directly into Dynatrace. The following integrations are available:
- AWS: Use Amazon Data Firehose integration, Amazon S3 forwarder, and direct AWS Lambda integration for cost-optimized flow logs setup with Dynatrace. For example, logging IP traffic using the VPC flow Logs enables you to capture information from network interfaces in S3 and forwarding to Dynatrace platform to support troubleshooting and security use cases.
- Azure: Stream logs from Azure Event Hubs into Dynatrace through the Azure Function App instance. Azure resource logs and activity logs are supported. Dynatrace purchased via Azure Marketplace comes with deep Azure platform logs integration. It offers streamlined configuration via Azure Portal, and simplifies financial settlments.
- GCP observability: Create a Pub/Sub subscription to facilitate the ingestion of metrics, logs, dashboards, and alerts into Dynatrace. This provides a comprehensive view of your Google Cloud Platform health, including resource and audit logs.
Learn more by accessing the Cloud log forwarding page.
Ingest via Dynatrace Extensions
Logs are observability data that Dynatrace Extensions collect and forward to Grail together with other monitoring singals to deliver holistic view of your technology. Extensions expand observability data and analytics capabilities, streamlining data configuration and integration with third-party systems.
You can use the local http://localhost:<port>/v2/logs/ingest API endpoint to push locally retrieved logs to Dynatrace over a secure and authenticated channel. Learn more by accessing the Extensions page.
Syslog ingestion
Syslog is a standard protocol for message logging and system logs management. Routers, printers, hosts, switches and other devices across platforms use syslog to log users' activity, system and software lifecycle events, status, or diagnostics.
When you ingest syslog logs into Dynatrace, you can eliminate the need for separate syslog server infrastructure, reducing both costs and maintenance efforts. With syslog data, you can create events, metrics, and alerts to address connectivity issues or configuration errors.
However, if you decide to keep using syslog servers, you can forward logs to Dynatrace for advanced analytics without disrupting existing setups. This ensures that logs from a multitude of devices can still be collected and multicast to various endpoints, including Dynatrace for syslog monitoring.
Syslog logs are ingested via syslog receiver available on the Environment ActiveGate.
For more information, see Syslog ingestion.