Detect threats in cloud-native environments using workflows

  • Latest Dynatrace
  • Tutorial

In this tutorial, you'll learn how to

  • Continuously detect and respond to active threats within your cloud-native environment
  • Find attackers trying to exploit your applications before they can negatively affect your business
  • Easily and flexibly search through huge amounts of data to find signs of suspicious behavior

Target audience

This article is intended for

  • DevOps, CloudOps, and SREs responsible for protecting workloads
  • Security analysts responsible for cloud-native environments

Prerequisites

  • Set up ingestion of Kubernetes audit logs, for example, through Amazon Firehose. Firehose can be used to ship logs from Amazon Elastic Kubernetes Service (EKS), but the provided template is compatible with any Kubernetes distribution.

  • Generate an access token with the openpipeline.events_security scope to allow us to ingest detection findings. To later access the token securely from within a workflow, store the token in the credential vault. For details, see Dynatrace API - Tokens and authentication.

Get started

After setting up log ingestion, you can start detecting threats.

Use Workflows to execute DQL queries regularly and look for suspicious behavior in your data.

  1. Import our workflow template.

  2. For the execute_query step, adapt the DQL query to your needs.

    The template query aims to find possibly compromised service accounts within our Kubernetes cluster.

    • You can tune it, adjust its logic or modify it completely to detect other potential threats.
    • You can find additional examples in the Dynatrace blog.

  3. For the ingest_findings step, go to Input > Authentication and select the token previously stored in the credential vault.

To trigger a detection finding based on the query provided, execute the following commands in a Dynatrace-monitored Kubernetes cluster:

  1. Create a test service account:

    kubectl create serviceaccount test-sa -n default

  2. Attempt unauthorized access (should trigger detection):

    kubectl --as=system:serviceaccount:default:test-sa get secret -A

You can define exceptions to the rule by adding allowlisting criteria to the apply_allowlist step in the workflow.

A detection finding event will not be created if all attributes of an entry match those of any single allowlisted entry.

  • Attribute comparison follows AND logic within an entry.
  • Allowlist evaluation uses OR logic across entries.

Use Threats & Exploits Threats & Exploits to examine and prioritize the detection findings created. The rich observability context provided in the detection finding enables fast analysis and confident assessment of the situation.

Sample result

sample result in threats and exploits app

Sample workflows for threat response

Dynatrace provides ready-to-use workflow templates that automatically trigger on new Kubernetes threat detection findings and send notifications via Slack, Microsoft Teams, or Email. Use these as starting points to build your own automated response workflows:

These templates show how to enrich findings with ownership context and integrate real-time alerting into your response workflows.

Conclusion

Workflows enable precise threat detection and response aligned with your environment’s context. Features like flexible scheduling, DQL-based filtering, allowlisting, ownership tagging, and notification integration allow for rapid, low-noise detection. The included blueprint offers a modular starting point that can be fine-tuned or extended to meet specific operational needs.

Related tags
Threat Observability