Threat hunting and forensics

Kubernetes audit logs are forwarded to Dynatrace with aws.log_group and log_stream values attached. To fetch all the unique AWS CloudWatch log groups that are ingested to Dynatrace, copy-paste the following DQL query in the query input.

fetch logs
| summarize count(), by: aws.log_group

Select Run to execute the query.

In the query results, find the record with the log group collecting the EKS control plane logs (in our example, /aws/eks/unguard-secla-demo/cluster) and add it as a filter to your DQL query.

To view only the control plane audit events, modify the filter command in the query input by adding the and operator and the contains string function as follows:

| filter aws.log_group == "/aws/eks/unguard-secla-demo/cluster" and contains(aws.log_stream, "audit")

In the query input, remove the summarize command and select Run to execute the query.

In the query results table, right-click on any cell in the Content field and select Extract fields to navigate to DPL Architect.

Select Saved patterns.

In Dynatrace patterns, select k8s > audit.

When extracting fields from a JSON structure, you can define only a partial schema for the fields that are relevant for your use case. To continue your investigation, you need to select only the relevant fields.

In the query input of DPL Architect, replace the pattern as follows:

JSON{
STRING:verb,
JSON{string:username}(flat=true):user,
JSON_ARRAY{ipaddr}(typed=true):sourceIPs,
JSON{string+:resource}(flat=true):objectRef,
JSON{int:code}(flat=true):responseStatus
}(flat=true)

Select Results for an overview of the fields that will be extracted from the match preview dataset.

Select Insert pattern to append the pattern to your DQL query.

Select Run to execute the query.

In the query input, add the following DQL snippet, then select Run to execute the query.

| expand sourceIPs
| summarize count(), by: {sourceIPs, username, verb, resource=objectRef, responseStatus}
| filter in(responseStatus, {401, 403})

In the results table menu, sort results by count to see which IP addresses had the most connections.

Right-click on 198.51.100.2, then select Suspicious IPs.

Right-click on 172.31.29.138, select New list, enter a name for your new evidence list, for example, "Suspicious pod", then select Create.

Using the query tree created during your investigation, navigate to the Fetch Kubernetes cluster audit logs step.

In the query results, find the record with the log group that contains your VPC flow logs (in our example, /aws/vpc/unguard-secla-demo/vpc-flow-logs), and add it as a filter to the DQL query.

In the query input, remove the summarize command, then select Run to execute the modified query.

Go to Evidence collection and select the evidence menu for the Suspicious pod collection to see the filtering options and the suitable field names that match the IPADDR type.

Select Filter for > Filter within field pkt_srcaddr. This appends the filter to your DQL query.

To get a better overview about the network connections, append the following command to the DQL query in the query input:

| summarize count(), by: { pkt_dstaddr, protocol, action, dstport}
| sort `count()` desc

Select Run to execute the query.

To fetch CoreDNS container logs, navigate to the Fetch Kubernetes cluster audit logs step and modify the query in the query input as follows:

fetch logs
| filter k8s.container.name == "coredns"

Select Run to execute the query. This creates a third branch in the query tree.

In the query input, add the following snippet to the DQL query:

| parse name, "ld* '.'? ( (ld '.' ld):domain '.' eos)"
| summarize count = count(), by: {domain}

Select Run to execute the query.

You notice that, besides internal or local domains, one suspicious domain (tiitha-maliciousdomain.com) is being resolved quite a lot!

In the query results table, right-click on the suspicious domain name, then select New list.

Enter a name for your new evidence list, for example, Attacker domain, then select Create.

In the query results table, select the cell with the suspicious domain, then select Filter for to add a filter to the query that fetches only requests that include the attacker domain.

In the query input, remove the summarize command and select Run to execute the query.

It looks like there's some kind of data movement between the suspicious pod and the attacker domain. Looking at the query names and considering the number of queries, data appears to be extracted via DNS tunneling.

In the query results, select the type column header, then select Summarize.

Select Run to execute the query.

You notice many A, AAAA, and TXT requests from the pod. You start investigating the A requests.

In the query results table, right-click on an A cell and select Filter for to add a filter to the DQL query.

In the query input, remove the summarize command and select Run to execute the query.

To determine what is being sent out, you need to extract the subdomain portion of the domain. For this, you need to analyze one of the DNS queries that is resolving the attacker domain.

Double-click on any cell in the query results table.

In the Details (…) window, inspect the data in the name field.

It seems that the DNS name is structured by an ID followed by the payload portion encoded in hex, and these parts are delimited by ..

To parse and decode the payload, add the following DQL snippet to the DQL query:

| parse name, """ld:id '.' ld:payload '.tiitha-maliciousdomain'"""
| fieldsAdd payload=replaceString(payload,".","")
| fields timestamp, id, payload=decodeBase16ToString(payload)
| sort timestamp, id

The result proves that data from your pod is definitely being extracted and sent out!

Using the query tree, navigate to the Fetch Kubernetes cluster audit logs step.

In the query results, find the record with your Route53 log group (in our example, /aws/route53/unguard-secla-demo/resolver-logs) and add it as a filter to your DQL query.

In the query input, remove the summarize command and select Run to execute the query. This creates a fourth branch in the query tree.

In the query results table, right-click on any cell in the content field and select Extract fields.

In DPL Architect, select Saved patterns.

In Dynatrace patterns, select aws > route53-query.

You need to extract the query_name, query_type, srcaddr, and answers values from the log record. In the query input, you can replace the pattern as follows:

json{
string:query_name,
string:query_type,
json_array:answers,
ipaddr:srcaddr
}(flat=true)

Select Insert pattern.

Select Run to execute the query.

In the query results, select the query_name column header, then select Filter for > Attacker domain.

Select Run to execute the query.

You only care about the queries with the TXT type, so you can append the filter with the appropriate expression. Since the answers portion is an array, expand the field so that each value in the array is a separate record. The only fields you need are srcaddr, query_name, and the Rdata element from the answer object.

In the query input, modify the filter command by adding the following snippet:

| filter endsWith(query_name, "tiitha-maliciousdomain.com.") and query_type == "TXT"
| expand answers
| fields srcaddr, uery_name, answer=answers[Rdata]

Select Run to execute the query.

Your hypothesis is correct: the TXT DNS queries were used to fetch the commands. The executed commands (including the Kubernetes control plane requests as curl commands) show up as responses in your DNS logs!