This document explains how to integrate AWS DevOps Agent with Dynatrace. After completing the steps in this guide, Dynatrace will initiate AWS DevOps Agent investigations for AWS-related Dynatrace problems (for example, issues running on AWS infrastructure). AWS DevOps Agent will analyze these and provide insights back to Dynatrace.
During the PoC phase, the Dynatrace AWS DevOps Agent integration consists of Dynatrace Workflows (for outbound and inbound communication) and a Dynatrace dashboard to explore AWS DevOps Agent investigation and mitigation results.
storage:events:writestorage:events:readstorage:buckets:readstorage:bizevents:readstorage:entities:readapp-engine:apps:runapp-engine:functions:runautomation:workflows:readautomation:workflows:runenvironment:roles:viewerdocument:documents:writedocument:documents:readBefore proceeding, complete the AWS setup documentation. For the Dynatrace AWS DevOps Agent integration, you’ll need the webhook details from the AWS setup:
// Example webhook details:{"webhookId": "b12043af-86ce-4c1e-99e5-8130eb6bbe80","webhookSecret": "aaaZdT1n16H/eeU+zXI7N97yXU+34KFitrAGyfvZBm1=","webhookUrl": "https://app.aidevops.aws.com/d1432fe6-08b3-4fc2-b5f4-e75ad27bbb82/investigation/c5f005a6-d272-4b46-9ef2-f905fe693ef6"}
Store the webhookSecret in Dynatrace Credential Vault:
Open Dynatrace Credential Vault and select Add new credential.
Set the following values:
webhookSecret propertySave and reopen the entry to retrieve its id (for example, CREDENTIALS_VAULT-0C4FCF561A3AA13C).
You’ll need this in step 2.1.
Dynatrace can trigger AWS DevOps Agent investigations and receive the results. It can also request mitigation steps for completed investigations.
Open
Workflows and install 01 Queue potentially AWS-relevant problems.yaml, 02_Dispatch pending analysis.yml, and 03_Check eligibility and invoke AWS DevOps Agent.yaml.
These workflows prepare
Problems to trigger AWS DevOps Agent investigations by calling the AWS DevOps Agent webhook.
Update the 03_Check eligibility and invoke AWS DevOps Agent workflow configuration:
Open Workflow options > Workflow input and result/Default Input.
Set:
credentialVaultId to the credential id created earlier.awsDevOpsAgentEndpoint to the webhookUrl provided by AWS.//Example default input:{"credentialVaultId": "CREDENTIALS_VAULT-0C4FCF561A3AA13C","awsDevOpsAgentEndpoint": "https://app.aidevops.aws.com/d1432fe6-08b3-4fc2-b5f4-e75ad27bbb82/investigation/c5f005a6-d272-4b46-9ef2-f905fe693ef6"}
Deploy the workflow 03_Check eligibility and invoke AWS DevOps Agent.
Go to the 02_Dispatch pending analysis workflow and select the run_analysis workflow action.
03_Check eligibility and invoke AWS DevOps Agent workflow.{"problemId": "{{ _.item.problemId }}"}
Deploy the rest of the workflows and make them all public.
01_Queue potentially AWS-relevant problems triggers on the arrival of a problem of type Error, Slowdown, or Availability and in active status. Queues the problem for further analysis.
02_Dispatch pending analysis runs every 1 minute. The trigger interval can be adapted to your needs. It picks up the queued problems and forwards these for processing to 03_Check eligibility and invoke AWS DevOps Agent.
03_Check eligibility and invoke AWS DevOps Agent is invoked by 02_Dispatch pending analysis. The Dynatrace Workflow evaluates whether the issue is potentially related to AWS. If the issue is determined to be AWS-related, an AWS DevOps Agent investigation is initiated; otherwise, the workflow concludes.
By default, workflows cannot access the Internet. To enable this, navigate to Settings > General > External requests and add a New host pattern to allow communication with the AWS DevOps Agent webhook.
Exemplary allowlist entry, e.g., app.aidevops.aws.com
Workflows, install 04_Handle AWS DevOps Agent responses.yaml. This workflow processes AWS DevOps Agent results sent to the Dynatrace generic events ingest endpoint.investigation_summary) will appear in Dynatrace within 5–15 minutes, but be aware that investigations may also take longer.fetch dt.davis.events| filter in(event.type,"CUSTOM_ANNOTATION")| filter in(problem.id, "TODO Add your problem id here")| sort event.end desc| limit 100
05_Handle AWS DevOps Agent investigation completed responses.yaml workflow. This workflow runs whenever an AWS DevOps Agent investigation concludes. It will trigger the mitigation action process in AWS DevOps Agent.If you want to disable the workflow 05_Handle AWS DevOps Agent investigation completed responses at a later stage, disable the trigger action.
06_Handle AWS DevOps Agent mitigation responses.yaml. This workflow processes AWS DevOps Agent results. The functionality is similar to step 2.3.Use
Dashboards to upload 07_Display AWS DevOps Agent investigation results.json. This dashboard provides an overview of AWS DevOps Agent investigation and mitigation results.
Use the dashboard from step 3 to monitor investigation and mitigation results.
Next to the out-of-the-box dashboard, you can also navigate to the Troubleshooting section in
Problems. The feature is disabled by default because it may result in the creation of many notebooks. For every problem shared with AWS DevOps Agent, a notebook will be created.
You can enable this feature by modifying the default input of the workflow 04_Handle AWS DevOps Agent responses installed in step 2.3.
//Example default input:{"createTSGNotebook": true}
Additionally, the default input of the workflow 06_Handle AWS DevOps Agent mitigation responses requires changes.
//Example default input:{"updateTSGNotebook": true}
For every AWS DevOps Agent response, a troubleshooting notebook will be automatically created and attached to the problem that triggered the AWS DevOps Agent investigation. The notebook is updated with mitigation recommendations.
03_Check eligibility and invoke AWS DevOps Agent workflowThe workflow 03_Check eligibility and invoke AWS DevOps Agent you installed in Step 2.1 can be executed in different modes:
dryRun mode.To manually trigger an AWS DevOps Agent investigation for a past problem, follow these steps:
03_Check eligibility and invoke AWS DevOps Agent workflow.{"problemId": "4364692318491551794_1762551360000V2", // Replace this with the actual Problem ID"fromQuery": "now()-30d", // Adjust the query time range if the problem occurred more than 30 days ago"toQuery": "now()" // Optimize the query range for efficient querying}
If a problem occurred more than 30 days ago, adjust the query timeframes.
Note that AWS DevOps Agent does not support multiple investigations for the same problem. Once a problem has been investigated in an AgentSpace, it cannot be investigated again.
By default, the workflow checks if a problem relates to AWS entities, for example, it runs on AWS infrastructure or interacts with resources operated in AWS. By changing the workflow configuration, you can trigger an AWS DevOps Agent investigation even if no AWS associations are detected.
Change the Workflow options > Workflow input and result/Default Input configuration:
{..."minAWSResources": 0,...}
03_Check eligibility and invoke AWS DevOps Agent workflow.{"dryRun": true,"problemId": "4364692318491551794_1762551360000V2", // Replace this with the actual Problem ID...}
Once you run the workflow, explore the logs to see what would have been sent to AWS DevOps Agent.
In the dryRun mode, no data is exchanged.
03_Check eligibility and invoke AWS DevOps Agent workflow.{"testConnectivity": true...}
Once you run the workflow, check the execution result of the workflow action start_aws_aidevops_investigation > Result.
It should be similar to:
status: 200eventId: testsuccess: trueawsdevopsagent: {"body": "{}","status": 200,"statusText": "OK"}
Verify that the user who created the Dynatrace OAuth Client has the openpipeline:events:ingest permission. To check this, navigate to Identity & Access Management > Effective Policies.
If the user does not have this permission, add it to their user. Once the permission is granted, you will start receiving responses from the AWS DevOps Agent.