Integrate PagerDuty with Dynatrace Cloud Automation

Integrating Dynatrace Cloud Automation with PagerDuty empowers teams to take the right action and resolve incidents faster, mobilizing responders whenever significant problems occur in Dynatrace while coordinating other actions in Cloud Automation.

To integrate Dynatrace Cloud Automation with PagerDuty, follow the instructions below.

Prerequisites

PagerDuty

Dynatrace

PagerDuty

Create a PagerDuty account.

You need a license that includes Event Routing Rules and Alert Field Enrichment and Extraction (typically Business or Digital Operations).
Add an Incoming Event Source for Cloud Automation.
1. Log in to your PagerDuty account and go to Automation > Event rules.
2. Select Create ruleset and enter the following values:
  
  Title: Provide a name, such as dynatrace-cloudautomation
  Team: Assign a team or leave blank to keep the event as a global ruleset.
3. Select Create ruleset.
Add a PagerDuty REST API key that will be used to configure the webhook translator in AWS Lambda.
1. In PagerDuty, go to Integrations > API access key.
2. Select Create new API key.
3. Enter a description for the API key and tick Read-only API key.
4. Select Create key.
5. On the Ruleset page, copy and save the Integration key. This value will be used later on to create a CA secret.

Set up a Cloud Automation webhook translator to receive PagerDuty events from the PagerDuty event ruleset and make the Cloud Automation API call to send back a Cloud Automation sequence task finished event.

To create the translator:

Log in to your AWS console and navigate to AWS Lambda.
Select Create function, then Author from scratch.
For Name, enter a function name, such as pd-to-cloudautomation.
For Runtime, select Node.js 14.x.
Select Create function.
Select Add trigger.
Select API Gateway, then Create API > HTTP API > Security > Open.
Select Add to add the trigger.

After the trigger is created, you can upload it and use the URL to configure the Event Rule in PagerDuty.

To upload the translator:

Download the pre-build Lambda code ZIP file.
In AWS Lambda, select Code, then upload and save the ZIP file.

To configure the translator:

In AWS Lambda, select Configuration > Environment variables > Edit.

Add the following variables:

Variable name	Value
`KEPTN_BASE_URL`	The base URL from your Cloud Automation environment (for example, `https://abcd.cloudautomation.live.dynatrace.com`)
`KEPTN_API_TOKEN`	Your Cloud Automation API token
`PD_API_ACCESS_KEY`	Your PagerDuty API access key

Select Save.

Dynatrace

Create a problem notification webhook.

In the Dynatrace menu, go to Settings > Integrations > Problem notifications.
Select Add notification and enter the following values:
- Notification type: Select Custom integration
- Display name: Enter Dynatrace Cloud Automation
- Webhook URL: Enter your URL endpoint, for example [YOUR CLOUD AUTOMATION BASE URL]/api/v1/event
In Additional HTTP headers, you need to add two headers. Select Add item and enter the following values:
- First header:
  - Name: Enter x-token
  - Value: Enter your Cloud Automation API token from the Cloud Automation bridge
  - Secret header value: Enabled
- Second header:
  - Name: Enter Content-Type
  - Value: Enter application/cloudevents+json
  - Secret header value: Disabled

Enter a Custom payload.

Example:

1{
2   "specversion":"1.0",
3   "source":"dynatrace notifications webhook",
4   "contenttype":"application/json",
5   "type": "sh.keptn.event.production.incident.triggered",
6   "data": {
7      "project":"incident-demo",
8      "stage":"production",
9      "service":"casdemoapp",
10      "dynatraceProblem": {
11         "State":"{State}",
12         "ProblemID":"{ProblemID}",
13         "PID":"{PID}",
14         "ProblemTitle":"{ProblemTitle}",
15         "ProblemURL":"{ProblemURL}",
16         "ProblemDetails":{ProblemDetailsJSON},
17         "Tags":"{Tags}",
18         "ImpactedEntities":{ImpactedEntities},
19         "ImpactedEntity":"{ImpactedEntity}"
20      }
21   }
22}

Select Save changes.

Create a Cloud Automation project with a shipyard file that has a custom task (for example, openticket) to trigger the PagerDuty subscription.

Set up the PagerDuty webhook subscription

Use case: The Cloud Automation sequence will wait for PagerDuty Cloud Automation webhook translator to send back a Cloud Automation sequence task finished event.

Cloud Automation

PagerDuty

Cloud Automation

On the Cloud Automation bridge, select your Cloud Automation project.
Select Settings, and then select Integrations.
Select webhook-service from the list, and then select Add subscription.
Enter the following values:
- Task: Select the task associated with your shipyard file (in this case, openticket).
- Task suffix: Select triggered.
- Request method: Select POST.
- URL: Enter your API endpoint using the integration key obtained in Prerequisites (for example, https://events.pagerduty.com/x-ere/<YOUR-INTEGRATION-KEY?).
Select Add custom header and enter the following values:
- Name: Enter Content-Type.
- Value: Enter application/json.

Enter a Custom payload.

Example:

1{
2 "data": {
3  "project": "{{.data.project}}",
4  "service": "{{.data.service}}",
5  "stage": "{{.data.stage}}"
6},
7"shkeptncontext": "{{.shkeptncontext}}",
8"id": "{{.id}}",
9"type": "{{.type}}",
10"incident": {
11  "ProblemID": "{{.data.incident.ProblemID}}",
12  "ProblemTitle": "{{.data.incident.ProblemTitle}}",
13   "ProblemURL":"https://<YOUR DYNATRACE BASE URL>/ui/problems?displayId={ProblemID}",
14  "State": "{{.data.incident.State}}",
15  "Tags": "{{.data.incident.Tags}}"
16 }
17}

Under Send started event, select automatically.
Under Send finished event, select by webhook receiver.
Select Create subscription.

PagerDuty

Below is a ruleset example. You can adjust the PagerDuty Event Rules to route to different services depending on the application, project, stage, and so on.

In PagerDuty, select Automation > Event rules, then select the ruleset whose integration key you used in Cloud Automation.
Select New Event Rule. You should be able to see the test event you sent to PagerDuty. Select the + button to expand it.
Choose to run this rule when the following condition is met: type equals sh.keptn.event.openticket.triggered.

Expand Customize event fields, then under Define custom variable, add the following variables:

Name	Regex	Source
`shkeptncontext`	`.*`	shkeptncontext
`triggeredid`	`.*`	triggeredid
`project`	`.*`	data.project
`service`	`.*`	data.service
`stage`	`.*`	data.stage
`ProblemID`	`.*`	incident.ProblemID
`ProblemTitle`	`.*`	incident.ProblemTitle
`ProblemURL`	`.*`	incident.ProblemURL
`Tags`	`.*`	incident.Tags

Under Replace event field, add the following fields:

Event Field	Value	Template/Regex
`dedup_key`	`{{shkeptncontext}} - {{triggeredid}}`	Template
`summary`	`{{ProblemID}} - {{ProblemTitle}}`	Template
`class`	`{{stage}}`	Template
`component`	`{{service}}`	Template
`source`	`{{project}}`	Template
`group`	openticket	Template
custom_details > `shkeptncontext`	`{{shkeptncontext}}`	Template
custom_details > `triggeredid`	`{{triggeredid}}`	Template
custom_details > `ProblemURL`	`{{ProblemURL}}`	Template
custom_details > `Tags`	`{{Tags}}`

Expand Do these things, then select Create an incident on a service and select an appropriate PagerDuty service.
Select Advanced and enter the following values:
- Custom Action: Enable
  - Enable Custom Action
  - Trigger automatically on incident creation
- Name: Enter, for example, Send finished event to Dynatrace Cloud Automation.
- API endpoint Enter the URL of the API gateway trigger that you configured for your translator Lambda in AWS.
Select Save rule.

Trigger a sequence

Cloud Automation triggers PagerDuty whenever a Dynatrace problem is detected, and the Dynatrace problem notification event is sent to Cloud Automation.

To trigger the sequence, follow the steps below.

Trigger a problem in Dynatrace.
In PagerDuty, go to Incidents > Alerts and search for an alert named Incident routed via a Ruleset that occurred around the time when you triggered the problem.

Select the alert, then select View message. Similar content should be displayed:

1{
2   "event": "sh.keptn.event.evaluation.finished",
3   "incident_key": "7ee74459136b4b7ca694efa7b6164764",
4   "project": "demo",
5   "result": "fail",
6   "score": 61.11111111111111,
7   "service": "casdemoapp",
8   "stage": "production"
9
10}

If no PagerDuty incident is created, review the configuration steps and make sure you entered all information correctly.

On the Cloud Automation bridge, go to Sequences and check if the incident sequence was executed.
Select the incident sequence, then select the openticket task in the sequence.

Expected results:
- webhook-service should have started
- The pagerduty task should have finished successfully (pass)

Integrate PagerDuty with Dynatrace Cloud Automation

Prerequisites

PagerDuty

Dynatrace

Set up the PagerDuty webhook subscription

Cloud Automation

PagerDuty

Trigger a sequence

Related topics