Upgrade Problems Classic to Problems on Grail

The problem feed allows operations and SRE users to review all incoming active problems and the history of resolved ones. The general feed layout and structure is preserved in Problems, ensuring a smooth upgrade with little visual changes.

An example of the Problems app problem feed view

The problem feed layout in Problems features a similar problem feed with a filter bar, chart, column table, and a left side panel with default filters. Additionally, Problems introduces a number of new elements, such as:

Global problem indicator and default problem filters

The global problem indicator displays a total number of currently active problems. The count is based on your platform policy-related read access within the problem events table in Grail and on the default filter that you have set in the problem feed.

Access policies that are defined on top of events reduce the number of problems you can see both in the problem feed and in the global problem indicator. Additionally, if you set a personal default problem filter directly within the problem feed, it'll also apply to the global problem indicator and impact how many active problems you see. For example, you can reduce the global active problem count by configuring the default filter categories on the side panel. You can save your new default filter by selecting as seen below:

An example of applying changes to the default filter

You can also turn on email notification for problems matching the default filter by selecting > Turn on notifications.

Segment filters and data segmentation

Segment filters were introduced across the Dynatrace platform to replace Management Zones for UI-based filtering and sharing preconfigured filters to onboard your teams.

Filter by segments in the Problems app.

Compared to Management Zones, Segments help your teams manage their filters by offering the ability to use variables that form dynamic conditions of include blocks. This allows you to avoid creating multiple individual configurations and helps deduplicate other segments. By defining one variable that uses Grail field values, multiple individual management zones can be replaced with a single segment config as shown below:

An example of selecting segment's values in Problems

Instead of creating a separate management zone per AWS account ID, you can use a single segment with several selectable ID variables instead.

Filtering the problem feed with filter bar

Aside from Segment filters, the new problem feed offers multiple ways of filtering open problems for improved triaging including the filter bar.

The filter bar offers a number of filter possibilities such as logical operators AND and OR combined with complex filter statements and the possibility to use wildcard text filtering as shown below:

Advanced filtering with logical operators and wildcards within the Problems filter bar

Custom problem fields and user-defined columns

Problems introduces a new feature unavailable in Problems Classic—custom problem fields that you can define and display in the problem feed. By surfacing customer-specific information within the problem feed, operation teams can speed up the process of triaging the ownership and responsibility of incoming problems.

By configuring which event fields are extracted and propagated into a problem via Settings, you can:

Custom problem fields can also be used in any problem DQL query in Notebooks and Dashboards.

Additionally, you can use Settings to check which fields are surfaced in problems and where they come from. For example, policy-relevant fields such as cloud.region, dt.host_group.id, or k8s.cluster.name and fields related to cost allocation, such as dt.cost.costcenter or dt.cost.product, are displayed in problems by default. The owner of these default fields is listed as Problems, indicating that they come from the app and weren't created by a user.

Similar to Problems Classic, Problems provides a problem details page that shows impact and root cause of a larger scale incident.

The details page is structured similarly to Problems Classic, where the key important information about the problem is shown in the header, such as the problem name, its status, unique problem number (for example, P-25032996), its category (for example, Error), and its start time and duration.

Additionally, the problem header in Problems offers a possibility to ask Dynatrace Intelligence to summarize the entire problem and provide hints about possible remediation steps. You can do so by selecting Explain problem.

In Problems, the problem details page shows the general information at the top, the impact on the left side and surfacing the root cause on the right side.

An example of problem details page overview

The general information at the top of the page includes a number of KPI measures, such as the number of affected SLOs, affected frontends, affected services, infrastructure entities, and the number of affected users.

You can see the root cause identified by the Dynatrace causal AI highlighted in red both in the Impact on the left and the Root cause on the right side of the details page in the Overview tab. Aside from the general key data, the Overview tab shows a list of connected running workflows below the Visual resolution path and Comments that serves the same function as in Problems Classic.

Additionally, problem details page in Problems introduces other new capabilities, such as

Visual resolution path

The Visual resolution path helps you understand how Dynatrace Intelligence identified the root cause backend service. It graphically illustrates the relationships between frontends, services, and backends involved in the issue. Each node represents a Smartscape entity, such as frontend, service, or backend, where a health issue was detected.

By selecting Maximize on the top right, you can open the detailed view, complete with a timeline on the very bottom that shows when related events have been fired:

An example of maximized Visual resolution path in Problems

Drill-down actions

The drill-down actions available on the problem details page help you pinpoint the cause on a code level by navigating you to a specialized app like Services or Distributed Tracing. These apps have an integrated investigation mode that allows you to work across apps by carrying over the information from the problem details page. For example:

The drill-down action available to the left of the (for example, Analyze slowdown) shows the primary analysis option recommended by Dynatrace based on the identified root cause. Meanwhile, displays a list of relevance-ranked secondary actions. All the app-related actions automatically lead to a Problem investigation mode on the receiving application side, which is indicated by a badge at the top of the page.

For example, by navigating to Services via drill-down actions in the Root cause section, you can check the response time analysis of a detected service slowdown:

An example of problems analysis in Service after selecting View service action in Problems

Automation and remediation triggered in related workflows

Automation and remediation triggered by <problem-number> section in Overview tab displays a list of connected workflows and their status.

Workflows offers dedicated triggers to either react to incoming active events or newly created active problems. These Workflow triggers are used to send alert notifications or implement automations on top of Dynatrace-detected situations, which allows you to automate fixes to known potential problems or receive notifications for workflows configured with a problem trigger.

Automation and remediation triggered by <problem-number> allows you to track and check the status of all connected workflows without navigating to Workflows and checking individual configurations.

Problems and events are stored in Grail along with their record fields. The problem feed and problem details view use DQL to surface all details in a UI tailored for operations teams and SREs.

You can write DQL queries to directly access the same information within Dashboards, Notebooks, and Workflows to define custom reports and implement automations on top of Dynatrace-detected events and problems:

Which DQL query you should use depends on your particular use case. For example:

The same principles apply to the problem queries, which deliver all the details about problems detected and analyzed by Dynatrace Intelligence.

All four queries implicitly apply a bucket filter that only fetches events from Davis event buckets instead of loading from the much larger general event bucket.

The flexibility of querying events and problems allows you to freely create custom dashboards and reports, as shown in this example:

Integration partners building apps on top of the platform or outside of Dynatrace use the Grail Service API along with DQL to query events and problems detected by Dynatrace Intelligence.

Both the Grail Service API and the Grail Service SDK apply table-based access policies. This ensures consistency in accessing records independently of the client context like Notebooks, Dashboards, workflow automations, SDK, or API.

Dynatrace Classic uses Management Zones to define data access. In latest Dynatrace we replace Management Zones by introducing two new concepts, IAM Policies and Boundaries, for configuring data access management.

These new concepts apply to Problems and event records in Grail, meaning that access policies can be used to restrict a user's ability to see problems to a smaller defined subset.

Since defining policies on records within Grail is restricted to a subset of well-defined record fields, called Record level permissions in Grail, it's important that those fields are properly set and enriched in Problems.

If you need to use one or more custom fields in Problems to define the access scope, we recommend using the dt.security_context field. There are several ways to add a dt.security_context field to a problem or an event: