Observe GitLab pipelines and merge requests with Dashboards and SDLC events

  • Latest Dynatrace
  • Tutorial
  • 3-min read
  • Preview

In this use case, you'll

  • Integrate GitLab and Dynatrace.
  • Use Dashboards Dashboards to observe GitLab pipelines and Workflows Workflows.
  • Use this information to streamline CI/CD pipelines, improve productivity, and generate data-driven insights.

Concepts

Software Development Lifecycle (SDLC) events

SDLC events are events with a separate event kind in Dynatrace that follow a well-defined semantics for capturing data points from a software component's software development lifecycle. The SDLC event specification defines the semantics of those events.

Why were GitLab webhook events changed into SDLC events?

The main benefit is data normalization and tool agnosticism. As a result, Dashboards Dashboards, apps, and Workflows Workflows can be built on SDLC events with well-defined properties rather than tool-specific details.

Target audience

This information is intended for platform engineers and DevOps engineers who use GitLab in their Internal Development Platform (IDP).

Learning outcome

In this tutorial, you'll learn how to

  • Forward GitLab webhook events to Dynatrace.
  • Normalize the ingested event data.
  • Use Dashboards Dashboards to analyze data and drive improvements.

Prerequisites

Install the Configuration As Code tool of your choice. Either install the Terraform CLI or install the Monaco CLI. Depending on the tool you choose, pick the correct setup section below.

Observe GitLab pipelines and merge requests

1. Setup: Prepare the configuration

  1. Create a new platform token with the following permissions and store it in a secure place:

    • Run apps: app-engine:apps:run.
    • View OpenPipeline configurations: settings:objects:read.
    • Edit OpenPipeline configurations: settings:objects:write.
    • Create and edit documents: document:documents:write.
    • View documents: document:documents:read.

  2. Clone the Dynatrace configuration as code sample repository using the following command.

    git clone https://github.com/Dynatrace/dynatrace-configuration-as-code-samples.git

2. Setup: Configuration as Code

You can choose between two options:

  • Terraform.
  • Monaco.

  1. Set up Terraform.

    1. Prepare the Terraform configuration.

      The configuration consists of

      1. Move to the gitlab_pipeline_observability_terraform directory with the following command.

        cd dynatrace-configuration-as-code-samples/gitlab_pipeline_observability_terraform

      2. Store the retrieved platform token in an environment variable.

        $env:DYNATRACE_PLATFORM_TOKEN='<YOUR_PLATFORM_TOKEN>'

      3. Store your Dynatrace environment URL in an environment variable. Make sure to replace <YOUR-DT-ENV-ID> with your Dynatrace environment ID, e.g. abc12345.

        $env:DYNATRACE_ENV_URL='https://<YOUR-DT-ENV-ID>.apps.dynatrace.com'

    2. Check the OpenPipeline configuration for SDLC events.

      These steps modify the OpenPipeline configuration for SDLC events. If your OpenPipeline configuration contains only default/built-in values, you can apply it directly. If you have any dynamic routes, first download your configuration and then manually merge it into the provided configuration.

      Step 3 indicates if a configuration merge is needed or if you can apply the provided configuration directly.

      1. Go to Settings > Process and contextualize > OpenPipeline > Software Development Lifecycle.

      2. Check the Dynamic routing section, are there any other routes than Default route?

      3. If the answer is yes, follow the steps in the next section. Otherwise, skip ahead.

    3. Merge OpenPipeline routing entries.

      If you have any OpenPipeline routing entries that aren't default, then you need to merge them before you can proceed.

      • Download your OpenPipeline configuration. You need to set up the DYNATRACE_ENV_URL and DYNATRACE_PLATFORM_TOKEN environment variables according to the Dynatrace Terraform provider installation guide. Then you can use the export utility of the Dynatrace Terraform provider to download all routing entries for SDLC events. Navigate to the directory that includes the terraform-provider-dynatrace_vx.y.z binary and use it to export the dynatrace_openpipeline_v2_events_sdlc_routing configurations, for example, ./terraform-provider-dynatrace_v1.86.0 -export dynatrace_openpipeline_v2_events_sdlc_routing.

        You should see an output similar to this example:

        The environment variable DYNATRACE_TARGET_FOLDER has not been set. Use the'configuration'  folder as default.
        Downloading "dynatrace_openpipeline_v2_events_sdlc_routing" Count:  1
        Post-Processing Resources ...
        - [POSTPROCESS] dynatrace_openpipeline_v2_events_sdlc_routing - openpipeline_v2_events_sdlc_routingPost-Processing Resources - Group child configs with parent configs ...
        Finishing touches ...
        Writing ___resources___.tf
        Writing ___datasources___.tf
        Writing main.tf
        Writing ___variables___.tf
        Writing main ___providers___.tf
        Writing modules ___providers___.tf
        Remove Non-Referenced Modules ...
        Finish Export ...
        Terraform executable path:  /usr/local/bin/terraform
        Executing 'terraform init'
        ... finished after 3 seconds

      • The export utility has created a folder configuration. You can now open the following files:

        • Your downloaded configuration file containing all available routing entries for SDLC events ./configuration/modules/openpipeline_v2_events_sdlc_routing/openpipeline_v2_events_sdlc_routing.openpipeline_v2_events_sdlc_routing.tf.
        • The provided file main.tf, which contains the routing configurations for the GitHub pipelines.

      • Merge all routing_entry blocks of your downloaded routing file into the routing_entries block of the resource events_sdlc_global_routing_table in main.tf, and then save the file.

      This is mandatory because the Dynamic Routing table is a global configuration, and the order of the entries, as well as the matcher clauses, determines the overall routing.

    4. Apply the Terraform configuration.

      Run this command to verify the provided Terraform configuration.

      terraform plan

      Run this command to apply the provided Terraform configuration.

      terraform apply

3. Setup: Create a Dynatrace access token

To receive events processed by OpenPipeline, you need an access token with the following OpenPipeline scopes:

  • openpipeline.events_sdlc.
  • openpipeline.events_sdlc.custom.

To generate an access token:

  1. Go to Access tokens Access Tokens.
  2. Select Generate new token.
  3. Enter a name for your token.
    Dynatrace doesn't enforce unique token names. You can create multiple tokens with the same name. Be sure to provide a meaningful name for each token you generate. Proper naming helps you to efficiently manage your tokens and perhaps delete them when they're no longer needed.
  4. Select the required scopes for the token.
  5. Select Generate token.
  6. Copy the generated token to the clipboard. Store the token in a password manager for future use.

    You can only access your token once upon creation. You can't reveal it afterward.

  1. Select these scopes:

    • OpenPipeline - Ingest Software Development Lifecycle Events (Built-in) (openpipeline.events_sdlc)
    • OpenPipeline - Ingest Software Development Lifecycle Events (Custom) (openpipeline.events_sdlc.custom)

  2. Save the generated token securely for subsequent steps. We refer to it as <YOUR-ACCESS-TOKEN>.

4. Setup in GitLab: Create the GitLab webhook

  1. In GitLab, create the webhook with the following settings:

    1. Enter the URL of your placeholders for your Dynatrace environment ID <YOUR-DT-ENV-ID>.

      https://<YOUR-DT-ENV-ID>.live.dynatrace.com/platform/ingest/custom/events.sdlc/gitlab

    2. You can enter an optional webhook name and description, but skip the Secret token setting since a custom header manages request validation.

    3. In the Trigger section, select the following events to trigger the webhook.

    • Merge request events
    • Job events
    • Pipeline events
    • Deployment events
    • Release events

  2. Add a custom header to your webhook with the name Authorization and value Api-Token <YOUR-ACCESS-TOKEN>.

5. Unlock enhanced development insights with GitLab Dashboards

Now that you've successfully configured GitLab and Dynatrace, you can use Dashboards Dashboards and SDLC events to monitor your GitLab pipelines and merge requests within the entire development organization.

Analyze

In Dynatrace, open the GitLab Pipeline Pulse, GitLab Merge Request, and GitLab Deployments dashboards to:

  • Track real-time activities of merge requests.
  • Analyze CI/CD pipeline execution details and pipeline health.
  • Gain job insights.
  • Review step durations for pipelines.
  • Analyze deployment and release activities.

To try out, go to Dynatrace Playground.

Optimize

Use these insights for the following improvement areas:

  • Increase CI/CD pipeline efficiency.

    Observing workflow executions lets you identify bottlenecks and inefficiencies in your CI/CD pipelines.

    Knowing about these bottlenecks and inefficiencies helps optimize build and deployment processes, leading to faster and more reliable releases.

  • Improve developer productivity.

    Automated pipelines reduce the manual effort required for repetitive tasks, such as running tests and checking coding standards. This automation allows developers to focus more on writing code and less on administrative tasks.

  • Get data-driven development insights. Analyzing telemetry data from CI/CD pipelines provides valuable insights into the development process. You can use the telemetry data to make informed decisions and continuously improve the development flows.

Continuous improvements

Check and adjust your CI/CD pipelines regularly to make sure they're running smoothly.

In Dynatrace, adjust the timeframe of the relevant dashboards to monitor the long-term impact of your improvements.

Call to action

We highly value your insights on pipeline observability. Your feedback is crucial in helping us enhance our tools and services. Visit the Dynatrace Community page to share your experiences, suggestions, and ideas directly on the Feedback channel for CI/CD Pipeline Observability.

Further reading

Related topics

Related tags
Software DeliveryDashboardsDashboardsOpenPipelineOpenPipeline