Observe GitLab pipelines and merge requests with Dashboards and SDLC events

  • Latest Dynatrace
  • Tutorial
  • 6-min read
  • Preview

Preview

In this use case, you'll

  • Integrate GitLab and Dynatrace.
  • Use Dashboards to observe GitLab pipelines and Workflows.
  • Use this information to make decisions about streamlining CI/CD pipelines, improving productivity, and getting data-driven insights.

Concepts

Software Development Lifecycle (SDLC) events

SDLC events

Why were GitLab webhook events changed into SDLC events?

The main benefit is data normalization and becoming tool agnostic. As a result, Dynatrace Dashboards, Apps, and Workflows can build on SDLC events with well-defined properties rather than tool-specific details.

Target audience

This information is intended for platform engineers who use GitLab in their Internal Development Platform (IDP).

What will you learn

In this tutorial, you'll learn how to

  • Forward GitLab webhook events to Dynatrace.
  • Normalize the ingested event data.
  • Use Dashboards to analyze the data and derive improvements.

Prerequisites

Install Dynatrace Configuration as Code via Monaco

Steps

  1. Create an OAuth client for the Dynatrace Monaco CLI with the following permissions:

    • Run apps: app-engine:apps:run
    • View OpenPipeline configurations: openpipeline:configurations:read
    • Edit OpenPipeline configurations: openpipeline:configurations:write
    • Create and edit documents: document:documents:write
    • View documents: document:documents:read

  2. Store the retrieved client ID and secret as separate environment variables.

    $env:OAUTH_CLIENT_ID='<YOUR_CLIENT_ID>'

    $env:OAUTH_CLIENT_SECRET='<YOUR_CLIENT_SECRET>'

  3. Clone the Dynatrace configuration as code sample repository using the following commands and move to the gitlab_pipeline_observability directory.

    git clone https://github.com/Dynatrace/dynatrace-configuration-as-code-samples.git
    cd dynatrace-configuration-as-code-samples/gitlab_pipeline_observability

  4. Edit the manifest.yaml by exchanging the <YOUR-DT-ENV-ID> placeholder with your Dynatrace environment ID at the name property and within the URL of the value property.

    manifestVersion: 1.0
    projects:
    - name: pipeline_observability
    environmentGroups:
    - name: group
        environments:
        - name: <YOUR-DT-ENV-ID>
            url:
            type: value
            value: https://<YOUR-DT-ENV-ID>.apps.dynatrace.com
            auth:
                oAuth:
                clientId:
                    name: OAUTH_CLIENT_ID
                clientSecret:
                    name: OAUTH_CLIENT_SECRET

These steps modify the OpenPipeline configuration for SDLC events. If your OpenPipeline configuration contains only default/built-in values, you can directly apply the Monaco configuration. If you have any custom ingest sources, dynamic routes, or pipelines, you'll first need to download your configuration and manually merge it into the Monaco configuration.

Step 3 will indicate if a configuration merge is needed or if you can apply the provided configuration directly.

  1. Go to Settings Settings > Process and contextualize > OpenPipeline > Events > Software development lifecycle.

  2. Check the Ingest sources, Dynamic routing, and Pipelines.

    • Under Ingest sources, are there any other sources than Endpoint for Software Development Lifecycle events?
    • Under Dynamic routing, are there any other routes than Default route?
    • Under Pipelines, are there any other pipelines than events.sdlc?

  3. If the answer to one of those questions is "yes", follow the steps below. Otherwise, skip ahead to step 4.

    • Download your OpenPipeline configuration

      monaco download -e <YOUR-DT-ENV-ID> --only-openpipeline

    • Open the following files:

      • Your downloaded configuration file, download_<DATE>_<NUMBER>/project/openpipline/events.sdlc.json.
      • The provided configuration file, pipeline_observability/openpipline/events.sdlc.gitlab.json.

    • Merge the contents of events.sdlc.json into events.sdlc.gitlab.json, and then save the file.

  4. Apply the Monaco configuration.

    The configuration consists of

  • Dashboards to analyze GitLab activities.

  • OpenPipeline configuration to normalize GitLab events into SDLC events.

    Run this command to apply the provided Monaco configuration.

    monaco deploy manifest.yaml

To generate an access token:

  1. Go to Access tokens Access Tokens.
  2. Select Generate new token.
  3. Enter a name for your token.
    Dynatrace doesn't enforce unique token names. You can create multiple tokens with the same name. Be sure to provide a meaningful name for each token you generate. Proper naming helps you to efficiently manage your tokens and perhaps delete them when they're no longer needed.
  4. Select the required scopes for the token.
  5. Select Generate token.
  6. Copy the generated token to the clipboard. Store the token in a password manager for future use.

    You can only access your token once upon creation. You can't reveal it afterward.

  1. Select these scopes:

    • OpenPipeline - Ingest Software Development Lifecycle Events (Built-in)(openpipeline.events_sdlc)
    • OpenPipeline - Ingest Software Development Lifecycle Events (Custom)(openpipeline.events_sdlc.custom)

  1. In GitLab, create the GitLab webhook with the following settings

    1. Enter the URL of your placeholders for your Dynatrace environment ID <YOUR-DT-ENV-ID> and access token <YOUR-ACCESS-TOKEN>.

      https://<YOUR-DT-ENV-ID>.live.dynatrace.com/platform/ingest/custom/events.sdlc/gitlab

    2. You can enter an optional webhook name and description, but skip the Secret token setting since a custom header manages request validation.

    3. In the Trigger section, select the following events to trigger the webhook.

    • Merge request events
    • Job events
    • Pipeline events
    • Deployment events
    • Release events

  2. Add custom header to your webhook with the name Authorization and value Api-Token <YOUR-ACCESS-TOKEN>.

Now that you've successfully configured GitLab and Dynatrace, you can use Dashboards and SDLC events to observe your GitLab pipelines and merge requests within the entire development organization.

Analyze

In Dynatrace, open the GitLab Pipeline Pulse, GitLab Merge Request and GitLab Deployments dashboards to:

  • Track real-time activities of merge requests.
  • Analyze CI/CD pipeline execution details and pipeline health.
  • Gain job insights.
  • Review step durations for pipelines.
  • Analyze deployment and release activities.

Optimize

Leverage those insights for the following improvement areas:

  • Streamline CI/CD pipeline: Observing pipeline executions lets you identify bottlenecks and inefficiencies in your CI/CD pipelines.

    Knowing about these bottlenecks and inefficiencies helps optimize build and deployment processes, leading to faster and more reliable releases.

  • Improve developer productivity: Automated pipelines reduce the manual effort required for repetitive tasks, such as running tests and checking coding standards. This automation allows developers to focus more on writing code and less on administrative tasks.

  • Get data-driven development insights: Analyzing telemetry data from merge requests and pipelines provides valuable insights into the development process.

    You can use the telemetry data to make informed decisions and continuously improve the development flows.

Continuous improvements

Regularly review and tweak your CI/CD pipelines to ensure they are optimized for performance.

In Dynatrace, adjust the timeframe of the GitLab Pipeline Pulse and GitLab Merge Request dashboards to monitor the long-term impact of your improvements.

Call to action

We highly value your insights on GitLab pipeline observability. Your feedback is crucial in helping us enhance our tools and services. Visit the Dynatrace Community page to share your experiences, suggestions, and ideas directly in Feedback channel for CI/CD Pipeline Observability.

Further reading

Related tags
Software DeliveryDashboardsDashboardsOpenPipelineOpenPipeline