Try it free

Google Cloud Dataflow monitoring

  • How-to guide
  • 2-min read
  • Published Jan 17, 2022

Dynatrace Google Cloud integration leverages data collected from the Google Operation API to constantly monitor health and performance of Google Cloud Services. While combining all relevant data into dashboards, it also enables alerting and event tracking.

Prerequisites

Set up integration

Add services and feature sets Optional

After integration, Dynatrace automatically monitors a number of preset Google Cloud services and feature sets (metrics). Besides these, you can add more services or feature sets to monitoring. For details, see Add or remove services.

For a list of feature sets available for this service, see Metric table.

View metrics

After deploying the integration, you can see metrics from monitored services in the Metrics browser, Data Explorer, and your dashboard tiles.

Metric table

The following feature sets are available for Google Cloud Dataflow.

Feature setNameUnitGCP metric identifier

dataflow_job/default_metrics

Billable shuffle data processed

Byte

dataflow.googleapis.com/job/billable_shuffle_data_processed

dataflow_job/default_metrics

Current number of vCPUs in use

Count

dataflow.googleapis.com/job/current_num_vcpus

dataflow_job/default_metrics

Current shuffle slots in use

Count

dataflow.googleapis.com/job/current_shuffle_slots

dataflow_job/default_metrics

Data watermark lag

Second

dataflow.googleapis.com/job/data_watermark_age

dataflow_job/default_metrics

Elapsed time

Second

dataflow.googleapis.com/job/elapsed_time

dataflow_job/default_metrics

Element count

Count

dataflow.googleapis.com/job/element_count

dataflow_job/default_metrics

Elements Produced

Count

dataflow.googleapis.com/job/elements_produced_count

dataflow_job/default_metrics

Estimated byte count

Byte

dataflow.googleapis.com/job/estimated_byte_count

dataflow_job/default_metrics

Estimated Bytes Produced

Count

dataflow.googleapis.com/job/estimated_bytes_produced_count

dataflow_job/default_metrics

Failed

Count

dataflow.googleapis.com/job/is_failed

dataflow_job/default_metrics

Per-stage data watermark lag

Second

dataflow.googleapis.com/job/per_stage_data_watermark_age

dataflow_job/default_metrics

Per-stage system lag

Second

dataflow.googleapis.com/job/per_stage_system_lag

dataflow_job/default_metrics

PubsubIO.Read requests from Dataflow jobs

Count

dataflow.googleapis.com/job/pubsub/read_count

dataflow_job/default_metrics

Pub/Sub Pull Request Latencies

MilliSecond

dataflow.googleapis.com/job/pubsub/read_latencies

dataflow_job/default_metrics

Pub/Sub Publish Requests

Count

dataflow.googleapis.com/job/pubsub/write_count

dataflow_job/default_metrics

Pub/Sub Publish Request Latencies

MilliSecond

dataflow.googleapis.com/job/pubsub/write_latencies

dataflow_job/default_metrics

dataflow.googleapis.com/job/status

dataflow_job/default_metrics

System lag

Second

dataflow.googleapis.com/job/system_lag

dataflow_job/default_metrics

Total memory usage time

GigaByte

dataflow.googleapis.com/job/total_memory_usage_time

dataflow_job/default_metrics

Total PD usage time

GigaByte

dataflow.googleapis.com/job/total_pd_usage_time

dataflow_job/default_metrics

Total shuffle data processed

Byte

dataflow.googleapis.com/job/total_shuffle_data_processed

dataflow_job/default_metrics

Total streaming data processed

Byte

dataflow.googleapis.com/job/total_streaming_data_processed

dataflow_job/default_metrics

Total vCPU time

Second

dataflow.googleapis.com/job/total_vcpu_time

dataflow_job/default_metrics

User Counter

Count

dataflow.googleapis.com/job/user_counter

Related topics

  • Google Cloud integrations
Related tags
Infrastructure Observability