Google Cloud Dataflow monitoring

Dynatrace Google Cloud integration leverages data collected from the Google Operation API to constantly monitor health and performance of Google Cloud Services. While combining all relevant data into dashboards, it also enables alerting and event tracking.

Prerequisites

Set up integration

Add services and feature sets optional

After integration, Dynatrace automatically monitors a number of preset Google Cloud services and feature sets (metrics). Besides these, you can add more services or feature sets to monitoring. For details, see Add or remove services.

For a list of feature sets available for this service, see Metric table.

View metrics

After deploying the integration, you can see metrics from monitored services in the Metrics browser, Data Explorer, and your dashboard tiles.

Metric table

The following feature sets are available for Google Cloud Dataflow.

Feature set
Name
Unit
GCP metric identifier
dataflow_job/default_metrics
Billable shuffle data processed
Byte
dataflow.googleapis.com/job/billable_shuffle_data_processed
dataflow_job/default_metrics
Current number of vCPUs in use
Count
dataflow.googleapis.com/job/current_num_vcpus
dataflow_job/default_metrics
Current shuffle slots in use
Count
dataflow.googleapis.com/job/current_shuffle_slots
dataflow_job/default_metrics
Data watermark lag
Second
dataflow.googleapis.com/job/data_watermark_age
dataflow_job/default_metrics
Elapsed time
Second
dataflow.googleapis.com/job/elapsed_time
dataflow_job/default_metrics
Element count
Count
dataflow.googleapis.com/job/element_count
dataflow_job/default_metrics
Elements Produced
Count
dataflow.googleapis.com/job/elements_produced_count
dataflow_job/default_metrics
Estimated byte count
Byte
dataflow.googleapis.com/job/estimated_byte_count
dataflow_job/default_metrics
Estimated Bytes Produced
Count
dataflow.googleapis.com/job/estimated_bytes_produced_count
dataflow_job/default_metrics
Failed
Count
dataflow.googleapis.com/job/is_failed
dataflow_job/default_metrics
Per-stage data watermark lag
Second
dataflow.googleapis.com/job/per_stage_data_watermark_age
dataflow_job/default_metrics
Per-stage system lag
Second
dataflow.googleapis.com/job/per_stage_system_lag
dataflow_job/default_metrics
PubsubIO.Read requests from Dataflow jobs
Count
dataflow.googleapis.com/job/pubsub/read_count
dataflow_job/default_metrics
Pub/Sub Pull Request Latencies
MilliSecond
dataflow.googleapis.com/job/pubsub/read_latencies
dataflow_job/default_metrics
Pub/Sub Publish Requests
Count
dataflow.googleapis.com/job/pubsub/write_count
dataflow_job/default_metrics
Pub/Sub Publish Request Latencies
MilliSecond
dataflow.googleapis.com/job/pubsub/write_latencies
dataflow_job/default_metrics
dataflow.googleapis.com/job/status
dataflow_job/default_metrics
System lag
Second
dataflow.googleapis.com/job/system_lag
dataflow_job/default_metrics
Total memory usage time
GigaByte
dataflow.googleapis.com/job/total_memory_usage_time
dataflow_job/default_metrics
Total PD usage time
GigaByte
dataflow.googleapis.com/job/total_pd_usage_time
dataflow_job/default_metrics
Total shuffle data processed
Byte
dataflow.googleapis.com/job/total_shuffle_data_processed
dataflow_job/default_metrics
Total streaming data processed
Byte
dataflow.googleapis.com/job/total_streaming_data_processed
dataflow_job/default_metrics
Total vCPU time
Second
dataflow.googleapis.com/job/total_vcpu_time
dataflow_job/default_metrics
User Counter
Count
dataflow.googleapis.com/job/user_counter

Related topics