Monitor Couchbase Capella and Server clusters via Prometheus and API metrics.
The Couchbase extension is designed to monitor Couchbase databases. It provides monitoring metrics at the cluster and node level.
Monitoring Cappella is handled by the extension runnng on seleced ActiveGate or ActiveGate group.
Virtual Machine
If your Couchbase is running on a virtual machine directly, you will need to install OneAgent on that virtual machine to get started.
Kubernetes workload
If your Couchbase is running as a workload in Kubernetes, deploy Dynatrace on Kubernetes. See Kubernetes deployment start guide.
Openshift workload
If your Couchbase is running as a workload in OpenShift, deploy Dynatrace on OpenShift. See OpenShift deployment start guide.
If you are instrumenting Couchbase with OpenTelemetry:
The extension requires several corresponding settings in the Capella/Couchbase Server Admin UI:
Capella
This is the Capella Connection String. In the Capella UI, go to the Connect tab on a Database, copy the full connection string and save paste in the monitoring configuration Connection string field.
Server
IP of a Couchbase Node. The Couchbase Node must be accessible from the ActiveGate the extension is running on.
User Credentials
The extension requires a set of Capella credentials (username / password) that has Read Access to All Buckets and All Scopes in your Database. The required external_stats_reader role is only granted when a Database Credential is given Read Access to all Buckets in a Database. See Manage database users in Couchbase documentation.
To create credentials, in the Capella UI, go to Settings > Database Access > Create Database Access. See Manage Cluster Access Credentials in Couchbase documentation.
Create a user with Read-Only Admin permissions.
The Security Certificate for your Database. In the Capella UI, select the Database, go to Settings > Security Certificate, and select Copy.
This will give you a PEM-formatted text. You can paste this in the Certificate Contents monitoring configuration field that is available when the Security Certificate Text checkbox is checked.
Alternatively, you can download the certificate and add it to your ActiveGate. Make sure that the user dtuserag is able to read the file.
Before any client can connect to a Capella Database, the client's IP address must be added to the Database's Allowed IP list.
Note: For Allowed IP address, you must find the external IP address of the ActiveGate(s) you are running the extension on and add those IPs to the allowed IP address setting.
The Couchbase extension is designed to monitor Couchbase databases. It provides monitoring metrics at the cluster and node level.
The extension includes FeatureSets that can be enabled based on your monitoring needs and desired licensing consumption. At minimum, we recommend enabling the Cluster metrics FeatureSet.
These FeatureSets collect metrics from Couchbase's Prometheus /metrics endpoint. Use these FeatureSets to continue monitoring your Couchbase deployment the way you have in previous versions of the extension:
These FeatureSets collect metrics from various Couchbase API endpoints. These metrics provide more detail than can be gathered from the Prometheus FeatureSets. They can be used instead of, or alongside the Prometheus FeatureSets.
If you have been previously monitoring Couchbase via the Couchbase UI, these metrics are directly equivalent to what is available in the UI.
These FeatureSets are specifically for monitoring Couchbase Sync Gateway (only enable if monitoring Sync Gateway):
When activating your extension using monitoring configuration, you can limit monitoring to one of the feature sets. To work properly the extension has to collect at least one metric after the activation.
In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.
All metrics that aren't categorized into any feature set are considered to be the default and are always reported.
A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.
| Metric name | Metric key | Description |
|---|---|---|
| Current MCBP/DCP Connections | couchbase.kv_curr_connections | Current number of MCBP/DCP connections |
| Waiting for commit (bucket) | couchbase.kv_disk_seconds_bucket | Waiting for a commit after a batch of updates |
| Waiting for commit (count) | couchbase.kv_disk_seconds_count.gauge | Waiting for a commit after a batch of updates |
| Waiting for commit (sum) | couchbase.kv_disk_seconds_sum | Waiting for a commit after a batch of updates |
| Disk used by Bucket | couchbase.kv_ep_db_data_size_bytes | Total size of valid data in db files |
| Disk drain rate | couchbase.kv_ep_diskqueue_drain | Total drained items on disk queue |
| Enqueued items on disk queue | couchbase.kv_ep_diskqueue_fill | Total enqueued items on disk queue |
| Total memory used in disk queue | couchbase.kv_ep_diskqueue_memory_bytes | Total memory used in disk queue |
| Disk write queue - being written | couchbase.kv_ep_flusher_todo | Number of items currently being written |
| Disk used percent by node | couchbase.kv_ep_magma_total_disk_usage_bytes | Disk used percent by node |
| Items queued for storage | couchbase.kv_ep_queue_size | Number of items queued for storage |
| Temporary out of memory errors by bucket | couchbase.kv_ep_tmp_oom_errors | Number of times temporary OOMs happened while processing operations |
| Total engine memory usage | couchbase.kv_mem_used_bytes | Engine's total memory usage |
| Get ops per second | couchbase.kv_vb_ops_get | Get Ops Per Second |
| Set ops per second | couchbase.kv_vb_ops_set | Set Ops Pet Second |
| Active item resident ratio | couchbase.kv_vb_perc_mem_resident_ratio | Active item resident ratio |
| VB queue drain | couchbase.kv_vb_queue_drain | VB queue drain |
| VB queue fill | couchbase.kv_vb_queue_fill | VB queue fill |
| VB queue pending | couchbase.kv_vb_queue_pending_bytes | VB queue pending |
| VB queue size | couchbase.kv_vb_queue_size | VB queue size |
| Metric name | Metric key | Description |
|---|---|---|
| Unsuccessful authentications | couchbase.sync_gateway.sgw_security_auth_failed_count | The total number of unsuccessful authentications. |
| Successful authentications | couchbase.sync_gateway.sgw_security_auth_success_count | The total number of successful authentications. |
| Total authentication time | couchbase.sync_gateway.sgw_security_total_auth_time | The total time spent in authenticating all requests. |
| Metric name | Metric key | Description |
|---|---|---|
| Delta cache hit | couchbase.sync_gateway.sgw_delta_sync_delta_cache_hit | The total number of requested deltas that were available in the revision cache. |
| Delta cache miss | couchbase.sync_gateway.sgw_delta_sync_delta_cache_miss | The total number of requested deltas that were not available in the revision cache. |
| Total deltas sent | couchbase.sync_gateway.sgw_delta_sync_deltas_sent | The total number of revisions sent to clients as deltas |
| Metric name | Metric key | Description |
|---|---|---|
| Documents indexed per second | couchbase.index_avg_drain_rate | Average number of documents indexed per second, for this index |
| Average size of index items | couchbase.index_avg_item_size | Average size of the indexed items |
| Uncompressed index data size | couchbase.index_data_size | The approximate size of the valid uncompressed index data |
| Index total disk space | couchbase.index_disk_size | Total disk space taken up by this index after compression. This includes index data files, checkpoints etc. |
| Records in memory ratio | couchbase.index_resident_percent | Ratio of records in memory to total records, for this index |
| Metric name | Metric key | Description |
|---|---|---|
| Queries > 500ms | couchbase.query.requests_500ms.count | Number of queries that take longer than 500ms. |
| Queries > 1000ms | couchbase.query.requests_1000ms.count | Number of queries that take longer than 1000ms. |
| Queries > 5000ms | couchbase.query.requests_5000ms.count | Number of queries that take longer than 5000ms. |
| N1QL Selects | couchbase.query.selects.count | Total number of N1QL SELECT requests. |
| N1QL Requests | couchbase.query.requests.count | Total number of N1QL requests. |
| N1QL Warnings | couchbase.query.warnings.count | Total number of N1QL warnings. |
| N1QL Invalid Requests | couchbase.query.invalid_requests.count | Total number of N1QL invalid requests. |
| N1QL Errors | couchbase.query.errors.count | Total number of N1QL errors. |
| Query Request Time | couchbase.query.request_time.count | End-to-end time to process all queries. |
| Query Execution Time | couchbase.query.service_time.count | Time to execute all queries. |
| — | couchbase.query.result_count.count | — |
| Query Result Size | couchbase.query.result_size.count | Total size of data returned by the query engine. |
| Metric name | Metric key | Description |
|---|---|---|
| Channel cache hits | couchbase.sync_gateway.sgw_cache_chan_cache_hits | The total number of channel cache requests fully served by the cache. |
| Channel cache largest | couchbase.sync_gateway.sgw_cache_chan_cache_max_entries | The total size of the largest channel cache. |
| Channel cache misses | couchbase.sync_gateway.sgw_cache_chan_cache_misses | The total number of channel cache requests not fully served by the cache. |
| Total channels cached | couchbase.sync_gateway.sgw_cache_chan_cache_num_channels | The total number of channels being cached. |
| Total active channels | couchbase.sync_gateway.sgw_cache_num_active_channels | The total number of active channels. |
| Revision cache hits | couchbase.sync_gateway.sgw_cache_rev_cache_hits | The total number of revision cache hits. |
| Revision cache misses | couchbase.sync_gateway.sgw_cache_rev_cache_misses | The total number of revision cache misses. |
| Metric name | Metric key | Description |
|---|---|---|
| XDCR replicated docs | couchbase.xdcr_docs_processed_total | Number of documents processed for a Replication |
| XDCR docs received | couchbase.xdcr_docs_received_from_dcp_total | Number of Document Mutations received from the Data Service |
| XDCR docs written | couchbase.xdcr_docs_written_total | Number of docs Document Mutations written/sent to the Target |
| XDCR write operations | couchbase.xdcr_set_docs_written_total | Number of Set operations successfully written to the Target |
| Metric name | Metric key | Description |
|---|---|---|
| CPU Utilization | couchbase.node.system.cpu_utilization_rate | CPU utilization rate |
| — | couchbase.node.system.cpu_stolen_rate | — |
| — | couchbase.node.system.swap_total | — |
| — | couchbase.node.system.swap_used | — |
| — | couchbase.node.system.mem_total | — |
| — | couchbase.node.system.mem_free | — |
| — | couchbase.node.system.mem_limit | — |
| — | couchbase.node.system.cpu_cores_available | — |
| — | couchbase.node.system.allocstall | — |
| — | couchbase.node.interesting.cmd_get | — |
| — | couchbase.node.interesting.couch_docs_actual_disk_size | — |
| — | couchbase.node.interesting.couch_docs_data_size | — |
| — | couchbase.node.interesting.couch_spatial_data_size | — |
| — | couchbase.node.interesting.couch_spatial_disk_size | — |
| — | couchbase.node.interesting.couch_views_actual_disk_size | — |
| — | couchbase.node.interesting.couch_views_data_size | — |
| — | couchbase.node.interesting.curr_items | — |
| — | couchbase.node.interesting.curr_items_tot | — |
| — | couchbase.node.interesting.ep_bg_fetched | — |
| — | couchbase.node.interesting.get_hits | — |
| — | couchbase.node.interesting.index_data_size | — |
| — | couchbase.node.interesting.index_disk_size | — |
| Memory Used | couchbase.node.interesting.mem_used | Memory used by the node |
| Total Operations | couchbase.node.interesting.ops | Total Operations per second to this bucket |
| Total swap | couchbase.node.interesting.swap_total | Total swap memory |
| Used swap | couchbase.node.interesting.swap_used | Used swap memory |
| — | couchbase.node.interesting.vb_active_num_non_resident | — |
| — | couchbase.node.interesting.vb_replica_curr_items | — |
| Cluster Uptime | couchbase.node.system.uptime | Uptime of the cluster |
| — | couchbase.node.interesting.cmd_set | — |
| — | couchbase.node.interesting.couch_docs_fragmentation | — |
| — | couchbase.node.interesting.couch_total_disk_size | — |
| — | couchbase.node.interesting.ep_cache_miss_rate | — |
| — | couchbase.node.interesting.ep_diskqueue_drain | — |
| — | couchbase.node.interesting.ep_diskqueue_fill | — |
| — | couchbase.node.interesting.ep_queue_size | — |
| — | couchbase.node.interesting.vb_active_resident_items_ratio | — |
| Metric name | Metric key | Description |
|---|---|---|
| Total active requests | couchbase.n1ql_active_requests | Total number of active requests. |
| Total N1QL requests with at_plus index consistency | couchbase.n1ql_at_plus.count | Total number of N1QL requests with at_plus index consistency. |
| Total audit records sent | couchbase.n1ql_audit_actions.count | The total number of audit records sent to the server. Some requests cause more than one audit record to be emitted. Records in the output queue that have not yet been sent to the server are not counted. |
| Total failed audit records sent | couchbase.n1ql_audit_actions_failed.count | The total number of audit records sent to the server that failed. |
| Potentially auditable requests with no audit action | couchbase.n1ql_audit_requests_filtered.count | The number of potentially auditable requests that cause no audit action to be taken. |
| Potentially auditable requests sent to query engine | couchbase.n1ql_audit_requests_total.count | The total number of potentially auditable requests sent to the query engine. |
| Cancelled requests | couchbase.n1ql_cancelled.count | Total number of cancelled requests. |
| Delete operations | couchbase.n1ql_deletes.count | Total number of DELETE operations. |
| N1QL error count | couchbase.n1ql_errors.count | The total number of N1QL errors returned so far. |
| Secondary index scans | couchbase.n1ql_index_scans.count | Total number of secondary index scans. |
| Insert operations | couchbase.n1ql_inserts.count | Total number of INSERT operations. |
| Invalid requests | couchbase.n1ql_invalid_requests.count | Total number of requests for unsupported endpoints. |
| Document mutations | couchbase.n1ql_mutations.count | Total number of document mutations. |
| Prepared statements executed | couchbase.n1ql_prepared.count | Total number of prepared statements executed. |
| Primary index scans | couchbase.n1ql_primary_scans.count | Total number of primary index scans. |
| Query process time | couchbase.n1ql_request_time.count | Total end-to-end time to process all queries. |
| N1QL requests | couchbase.n1ql_requests.count | Total number of N1QL requests. |
| Queries longer than 1000ms | couchbase.n1ql_requests_1000ms.count | Number of queries that take longer than 1000ms. |
| Queries longer than 250ms | couchbase.n1ql_requests_250ms.count | Number of queries that take longer than 250ms. |
| Queries longer than 5000ms | couchbase.n1ql_requests_5000ms.count | Number of queries that take longer than 5000ms. |
| Queries longer than 500ms | couchbase.n1ql_requests_500ms.count | Number of queries that take longer than 500ms. |
| Results (document) returned count | couchbase.n1ql_result_count | Total number of results (documents) returned by the query engine. |
| Result size | couchbase.n1ql_result_size.count | Total size of data returned by the query engine. |
| N1QL requests with request_plus consistency | couchbase.n1ql_scan_plus.count | Total number of N1QL requests with request_plus index consistency. |
| Select requests | couchbase.n1ql_selects.count | Total number of SELECT requests. |
| Time to execute queries | couchbase.n1ql_service_time.count | Time to execute all queries. |
| Elapsed transaction time | couchbase.n1ql_transaction_time.count | Total elapsed time of transactions so far. |
| Transaction count | couchbase.n1ql_transactions.count | Total number of transactions. |
| N1QL requests with not_bounded consistency | couchbase.n1ql_unbounded.count | Total number of N1QL requests with not_bounded index consistency. |
| Update requests | couchbase.n1ql_updates.count | Total number of UPDATE requests. |
| N1QL warnings | couchbase.n1ql_warnings.count | The total number of N1QL warnings returned so far. |
| Metric name | Metric key | Description |
|---|---|---|
| Documents imports | couchbase.sync_gateway.sgw_shared_bucket_import_import_count | The total number of docs imported. |
| Document import errors | couchbase.sync_gateway.sgw_shared_bucket_import_import_error_count | The total number of errors arising as a result of a document import. |
| Document import processing time | couchbase.sync_gateway.sgw_shared_bucket_import_import_processing_time | The total time taken to process a document import. |
| Metric name | Metric key | Description |
|---|---|---|
| Access count queries | couchbase.sync_gateway.sgw_gsi_views_access_count | The total number of queries performed. |
| Access count queries errors | couchbase.sync_gateway.sgw_gsi_views_access_error_count | The total number of errors that occurred when performing the query. |
| allDocs queries | couchbase.sync_gateway.sgw_gsi_views_allDocs_count | The total number of queries performed. |
| allDocs queries errors | couchbase.sync_gateway.sgw_gsi_views_allDocs_error_count | The total number of errors that occurred when performing the query. |
| cahnnelsStar queries | couchbase.sync_gateway.sgw_gsi_views_channelsStar_count | The total number of queries performed. |
| cahnnelsStar queries errors | couchbase.sync_gateway.sgw_gsi_views_channelsStar_error_count | The total number of errors that occurred when performing the query. |
| Channels queries | couchbase.sync_gateway.sgw_gsi_views_channels_count | The total number of queries performed. |
| Channels queries errors | couchbase.sync_gateway.sgw_gsi_views_channels_error_count | The total number of errors that occurred when performing the query. |
| Principals queries | couchbase.sync_gateway.sgw_gsi_views_principals_count | The total number of queries performed. |
| Principals queries errors | couchbase.sync_gateway.sgw_gsi_views_principals_error_count | The total number of errors that occurred when performing the query. |
| Resync queries | couchbase.sync_gateway.sgw_gsi_views_resync_count | The total number of queries performed. |
| Resync queries errors | couchbase.sync_gateway.sgw_gsi_views_resync_error_count | The total number of errors that occurred when performing the query. |
| roleAccess queries | couchbase.sync_gateway.sgw_gsi_views_roleAccess_count | The total number of queries performed. |
| roleAccess queries errors | couchbase.sync_gateway.sgw_gsi_views_roleAccess_error_count | The total number of errors that occurred when performing the query. |
| Sequences queries | couchbase.sync_gateway.sgw_gsi_views_sequences_count | The total number of queries performed. |
| Sequences queries errors | couchbase.sync_gateway.sgw_gsi_views_sequences_error_count | The total number of errors that occurred when performing the query. |
| Sessions queries | couchbase.sync_gateway.sgw_gsi_views_sessions_count | The total number of queries performed. |
| Sessions queries errors | couchbase.sync_gateway.sgw_gsi_views_sessions_error_count | The total number of errors that occurred when performing the query. |
| Tombstones queries | couchbase.sync_gateway.sgw_gsi_views_tombstones_count | The total number of queries performed. |
| — | couchbase.sync_gateway.sgw_gsi_views_tomestones_error_count | — |
| Users queries | couchbase.sync_gateway.sgw_gsi_views_users_count | The total number of queries performed. |
| Users queries errors | couchbase.sync_gateway.sgw_gsi_views_users_error_count | The total number of errors that occurred when performing the query. |
| Metric name | Metric key | Description |
|---|---|---|
| CPU Utilization (Deprecated in CB v7.6.0) | couchbase.sysproc_cpu_utilization | CPU utilization rate |
| Resident memory used | couchbase.sysproc_mem_resident | Amount of resident memory used |
| Memory used | couchbase.sysproc_mem_size | Amount of memory used |
| CPU Utilization | couchbase.sys_cpu_utilization_rate | CPU utilization rate |
| Metric name | Metric key | Description |
|---|---|---|
| Admin bytes received | couchbase.sync_gateway.sgw_resource_utilization_admin_net_bytes_recv.count | The total number of bytes received (since node start-up) on the network interface to which the Sync Gateway api.admin_interface is bound. |
| Admin bytes sent | couchbase.sync_gateway.sgw_resource_utilization_admin_net_bytes_sent.count | The total number of bytes sent (since node start-up) on the network interface to which the Sync Gateway api.admin_interface is bound. |
| Error count | couchbase.sync_gateway.sgw_resource_utilization_error_count | The total number of errors logged. |
| Go routine high watermark | couchbase.sync_gateway.sgw_resource_utilization_goroutines_high_watermark | Peak number of go routines since process start. |
| Total number of goroutines | couchbase.sync_gateway.sgw_resource_utilization_num_goroutines | The total number of goroutines. |
| CPU utilization | couchbase.sync_gateway.sgw_resource_utilization_process_cpu_percent_utilization | The CPU’s utilization as percentage value. Based on user and system CPU time, but it doesn’t include components such as iowait. |
| Memory utilization | couchbase.sync_gateway.sgw_resource_utilization_process_memory_resident | The memory utilization (Resident Set Size) for the process, in bytes |
| Public bytes received | couchbase.sync_gateway.sgw_resource_utilization_pub_net_bytes_recv.count | The total number of bytes received (since node start-up) on the network interface to which the Sync Gateway api.public_interface is bound. |
| Public bytes sent | couchbase.sync_gateway.sgw_resource_utilization_pub_net_bytes_sent.count | The total number of bytes sent (since node start-up) on the network interface to which Sync Gateway api.public_interface is bound. |
| Total available memory | couchbase.sync_gateway.sgw_resource_utilization_system_memory_total | The total memory available on the system in bytes. |
| Uptime | couchbase.sync_gateway.sgw_resource_utilization_uptime_up | The total uptime. |
| Metric name | Metric key | Description |
|---|---|---|
| Total RAM | couchbase.cluster.ram.total | Total RAM available in the cluster |
| Used RAM | couchbase.cluster.ram.used | Total RAM used in the cluster |
| Total RAM Quota | couchbase.cluster.ram.quotaTotal | Total RAM quota available in the cluster |
| Used RAM Quota | couchbase.cluster.ram.quotaUsed | RAM quota used in the cluster |
| Total HDD | couchbase.cluster.hdd.total | Total HDD available in the cluster |
| Used HDD | couchbase.cluster.hdd.used | HDD used by the cluster |
| Total HDD Quota | couchbase.cluster.hdd.quotaTotal | Total HDD quota available in the cluster |
| Used HDD Quota | couchbase.cluster.hdd.quotaUsed | HDD quota used by the cluster |
| Metric name | Metric key | Description |
|---|---|---|
| DCP mutations added | couchbase.sync_gateway.sgw_database_dcp_caching_count | The total number of DCP mutations added to Sync Gateway’s channel cache. |
| DCP mutation time | couchbase.sync_gateway.sgw_database_dcp_caching_time | The total time between a DCP mutation arriving at Sync Gateway and being added to channel cache. |
| DCP received time | couchbase.sync_gateway.sgw_database_dcp_received_time | The time between a document write and that document being received by Sync Gateway over DCP. If the document was written prior to Sync Gateway starting the feed, it is recorded as the time since the feed was started. |
| Active replications | couchbase.sync_gateway.sgw_database_num_replications_active | The total number of active replications. |
| Total replications | couchbase.sync_gateway.sgw_database_num_replications_total | The total number of replications created since Sync Gateway node startup. |
| Channel name size warnings | couchbase.sync_gateway.sgw_database_warn_channel_name_size_count | The total number of warnings relating to the channel name size. |
| Channel count threshold exceeded | couchbase.sync_gateway.sgw_database_warn_channels_per_doc_count | The total number of warnings relating to the channel count exceeding the channel count threshold. |
| Grant count threshold exceeded | couchbase.sync_gateway.sgw_database_warn_grants_per_doc_count | The total number of warnings relating to the grant count exceeding the grant count threshold. |
| Xattr sync data threshold exceeded | couchbase.sync_gateway.sgw_database_warn_xattr_size_count | The total number of warnings relating to the xattr sync data being larger than a configured threshold. |
| Metric name | Metric key | Description |
|---|---|---|
| Index Mutations Remaining | couchbase.index.num_docs_pending | Number of documents remaining to be indexed. |
| Index Drain Rate | couchbase.index.avg_drain_rate | Number of documents indexed by the indexer |
| Index Resident Percent | couchbase.index.resident_percent | Percentage of index data resident in memory |
| Index RAM Used | couchbase.index.memory_used | Bytes in memory for this index |
| Indexed items | couchbase.index.items_count.gauge | Current total indexed documents |
| Index Data Size | couchbase.index.data_size | Total data in index |
| Index Disk Size | couchbase.index.disk_size | Total data on disk for index |
| Index Item Size | couchbase.index.avg_item_size | Average size of each item in Index |
| Index Scan Latency | couchbase.index.avg_scan_latency | Average time to serve a scan request |
| Index Requests | couchbase.index.num_requests | Number of requests served by the indexer |
| Index Scan Bytes | couchbase.index.scan_bytes_read | Number of bytes scanned by the indexer |
| Index Fragmentation | couchbase.index.frag_percent | Percentage fragmentation of the index |
| Index Cache Hit Percent | couchbase.index.cache_hit_percent | Percentage of access to index data from RAM as opposed to disk |
| Items scanned | couchbase.index.num_rows_returned.count | Number of index items scanned by the indexer per second. |
| Number of documents indexed | couchbase.index.num_docs_indexed.count | Number of documents indexed by the indexer since last startup. |