Remote Unix Monitoring 2.0 extension
- Latest Dynatrace
- Extension
Extension that remotely collects Unix OS data by executing commands via SSH.
Get started
Overview
Unix/Linux is often used to run critical applications. A OneAgent is the best way to monitor such systems, however this is not always possible. Legacy unsupported versions or agreements with vendors may prevent you from installing a OneAgent on these important hosts. In these cases you can use the Remote Unix Monitoring extension to collect valuable telemetry on your hosts and the applications they are running. It works by connecting to your Unix/Linux hosts over SSH and running commands.
Use cases
- Monitor and alert on important host metrics
- Monitor the availability and resource usage of your key processes and applications
- Identify host availability issues
Requirements
- Minimum Dynatrace environment and ActiveGate version 1.269
- An OS in the list of the supported distributions
- SSH enabled and reachable from assigned ActiveGates
- A user with permission to connect and run the required commands
- No commands used require root/sudo privileges so as a security best-practice we advise against using a root account or a user with sudo privileges. A normal user is adequate and recommended.
- Either a valid password or key for authentication
Compatibility information
The following OSs are explicitly supported and can be selected in the configuration:
- AIX
- Centos
- FreeBSD
- HP-UX
- MacOS
- Oracle Linux
- Red Hat Enterprise Linux (RHEL)
- Solaris
- Suse
Many other distributions such as Ubuntu can be monitored by selecting 'Debian Linux.'
As a goal of this extension is to provide visibility into systems that cannot be monitored by a OneAgent it does not have the same support policy as other areas of Dynatrace where we support what the vendor supports. Rather we try to support as many distributions as possible regardless of their support status to the best of our ability.
Activation and setup
Dynatrace configuration
Find 'Remote Unix Monitoring 2.0' in the in-product Extensions or Hub page and activate (if offline you can download the extension from this Hub page in the 'Versions' section and install as a custom extension).
Extension monitoring configurations
Once activated in your environment you can create monitoring configurations. Each monitoring configuration can have one or more remote unix target hosts configured.
First select the desired ActiveGate group that will run the monitoring configuration.
For each cluster configure a Remote Unix Endpoint:
- Hostname: hostname or IP address to connect to
- Port: port where the SSH server is listening (default 22)
- Host alias: an optional alias for the target host, defaults to hostname configured
- Operating system
- Authentication
- Password: supports credential vault
- Key contents: SSH private key contents as copy-pasted from key file
- Key path: a path to a key file. Must be present and readable on any ActiveGate to work
- Additional properties: additional properties to be added as dimensions to ingested metrics
- Top process count: when reporting top process metrics how many should be collected (default 10)
- Report as log events: report more detailed top process data as log events
- Process filters: a set of process filters to collect filtered process data for
- Group key: process group to which matches will be assigned
- Pattern: regex to match the process commands
- User: optional user filter
- Mount filters: filters to control which mounts to report metrics for
- Advanced settings
- Custom PATH: if the default non-login path does not have all required commands, specify a custom PATH to be used when looking for commands
- SSH connection behavior: disconnect on each execution or reuse between runs and keep open
- RSA2 algorithm: some older systems require RSA2 to be disabled. See here for details.
- Top mode
- Metrics that require top for process level metrics will be run in threads mode (-H) and report metrics at the process thread level
- Can be used if in multi-processor environments percentages of CPU greater than 100% are reported
- Max concurrent channels: Determines number of simultaneous channels that can be open over SSH connection. Default and Max: 5.
- Some systems may experience channels errors due to simultaneous channels being used. In these cases you can try lowering this value.
- Log command output: command output will be recorded in debug logs to assist with troubleshooting. DEBUG logging must be enabled.
Details
Metrics
Review the list of feature sets to see which metrics are collected. Note that for some distributions different metrics/metric keys may be used for some types of metrics due to similar but not identical metrics being available on that distribution as compared to what is more commonly collected.
Log events
Top process data
When configured, details about the 'top' processes running over time will be ingested as log events. These will include more complete data about the process (including the process ID) as compared to what the metrics alone will provide.
Filtered process PID change
When monitoring a group of filtered processes, if the set of PIDs in that group changes an event will be reported.
Connection issues
An availability event will be reported in cases of connection issues to the target host.
Licensing and costs
There is no charge to use the extension. You are only charged for the data that the extension ingests.
The Remote Unix Monitoring 2.0 extension ingests custom metrics, which consume Davis Data Units (DDUs) (Dynatrace classic license) or Metrics powered by Grail (DPS), according to your license model.
Assuming all feature sets are enabled, the approximate number of metric data points per minute is:
21 + (4 * <number of CPUs>) + (2 * <matching_filtered_processes>) + (2 * <process_groups>) + (3 * <matching_mounts>) + (8 * <network_interfaces>) + (2 * <disks>)
- See description of 'Process filter' configuration for explanation of what a group and filtered processes will be.
- Matching mounts will be those that meet criteria of 'Mounts to include' and 'Mounts to exclude' configurations.
Log records will be ingested for top process records, if enabled, based on the configured number of top process records once per minute. Each custom event reported will also consume license, based on the frequency of things like connection errors where an availability event is reported.
Dynatrace Platform Subscription
In the Dynatrace Platform Subscription, metric ingestion consumes Metrics powered by Grail according to the number of ingested metric data points.
To calculate the approximate yearly consumption, apply the following calculation: <metric data points per minute> * 60 minutes * 24 hours * 365 days.
For logs, regular consumption applies. See Log Analytics.
Dynatrace classic license
In the classic licensing model, metric ingestion consumes Davis Data Units (DDUs) at the rate of .001 DDUs per metric data point. Multiply the above formula for annual data points by .001 to estimate annual DDU usage.
For logs, regular DDU consumption applies. See DDU consumption for Log Management and Analytics or DDUs for Log Monitoring Classic.
The DDU cost above does not include any possible log events or custom events that are triggered by the extension. For more information, see DDU events.
Feature sets
Review the available feature sets to determine which you want to enable and collect. The enabled feature sets will effect which commands are run to avoid running commands that are not needed.
When activating your extension using a monitoring configuration, you can limit monitoring to one of the feature sets. To work properly, the extension has to collect at least one metric after the activation.
In highly segmented networks, feature sets can reflect the segments of your environment. Then, when you create a monitoring configuration, you can select a feature set and a corresponding ActiveGate group that can connect to this particular segment.
All metrics that aren't categorized into any feature set are considered to be the default and are always reported.
A metric inherits the feature set of a subgroup, which in turn inherits the feature set of a group. Also, the feature set defined on the metric level overrides the feature set defined on the subgroup level, which in turn overrides the feature set defined on the group level.
default
| Metric name | Metric key | Description |
|---|---|---|
| Availability | remote_unix.availability | Availability of host based on initial SSH connection |
| Waiting processes | remote_unix.waiting_processes | Number of processes waiting for processor time |
| User count | remote_unix.users | Number of users on the system |
| Load average (1 min) | remote_unix.load_avg_1_min | One minute load average |
| Load average (5 min) | remote_unix.load_avg_5_min | Five minute load average |
| Load average (15 min) | remote_unix.load_avg_15_min | 15 minute load average |
network
| Metric name | Metric key | Description |
|---|---|---|
| Network bytes receieved | remote_unix.network_bytes_received_count | Network traffic (in bytes) incoming per interface |
| Network packets received | remote_unix.packets_received_count | Network traffic (in Packets) incoming per interface |
| Network errors incoming | remote_unix.network_errors_incoming_count | Network errors on incoming traffic per interface |
| Packets dropped incoming | remote_unix.packets_dropped_incoming_count | Incoming packets dropped per interface |
| Network bytes sent | remote_unix.network_bytes_sent_count | Outbound network traffic (in bytes) per interface |
| Network packets sent | remote_unix.packets_sent_count | Outbound network traffic (in packets) per interface |
| Network errors outgoing | remote_unix.network_errors_outgoing_count | Network errors on outgoing traffic per interface |
| Packets dropped outgoing | remote_unix.packets_dropped_outgoing_count | Outgoing packets dropped per interface |
disks-extended
| Metric name | Metric key | Description |
|---|---|---|
| Disk latency (average) | remote_unix.disk_average_latency | Disk latency (average) |
solaris-zpools
| Metric name | Metric key | Description |
|---|---|---|
| Zpool size | remote_unix.zpool_size | Zpool total size |
| Zpool allocated | remote_unix.zpool_allocated | Zpool space allocated |
| Zpool free | remote_unix.zpool_free | Zpool space free |
| Zpool capacity | remote_unix.zpool_capacity | Zpool used percent |
cpu
| Metric name | Metric key | Description |
|---|---|---|
| CPU Utilization | remote_unix.cpu_utilization | Overall CPU utilization |
| CPU User | remote_unix.cpu_user | Percentage of time the processor spent executing user/application code |
| CPU System | remote_unix.cpu_system | Percentage of time the processor spent executing system/kernel |
| CPU Idle | remote_unix.cpu_idle | Percentage of time the processor was idle |
| Individual CPU user time | remote_unix.individual_cpu_time_user | Percentage of time individual CPU spent executing user/application code |
| Individual CPU system time | remote_unix.individual_cpu_time_system | Percentage of time individual CPU spent executing system/kernel code |
| Individual CPU idle time | remote_unix.individual_cpu_time_idle | Percentage of time individual CPU was idle |
| Individual CPU iowait time | remote_unix.individual_cpu_time_iowait | Percentage of time individual CPU spent waiting on IO operations |
mounts
| Metric name | Metric key | Description |
|---|---|---|
| Mount used | remote_unix.mount_used | Mount space in use |
| Mount capacity | remote_unix.mount_capacity | Overall capacity of the mount |
| Mount available | remote_unix.mount_available | Mount space available |
disks
| Metric name | Metric key | Description |
|---|---|---|
| Disk read | remote_unix.disk_read_count | Amount of data read from disk |
| Disk write | remote_unix.disk_write_count | Amount of data written to disk |
| Disk read operations | remote_unix.disk_read_ops | Reads from disk |
| Disk write operations | remote_unix.disk_write_ops | Writes to disk |
| Bytes per transfer | remote_unix.bytes_per_transfer | Bytes per transfer |
| Disk transfers | remote_unix.transfers | Disk transfers |
memory
| Metric name | Metric key | Description |
|---|---|---|
| Paged in | remote_unix.paged_in | Memory pages swapped in from disk |
| Paged out | remote_unix.paged_out | Memory pages swapped to disk |
| Physical memory free | remote_unix.physical_memory_free | Amount of memory free as seen by OS |
| Physical memory used percent | remote_unix.physical_memory_used_percent | Percentage of memory used as seen by OS |
| Swap free | remote_unix.swap_free | Amount of swap space free |
| Swap total | remote_unix.swap_total | Total swap space available on system |
| Swap used percent | remote_unix.swap_used_percent | Percentage of available swap space in use |
| Swap used percent | remote_unix.swap_free_percent | Percentage of available swap space not in use |
top-processes
| Metric name | Metric key | Description |
|---|---|---|
| Top process CPU usage | remote_unix.top_process_cpu | CPU usage of a 'top' resource consuming detected process |
| Top process memory usage | remote_unix.top_process_size | Memory usage of a 'top' resource consuming detected process |
FAQ
Known issues
Delinea/Centrify on Solaris: there is a known issue where Delinea (formerly Centrify) can incorrectly fail password-based authentication on Solaris hosts even when a known good password is used. This issue is specifically related to the 'password' SSH authentication method. As a workaround you can disable the password authentication mechanism via the target server's sshd_config file (see snippet below). This will cause the extension to authenticate using the 'keyboard-interactive' method which should work. In this approach the extension and manual users can still use a password to authenticate but the authentication exchange is different and avoids the issue.
# To disable tunneled clear text passwords, change PasswordAuthentication to no.PasswordAuthentication no
Required commands
Commands used are based on what is available in a 'vanilla' installation of the given OS, so typically no additional installs will be required.
Required commands must be available on the 'PATH' set for non-login shells, which can be different from login shells which is what you use when manually SSHing into a server. If you see log entries that commands are missing check that the command is available and that it is in the 'PATH' for non-login shells. If not, you can either correct this via configuration on the host or use the custom path configuration field to manually set a path to use.
You can test commands in non-login shells by running the ssh command with the command to run appended:
ssh <host> <command>
To view the path you can try:
ssh <host> 'echo $PATH'
For reporting the path it is important to use single quotes. Double quotes may cause the environment variable to show the value on your local system rather than the value on the remote host.
Debian Based Linux (Generic Linux)
- vmstat
- w
- df
- cat
- top
- pgrep
- iostat
- for Red Hat Enterprise Linux systems you may need to install the sysstat package to collect disk IO metrics. No alternatives that exist by default were found.
- ip or netstat
- older OSs don't have required stats in 'ip' command and will 'fall back' to netstat
Solaris
- vmstat
- w
- df
- netstat
- ps
- grep
- prstat
- kstat
- mpstat
- iostat
- swap
- prtconf
AIX
- vmstat
- w
- df
- netstat
- ps
- grep
- svmon
- mpstat
- iostat
FreeBSD
- vmstat
- w
- df
- netstat
- ps
- top
- grep
- iostat
HP-UX
- vmstat
- w
- df
- netstat
- ps
- grep
- swapinfo
- top
- sar
- machinfo
Extension Logs
For some authentication errors you may need to check authentication/access logs on the target host for details as to why a connection is failing.
Debug logging should only be set when necessary and then reverted to avoid excessive logging.
