Learn how to troubleshoot OneAgent installation on AIX, Linux, and Windows.
Following installation of OneAgent, your Apache web server must be completely restarted to enable monitoring. To do this correctly, it's important to understand the difference between "partial" and "complete" restarts. In the case of partial restarts, the main Apache process re-reads its configuration files, re-opens its log files, and then restarts its worker processes. OneAgent however, requires a complete Apache web server restart in which all workers and—most importantly—the main Apache process are shut down entirely and then restarted.
See Stopping and Restarting Apache HTTP Server for more information on the different types of available restarts.
Linux and AIX
You may be accustomed to restarting Apache by issuing an apachectl restart command. However, this command only results in a partial Apache restart.
To execute a complete Apache restart and enable deep monitoring with Dynatrace OneAgent, you need to first invoke a complete shutdown using the apachectl stop command. Only following this step can you restart the server using apachectl start .
It's fine to use service apache2 restart on Ubuntu systems. Note however that whatever commands you use, you'll likely need superuser rights (sudo).
Windows
On Windows, you can either use the built-in Windows Service Management or Apache Service Monitor (httpd.exe) to restart Apache services. Restarting the Apache service with Windows Service Management guarantees the complete restart. With httpd.exe, you may be accustomed to restarting Apache by issuing a httpd.exe -k restart -n "Apache2.4" command. However, this command only results in a partial Apache restart.
To execute a complete Apache restart and enable deep monitoring with OneAgent, you need to first invoke a complete shutdown using the httpd.exe -k stop -n "Apache2.4" command. Only following this step can you restart the server using httpd.exe -k start -n "Apache2.4".
deprecated
Starting with OneAgent version 1.301, OneAgent doesn't use the TCP ports for its own inter-process communication. In case OneAgent occupies your applications' ports, upgrade OneAgent to version 1.301+.
OneAgent consists of different processes that communicate via a TCP port with a watchdog. At startup, OneAgent watchdog attempts to open the first available port between port 50000 and 50100. In some cases you may need this port for your own applications that are started after OneAgent. In such cases, you can change the port range that the OneAgent watchdog uses by calling OneAgent command-line interface.
You can change the watchdog listening port by using set-watchdog-portrange via oneagentctl command-line tool :
For example, to change port range to 50005:50105, go to the oneagentctl directory and run the following command:
./oneagentctl --set-watchdog-portrange 50005:50105.\oneagentctl.exe --set-watchdog-portrange 50005:50105See Which network ports does Dynatrace Server use? for information on the ports used by Dynatrace.
OneAgent is shipped with trusted Dynatrace SSL certificates, which are used to verify that OneAgent connects successfully to Dynatrace Server or ActiveGate.
If your environment uses a proxy (thereby requiring an update to the remote server's SSL certificate), then you may encounter a Server certificate check failed message during the initial connection check.
To resolve this issue
Identify the SSL certificate that your proxy uses.
Save this certificate as custom.pem.
Place the custom.pem file in the appropriate directory based on your operating system:
Linux: /var/lib/dynatrace/oneagent/agent/customkeys
Windows: %PROGRAMDATA%\dynatrace\oneagent\agent\customkeys
Make sure the custom.pem file contains the proxy’s certificate along with any intermediate certificates as required.
For HTTPS proxy servers requiring custom certificates (for example, between OneAgent and ActiveGate), you need to add the necessary certificates to the custom-proxy.pem file. This file should be located in the standard certificate directory for your operating system (the same as in the procedure above).
If your setup involves both an HTTPS proxy with a custom certificate and an environment ActiveGate with its own custom certificate, you need to provide a custom.pem file and a custom-proxy.pem file.
One of the following may have occurred
If you're still unable to resolve this issue, contact a Dynatrace product expert via live chat within your Dynatrace environment. Also, consider installing OneAgent on a different machine.
We do not support major operating system modifications to a host on which OneAgent is installed.
OS changes that can affect OneAgent include updates or modifications such as:
Major OS modifications may lead to problems such as:
To make major OS modifications to a OneAgent host
This information applies to all operating systems on which full-stack OneAgent installation is supported.
If the SDK stub encounters issues loading or initializing the OneAgent module (particularly if onesdk_initialize or onesdk_initalize_2 returns an error code), enable logging for the SDK stub to diagnose the problem.
Use one of these options to enable logging:
DT_LOGLEVELSDK={level} environment variable (the easiest option).onesdk_stub_set_logging_level(ONESDK_LOGGING_LEVEL_{LEVEL}) function.onesdk_stub_process_cmdline_args), use the --dt_loglevelsdk={level} command line argument.Whichever option you choose, be sure to apply it before calling onesdk_initialize or onesdk_initalize_2.
By default, after logging is enabled, the stub's log output is directed to stderr. If you need an alternative method to process stub log messages, see the onesdk_stub_set_logging_callback function documentation.
If initialization fails, the most frequently encountered error code is ONESDK_ERROR_LOAD_AGENT (numerical code 2952658951, -1342308345 or 0xaffe0007, error message "Could not load agent.").
The two primary causes of this issue are:
Cause: OneAgent is not installed on the host where the program is executed.
Solution: Install OneAgent and restart the program.
Cause: The program is initiated with a debugger, so OneAgent will not inject.
Solution: Launch the program without the debugger. You can still attach the debugger later, after the program is running.
After successfully initializing the SDK, you might still encounter issues, such as missing paths in the UI or unexpected error codes like ONESDK_INVALID_HANDLE. In such cases:
Check messages from the OneAgent logging callbacks. See the documentation for onesdk_agent_set_warning_callback and onesdk_agent_set_verbose_callback.
Examine the OneAgent log files.
See the following pages for exact locations of log files:
You can increase the OneAgent log level by setting the DT_LOGLEVELFILE={level} environment variable or passing the --dt_loglevelfile={level} command line argument to the SDK.
Alternatively, you can use DT_LOGLEVELCON={level} or --dt_loglevelcon={level} if you want to receive OneAgent log output via stderr.
In certain scenarios, onesdk_agent_get_current_state can provide further insights.
Even though you can successfully install OneAgent on machines hosting services deployed by Chef Habitat, OneAgent won't be able to inject into the processes, because in such deployments, the Chef Habitat uses its own supervisor host-specific glibc, not the system-level glibc OneAgent relies on.
As a workaround, create an ld.so.preload file for each glibc version installed by Chef whose content points to the OneAgent Process Module on the Chef Habitat supervisor host. Run the following command as root:
[ -d /hab/pkgs/core/glibc ] && for v in $(find /hab/pkgs/core/glibc -type d -name etc); do sudo echo "/opt/dynatrace/oneagent/agent/bin/current/linux-x86-64/liboneagentproc.so" > $v/ld.so.preload && echo "Installed workaround in '$v'"; done
You must run this command every time Chef Habitat updates the glibc version. You can also run it as a cron job. In such cases, make sure it runs before the start of the service you want to monitor. Otherwise, you'll need to restart the service to enable OneAgent injection.
If you see an Operation not permitted error in the Linux console or the installation logs, make sure that OneAgent installation isn't blocked by antivirus software installed on the host.
OneAgent supports SELinux only when the targeted policy is loaded, the multi-level security policy is not supported. If you attempt to install OneAgent on a system where SELinux with multi-level security mode policy, you will get the following error message: Installation with SELinux loaded in multi-level security mode is not supported. Dynatrace OneAgent may not work correctly.
If you are using a system with SELinux in enforcing mode and injected OneAgents are failing to communicate, yet communication works just fine for the OneAgent OS module, try the following actions. Note that the example below is based on the httpd process, but this can also happen for NGINX and other processes.
Check /var/log/audit/audit.log or journalctl for denials, for example:
# grep type=AVC /var/log/audit/audit.log# journalctl --utc -a -t "audit"
If you find a denial for the process in question, for example:
type=AVC msg=audit(1535366769.867:209537): avc: denied { name_connect } for pid=8348 comm="httpd" dest=9999 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:jboss_management_port_t:s0 tclass=tcp_socket`
first, check if SElinux allows the communication using the following command:
# sesearch -AC -s httpd_t -t jboss_management_port_t
To interpret the command output, see Using SELinux booleans.
To find out if the communication is not allowed, execute the following command:
# setsebool -P httpd_can_network_connect on
The command will persistently (retained across host reboots) enable the httpd_can_network_connect SELinux boolean, allowing OneAgent to be injected into the httpd process to establish connection to ActiveGate.
Restart the process and verify that the communication works.
OneAgent on Linux was reported to be unstable when deployed on poor quality NFS drives. In order for the automatic injection and automatic updates to work properly, ensure that your OneAgent deployment meets the following recommendations:
Customize OneAgent installation so that it's not located in the NFS directories (the default OneAgent location is /opt/dynatrace). Use the OneAgent INSTALL_PATH parameter. For more information, see Customize OneAgent installation on Linux.
Ensure that the runtime path /var/lib/dynatrace/oneagent is not located in NFS directories.
Filesystem availability is critical not only for OneAgent monitoring, but also for startup of any processes on the host. Even if you customize OneAgent installation, it still creates symbolic links to /opt/dynatrace for its deep monitoring modules and automatic injection. Make sure that /opt/dynatrace is available during system startup as early as possible. OneAgent starts up relatively early, so /opt/dynatrace needs to be available as early in the startup process as possible.
If, in the presence of NFS, you observe OneAgent update problems, make sure to stop any processes that may have OneAgent deep code monitoring modules enabled before you start OneAgent update.
Any file system that relies on FUSE is not supported.
We're working on fixing the NFS-deployment-related issues, so you can expect these guidelines to evolve over time.
The splunkd component of Splunk version 8.2+ crashes when OneAgent automatic injection is enabled.
According to Splunk issue SPL-207550 (external link), Splunk fails to start after installation on Linux if a Dynatrace OneAgent exists, with error ERROR: pid XXXX terminated with signal 4 (core dumped), because there is a conflict between the splunk watchdog and Dynatrace OneAgent libraries.
Set the following in server.conf, [watchdog]:
usePreloadedPstacks = false
Blocking mutex in the Linux kernel can cause CrowdStrike Falcon to block OneAgent when reading the process data from /proc, which contains one subdirectory per process running on the system.
/proc/<pid>, CrowdStrike Falcon blocks mutex in the kernel for process ID directory creation. OneAgent will be in uninterruptible state, which means you can clear the processes only by rebooting the server or waiting for the I/O to respond.OneAgent auto instrumentation is not possible when Oracle Database Server 19c is installed, due to incompatibility with OneAgent Process Module.
OneAgent Process Module requires basic functionality from system libc library to perform auto instrumentation. When another product overrides the functionality (__errno_location function in this case), the Process Module is unable to distinguish between the symbol provided by libc and the product. Calling the symbol provided by the product results in a crash, as it is not yet initialised at the time.
Linux hosts with Oracle Database Server 19c
Either of the options is sufficient:
builtin:host.monitoring.advanced schema / UIoneagentctl --set-auto-injection-enabled falseIf there are any processes on the hosts which require Code Module injection, they can be manually enabled via LD_PRELOAD=/lib{64}/liboneagentproc.so environment variable. For further assistance, reach out to Dynatrace support specialists in chat.
To ensure seamless functioning of OneAgent and to avoid overhead, we recommend excluding all files in the OneAgent installation directory from antivirus scan.
We also recommend that you configure your antivirus software so that it treats the OneAgent process as trusted and non-malicious. Refer to the documentation of your antivirus solution to learn how to do this.
If you're using McAfee, you may experience CPU overhead. To resolve this issue, set McAfee to Exploit Prevention Compatibility Mode.
HKEY_LOCAL_MACHINE\SOFTWARE\McAfee\Endpoint\Ips\BO\dwBOCompatibilityMode=1HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\McAfee\EndPoint\Ips\BO\dwBOCompatibilityMode=1One of the following may have occurred
If you're still unable to resolve this issue, contact a Dynatrace product expert via live chat within your Dynatrace environment. Also, consider installing OneAgent on a different machine.
The OneAgent Windows installer doesn't support the modify and repair operations. You can't reinstall OneAgent using the same installer version as the installed OneAgent.
To reinstall OneAgent on Windows, either uninstall it and then reinstall it, or install a newer version over the existing version.
OneAgent Windows installer utilizes the Windows Installer Cache, which is located by default at C:\Windows\Installer. It stores important files required for uninstalling and updating the product. If you encounter log entries similar to the following in the installation log (default location: C:\ProgramData\dynatrace\oneagent\log\installer\installation_msiexec_*.log):
MSI (s) (C0:E4) [09:27:14:308]: Warning: Local cached package 'C:\Windows\Installer\312c0.msi' is missing....Error 1714. The older version of Dynatrace OneAgent cannot be removed. Contact your technical support group. System Error 1612.MSI (s) (C0:54) [09:27:56:489]: Product: Dynatrace OneAgent -- Error 1714. The older version of Dynatrace OneAgent cannot be removed. Contact your technical support group. System Error 1612.
Try the following steps to resolve the issue:
C:\Windows\Installer, and rename it to match the name referenced in the log (in this example, 312c0.msi).For more information, see Missing Windows Installer cache requires a computer rebuild.
This is a known issue with Advanced Installer. As a workaround, the OneAgent installer clears the AI_RecycleBin at the end of the installation. However, this cleanup might not work if the installation fails in an early stage, such as in the case of a missing MSI package in Windows Installer Cache. For more details, see the discussion on Advanced Installer forums.
If you used OneAgent for AIX prior to version 1.137 you may have configured it via JAVA_OPTS using the dynatrace-java-env.sh script. You need to remove this prior to using the unified monitoring scripts dynatrace-agentXX.sh.
Make sure that dynatrace-java-env.sh isn't called anywhere in your shell when you use the dynatrace-agentXX.sh script.
dynatrace-java-env.sh is deprecated and should only be used as a fallback.
Check for and remove the following parameter from your Java command line or startup scripts (specific directory may vary):
-agentpath:/opt/dynatrace/oneagent/agent/lib64/liboneagentloader.so
You may encounter an error like this when you use dynatrace-agentXX.sh in a shell script.
Info: using DT_HOME: /opt/dynatrace/oneagent.profile[33] LDR_PRELOAD64: parameter not set
This happens if you use set -u to treat unset variables and parameters as errors. The dynatrace-agentXX.sh script exports variables which, though they may not yet exist in your script, are nevertheless needed and key to proper operation. To overcome this, call set +u ahead of the dynatrace-agentXX.sh script.
# avoid errorset +uexport DT_HOME=/opt/dynatrace/oneagent. $DT_HOME/dynatrace-agent64.sh