Install the zDC subsystem
The z/OS Data Collection (zDC) subsystem interacts with the CICS, IMS, and z/OS Java modules via shared memory object (SMO) on a LPAR. The zDC subsystem maintains this SMO, and the modules write their monitoring data to it.
The zLocal (libdtzagent.so
), hosted in the z/OS Unix System Services (USS) environment, runs as part of the zDC. It manages the TCP/IP socket connection to the zRemote module, reads monitoring data from SMO, and transfers these data to the zRemote.
The bootstrapper (dtzagent
), hosted in the z/OS USS environment, runs as part of the zDC. It manages the update process of the zLocal.
When the zDC initializes (system startup or manual start), it starts the bootstrapper, and the bootstrapper starts the zLocal. When the zDC terminates, the zLocal and the bootstrapper are stopped.
System security
The zDC (zLocal) uses TCP/IP socket connections, so it may be necessary to update your security system's access rules if they are configured to deny TCP/IP access by default.
The zDC doesn't limit which user IDs can enter operator commands.
You can control permissions assigned to files created by the zLocal and bootstrapper by setting DT_UMASK
. The default is umask(022)
.
Installation
The zDC runs as an authorized z/OS process (typically as a system task). This means the programs must reside in an authorized library. It is intended to start automatically as a started task at a system IPL. This gives the zDC continuous availability to collect monitoring data from modules.
The zDC can also be run as a batch job. The service class of the zDC must be high enough so that it is always available for keep-alive messages from the zRemote. The zDC should have a priority equal to or higher than the monitored CICS and IMS regions.
A zDC subsystem must be installed on each LPAR you want to monitor.
-
Download z/OS product datasets. Make a note of the high-level qualifier used in the download procedure.
-
Create the RACF user ID for the zDC process. This ID must have a z/OS USS segment. The recommended home directory name is
/u/dt
.If a home directory other than
/u/dt
is set, change in step 8 the path to the bootstrapper, which defaults to/u/dt/agent/lib64/dtzagent
. The path names below the home directory can't be changed.The home directory or the high-level qualifier that replaces it must be writeable by the zDC process.
- It writes the zLocal binary to
/u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so
, wherea.b.c.d
is the version of the zLocal (i.e. 1.0.1.0). - It writes the zDC and zLocal logs to
/u/dt/log/dtxxx.log
.
If write fails, the zDC processes use the directory
/tmp/dynaTrace
.If the zDC is run as a started task, the ID must be capable of representing the zDC started task and have access to the USS path above.
- It writes the zLocal binary to
-
Change the dataset high-level qualifier represented by <hlq> in member
COPYAGNT
ofSZDTSAMP
to the value you used in the Download z/OS product datasets procedure. -
Run the
COPYAGNT
job to create the required z/OS USS subdirectories under the home directory of the zDC started task. -
Copy the
dtzagent
binary fromSZDTSAMP
and set it to be executable.If you're using the default home directory of
/u/dt
, the resulting path is/u/dt/agent/lib64/dtzagent
.If a home directory other than
/u/dt
is used, the path in this job must be changed accordingly. File and directory names are case-sensitive. The path names below the home directory can't be changed.The
COPYAGNT
job should run under the user ID that was created for the zDC process, so that user ID owns thedtzagent
binary. If this is inconvenient, you can also use thechown
andchgrp
commands to reset the owning user and group for thedtzagent
binary after runningCOPYAGNT
. -
Authorize
<hlq>.SZDTAUTH
where<hlq>
is the high-level qualifier value you used in the Download z/OS product datasets procedure.For example, create a member named
SYS1.PARMLIB(PROGDT)
containing:1APF FORMAT(DYNAMIC)2APF ADD DSNAME(<hlq>.SZDTAUTH) VOLUME(XXXXXX)Then issue the console command:
1SET PROG=DT -
Copy the
ZDCMEPC
sample zDC started task PROC fromSZDTSAMP
to a system PROCLIB used for started tasks.The default name for the PROC is
MEPC
. Customize it as necessary for local standards. If a different subsystem name is chosen, change theSUBSYSTEM_ID()
parameter in the SYSIN parameters to match.1//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)23SUBSYSTEM_ID(MEPC)The PROC and the two JCL SYSIN members
ZDCSYSIN
andZDCMEPCA
that it uses contain dataset names that must be edited to replace<hlq>.
with the appropriate high-level qualifiers. -
Update the
DTAGTCMD()
parameter in the SYSIN parameter to reflect your definitions.1//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)23DTAGTCMD(/u/dt/agent/lib64/dtzagent4nobootstrap=false5zremoteagent=<ipaddress>[:port])-
/u/dt/agent/lib64/dtzagent
sets the path to the bootstrapper. The default home directory name is/u/dt
. If you have set a different home directory in step 2, change it accordingly. -
nobootstrap=false
enables the bootstrapper to automatically update the zLocal once a newer version is available. Per default, the zLocal receives automatic updates. Settrue
to disable automatic updates and manually update the zLocal. -
zremoteagent=<ipaddress>[:port]
set the IP address and port of a installed zRemote module. The IP address is required, the port is optional and defaults to 8898.
-
-
optional Add a command to your system startup to automatically start the zDC subsystem at IPL.
-
Use the
ZDCIVP
PROC inSZDTSAMP
to verify the zDC installation and the zRemote connectivity.Good test for the zRemote connectivity returns
Connection timeout
, as the zRemote does not support FTP protocol. If a zRemote connectivity problem exist, it is indicated byEDC8128I Connection refused
. This implies that the zRemote is not listening on the expected port.
SYSIN dataset
All values that tailor the execution of the zDC reside in the dataset pointed to by DDNAME SYSIN
. The sample started task PROC ZDCMEPC
refers to the JCL SYSIN member ZDCSYSIN
of SZDTSAMP
as shown below:
1//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)
The dataset must contain 80 character records, blocked to a multiple of 80 bytes.
When you make any change to the SYSIN dataset, you must restart the zDC process for the change to take effect.
SYSIN parameters
Positions 1 through 71 of each statement can contain parameter values. Positions 72 through 80 are ignored and can optionally contain a sequence number.
Each parameter uses KEYWORD(value)
format. Each keyword and the requirements for the values associated with it are documented in the JCL SYSIN member ZDCSYSIN
of SZDTSAMP
.
If a parameter value spans multiple lines, specify the value through position 71 and continue the value on the next statement starting in position 1.
If you want to use a comment, specify it by placing an asterisk in position 1 of a statement. The entire statement is considered part of the comment.
A comment can be enclosed between a slash-asterisk and an asterisk-slash.
1/* This is a comment. */
Multiple lines can be contained within comments of this type.
Parameter(default value) | Description |
---|---|
DTAGTCMD(/u/dt/agent/lib64/dtzagent) | Sets the path to the bootstrapper Additional parameters:
|
SUBSYSTEM_ID(MEPC) | Defines the name of the zDC subsystem. It must be four non-blank characters:
|
DEFAULT(YES) | Defines a zDC subsystem as default. Monitoring data is collected by the default zDC subsystem that defines |
DISPLAY_NAME() | Defines a display name for this zDC subsystem to identify it in certain log messages. |
DTLOGLEVEL(3) | Sets the log level of the zDC. Log level values range from |
DTMSG_SMOSIZE(1) | Sets the maximum amount of storage (MB) for messages that can be queued in the zDC shared memory object while awaiting writing to the z/OS modules. The default of 1 MB should be adequate in most cases when transaction buffering is enabled. When transaction buffering is disabled (not recommended), set the SMO size to 10 MB if very high volumes of transactions are traced. |
DTCHDIR(/u/dt) | Changes the z/OS USS current directory where the bootstrapper |
DTMSG_TRANBUFSIZE(n,m) | Overrides the default number and size of transaction buffers. Transaction buffers offer better performance for CICS and IMS modules by placing event messages related to each distributed trace into dedicated buffers. Rather than send individual event messages for each distributed trace as they occur, they are blocked into one or more buffers and sent together. Additionally, the buffer of related messages is processed more efficiently by the zRemote. The |
Logging
The log level of the zDC is set with the DTLOGLEVEL()
parameter in the SYSIN parameters. You can change this log level dynamically by issuing a z/OS MODIFY command.
1//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)2DTLOGLEVEL(3)34MODIFY <jobName>,DT1 LOG=1
View log message output in the zDC job spool. The job spool helps determine errors that may occur during the zDC startup or when connecting with the zRemote. Once the zDC successfully connects to the zRemote, error messages from the zDC subsystem and the CICS and IMS modules are routed to the zRemote.
The log level of the zLocal is set with the DTAGTCMD(loglevel=INFO)
parameter in the SYSIN parameters. You can change this log level dynamically by issuing a z/OS MODIFY command. The DTAGTCMD(name=zlocal_<lpar>)
defines the name of the zLocal for log files. The name should reflect that this is a zLocal. It could include the SMF ID of the LPAR that it services.
1//SYSIN DD DISP=SHR,DSN=<hlq>.SZDTSAMP(ZDCSYSIN)2DTAGTCMD(loglevel=INFO name=zlocal_<lpar>)34MODIFY <jobname>,DT1 ZLALOGLEVEL=FINE
There are two sets of logs created for the zLocal. Both are created in the OMVS file system. One set is a temporary set of logs only valid for the current execution of the zDC, and the location of these logs defaults to the home directory of the zDC user ID. You can override the location by using the DTCHDIR()
parameter in the SYSIN parameters.
The zLocal also creates a standard set of logs, one for the bootstrapper, and one for the zLocal itself. These logs are in the standard Dynatrace log locations on the OMVS file system.
Update and maintenance
-
Update the zDC job to point to the new
<hlq>.SZDTAUTH
. If you have defined an alias, redefine the alias. For example:1DELETE 'DT.DYNTRC.SZDTAUTH' NOSCRATCH2DEFINE ALIAS(NAME('DT.DYNTRC.SZDTAUTH') RELATE('DT.R12710.SZDTAUTH'))
A special handling is required to avoid abends in CICS regions that you're monitoring when updating the zDC subsystem from a version 1.211 or earlier.
Stop the zDC.
- Wait for 15 minutes for the CICS module to reset and clean up the control blocks.
Update the zDC to the newer version.
Start the zDC.
zLocal
By default, the zLocal receives automatic updates if DTAGTCMD(nobootstrap=false)
is set in the SYSIN parameters. At every start of the zDC, the bootstrapper queries the zRemote module for available updates. The latest zLocal is installed as part of the zRemote installation. If an update is available, the zLocal will be automatically downloaded from the zRemote and updated.
1info (native) The bootstrap channel connected successfully, requesting version: a.b.c.d2info (native) Interprocess lock acquired for /u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so3info (native) Fetching agent binary succeeded for /u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so
Manually update the zLocal
Don't use the manual update process for the zLocal when connecting the first time to your Dynatrace environment because the zDC won't start successfully.
- The zRemote module must be running when starting the zDC for the first time to avoid a
U103
user abend.
Suppose you don't want to receive automatic updates for the zLocal, set DTAGTCMD(nobootstrap=true)
in the SYSIN parameters. For the manual update process, the zLocal binary libdtzagent.so
must be present on the LPAR in the /u/dt/agent/lib64
directory.
To accomplish this
- Run the bootstrapper
dtzagent
at least once in either a production LPAR or non-production LPAR. - Copy the zLocal binary to the target LPAR. In the
SZDTSAMP
dataset, the memberOCOPYAGT
is a job that copies the zLocal binarylibdtzagent.so
to the/u/dt/agent/lib64
directory. Complete the instructions to start the zDC and connect it to your Dynatrace environment.
- If the zDC successfully connects to Dynatrace,
libdtzagent.so
is copied to the/u/dt/agent/downloads/native/a.b.c.d/zos-s390-64/libdtzagent.so
directory, wherea.b.c.d
is the version of the zLocal (i.e. 1.0.1.0)
1info (native) Configured to not download module - loading local module2info (native) Start loading local agent binary3info (native) Successfully loaded agent binary /u/dt/agent/lib64/libdtzagent.so
Troubleshooting
Verify that the zDC has started the correct version, initialized successfully, and has started the zLocal (messages in zDC SYSPRINT DD).
1ZDC000I INITIALIZATION STARTED FOR zDC VER 1.195.002ZDC052I zDC IS RUNNING ON Z/OS RELEASE 02.02.003ZDC053I LPAR NAME IBMSYS1 CVTSNAME S0W14.5.6.7.8ZDC955L Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z0219ZDC958L Dynatrace INIT completed, ZDC AgentId received ZDC-Job/ID:AFVBZ021/Z02110ZDC993I Opn1RFD:0008 /u/labuser/adcdk/ci/7.2build/log/dt_Z021_Z021_33620108.0.log11ZDC955I Dynatrace connection being processed ZDC-Job/ID:AFVBZ021/Z021
If the following message appears in the zDC job log (your subsystem name might be different than MEPC
), restart the zDC it with a different SUBSYSTEM_ID
in the SYSIN parameters.
111:55:30.419083 ZDC006E SUBSYSTEM MEPC ALREADY EXISTS AND IS ACTIVE
To restart the zDC, follow the steps in Terminate the zDC.
The zLocal writes the log messages to the zDC SYSPRINT DD in addition to /u/dt/log
in the z/OS USS environment. The zLocal log contains information relating to zLocal startup, versioning, and connectivity to the zRemote.
To verify that all the channels in zLocal are connected to the zRemote, look for the following messages in the SYSPRINT:
1info [native] dynaTrace z Remote Agent data channel connected successfully, performing handshake.2info [native] dynaTrace z Remote Agent client handshake performed.3info [native] dynaTrace z Remote Agent data channel handshake successful, version[rr.rr.rr.bbbb].4info [native] dynaTrace z Remote Agent control channel connected successfully, performing handshake5info [native] dynaTrace z Remote Agent handshakes are complete, all channels are fully operational.
Use the z/OS MODIFY command below to instruct the zDC to display the zLocal status and to locate the log file paths of zLocal and zRemote.
1MODIFY <zdc-jobname>,DT1 STDO
Once the zLocal log file is located, it can be accessed using the standard techniques. Use OMVS, ssh/telnet, or view it in the Dynatrace web UI like a regular OneAgent log file.
Also, look for the following messages in the corresponding zRemote log file (the values inside the bracket should reflect your live data).
1info [native] dynaTrace z Local Agent handshakes are complete, all channels are fully operational, version[rr.rr.rr.bbbb].2info [native] Data client socket listener thread started3info [native] ASID[48], smfID[S0W1], sysid[Z208], jobName[AF71Z208], subagentid[da57ff16] smfID.JobID[S0W1 .JOB92014], zDC release 65 was successfully initialized with protocol version=6.5.04info [native] zDC[Z208] SMO is initialized with size=10M.5info [native] Registering the zdc[48]
Check whether the zDC subsystem version is lower or equal to the zRemote module version. If the zDC version is higher than the zRemote version, the zLocal can not connect to the zRemote.
Check the job log of the zDC for any of the following error messages.
These do not appear in the zLocal log file, since the zLocal could not be started.
Invalid permission on the zLocal folder or libraries:
1ZDC996E Atx5!DTAGTCMD attach failed2ZDC997D RetRcRsn FFFFFFFF 0000006F 5B400002 EACCES: Permission is denied3ZDC997D Notice: unknown modid, reason text might be incorrect4ZDC992I IniA!z/OS Unix agent attach failed5ZDC996E @Abrtz/OS Unix agent attach failed,Abort
Correct the permission on the zLocal folder and binaries, so they are at least readable and executable.
TCP/IP access denied:
1ZDC996E Atx5!DTAGTCMD attach failed2ZDC997D RetRcRsn FFFFFFFF 0000006F EF076015 EACCES: Permission is denied3ZDC997D Description: SAF CKACC returned error.4ZDC996E @Abrtz/OS Unix agent attach failed,Abort
If your security system, such as Top Secret or ACF2, denies access to TCP/IP by default, add a rule to permit the zDC user ID to access TCP/IP.
Invalid location specified for the zLocal:
1ZDC996E Atx5!DTAGTCMD attach failed2ZDC997D RetRcRsn FFFFFFFF 00000081 0516006C ENOENT: No such file, directory, or IPC3 member exists4ZDC997D JRFileNotThere: The requested file does not exist5ZDC992I IniA!z/OS Unix agent attach failed6ZDC996E @Abrtz/OS Unix agent attach failed,Abort7ZDC975E z/OS Unix agent attach failed,Abort
Correct the DTAGTCMD
parameter in the SYSIN parameters to point to the correct zLocal binary location.
Browse the install library <hlq>.R1nnnxx.SZDTAUTH
, where <hlq>
is the high-level qualifier of the product dataset, to find the ZDCDTAGT
load module for the zLocal.
In the header example below, you can see 1.nnn.00
, which indicates this is the version of the load module without any maintenance. When service is applied, the field contains a subversion number like 1.nnn.01
. The dates vary over time.
1ZDCDTAGT 00000000 YYYYMMDD HH.MM VER 1.nnn.00 COPYRIGHT (C)...
The ZDCDTAGT
load module can produce diagnostic messages in the following format:
1ZDC99<n><i>
Where n
is the log level and i
is the severity.
zDC user abends
A zDC may terminate with a user abend due to internal errors and initialization failures. All user abends, except for U103
and U106
, create a dump file. See the list of possible user abends below.
Abend | Description |
---|---|
U100 | An internal error occurred due to zDC initialization failure. |
U101 | zDC initialization failed and is unable to display the failure message. |
U102 | ECSA storage is not available for the zDC. |
U103 | A problem occurred during a RETRY operation from a recovery exit routine. |
U104 | A queue task ended abnormally. This abend is accompanied by a |
U105 | A queue task ended abnormally while shutdown was NOT in progress. |
U106 | A fatal error occurred in TCPSP routine. |
U110 | This abend might occur for various internal error reasons. It is accompanied by one of the following zDC error messages in the SYSLOG:
|
U111 | An internal error occurred in one of the zDC modules. The abend is accompanied by one of the following error messages in the SYSLOG:
|
U205 | The zDC is unable to open SYSPRINT or SYSPRIN3 DD for displaying log messages. |
U222 | An internal error occurred in one of the zDC modules. The abend is accompanied by one of the following messages in the SYSLOG:
|