ZMAP Extract Facility Explanation and Examples
zVPS provides a full performance database (PDB) that holds data for as long as an installation desires. The PDB shows system-by-system and application-by-application storage, processor and I/O requirements. Many different reports can be created for specific trending, problem determination and especially capacity planning and hardware consolidation/uplift projects.
- Overview of ESAEXTR
- How to create/run an extract
- Top User Example
- User CPU Example
- User Wait Example
- LPAR Utilization
- Linux CPU Utilization
Overview of ESAEXTR Operations
There are over 3,000 variables defined in history keywords that can be extracted from a zVPS history archive (ESAEXTR). The format of the history keywords is "segment.field" where segment defines the source monitor record and the "field" defines the specific field. There are also over 3,000 variables defined in the Performance Data Reference manual that can be used for real time or historical data.
There are three kinds of data items:
- CP Monitor variables defined by IBM and described on one of the many monitor data descriptions. Learning the segment names will help in finding specific data items wanted for extraction. For example, to get the CPU utilization use the "SUBSUM.CPUUTIL" variable name (SUBSUM is the source monitor record and CPUUTIL is the specific field). There are two good resources for finding the variable names:
- The HISTORY KEYWORDS file on the ZMAP code disk (by default VMSYSVPS:ZVPS.CONFIG directory)
- The current zVPS Performance Data Reference (currently 5.2) in the ZMAP column
- Application variables defined by IBM for each application such as TCPIP or SFS.
- zVPS defined fields derived from snmp data collected by ZTCP.
Note that each variable may be a count, rate, or state. For example, one variable "USEACT.VMDTTIME" may be a count of CPU seconds consumed during the interval, but "USEACT.USERPCT" is the rate of consumption, and USEACT.VMDCTPRS is a 'state' value, being the number of pages resident for this user.
There are five files that make up the Performance Data Base (PDB) or "history files":
- ESASYS - system level data, such as channels and saved segments
- ESACPU - CPU related data, reported by the CPU
- ESADEV - DASD and Device data, reported by devices and control units
- ESAUSR - User data, reported by user, userclass and accounting codes
- ESATCP - Network data, Linux data, and distributed server data
How to create/run an extract:
Historical data is extracted via the ESAEXTR exec. On the ZVPS userid on z/VM (by default in the VMSYSVPS:ZVPS.CONFIG directory), there are several files with the file type of ESAEXTR. These can be used as is or as examples to create other desired extracts. All the detailed information about running extracts can be found in the ZMAP Installation and Operations Guide. Extract variable names relevant to ESAEXTR can be found in the ZMON Performance Data Reference.
ESAEXTR: Used to pull data from the performance database. It can create reports or files with raw data for exporting. It uses the HISTORY KEYWORDS file and ZMAP column in the Performance Data Reference for variable names.
- Logon to a userid that can access/link:
- the ZMAP SFS directory code disk (default is VPSYSVPS:ZMAP.CODE).
- the ZMAP 192 disk.
- Create a file with a file name that relates to the data being extracted and a file type of ESAEXTR (or copy one from the VMSYSVPS:ZVPS.CONFIG directory) and add the variables desired from the HISTORY KEYWORDS file. If this is to be a report, add column headers as shown in the examples below.
- Run the extract via ESAEXTR input (PARM fn - where input is a date such as 202201* for all days in January of 2022 and fn is the file name of the ESAEXTR file created in step 2. This will create a file called 202201 EXTRACT on the user's 191 disk. (An * can be used for multiple days/months or an exact date of yyyymmdd can be used for a single day.)
- Run an exec (like the top user exec below). An exec would allow a multitude of possibilities to massage the data for reports and/or displays.
Back to Top
Top User example
ESAEXTR: To determine the top users over a month of data would use the following extract (named "TOPUSER ESAEXTR"):
EXTRACT: X = 'STARTTIME' Y = 'USERID' Y = 'USERTYPE' Y = 'USEACT.VMDVTIME' Y = 'USEACT.VMDTTIME' Y = 'useact.VMDVDSCT' ;DASD I/O Y = 'useact.VMDCTPVR' ;Resident Storage INTERVAL='SU' criteria = 'USERTYPE = USER'Create - Put the extract above into a file named TOPUSER ESAEXTR on the 191 disk
To run - ESAEXTR yyyymmdd (PARM USERCPU - or use the exec below
EXEC: One method of running this command is a sample exec that calls the extract, processes it, and puts the data to the console. The rexx exec to do this kind of analysis is similar to the following, accepting the history filetype as the argument:
/* Exec to take extracted data, summarize and show top 10 */
Parse arg input .
'ESAEXTR' input '(PARM TOPUSER DATA'
input = STRIP(input,,'*')
'PIPE <' input 'EXTRACT A | sort w4 | stem data.'
curuser = ''
userdata.0 = 0
usercount = 0
DO I = 1 to data.0
Parse var data.i date start stop user type tsecs vsecs .
If curuser <> user
Then Do
usercount = usercount + 1
If curuser <> ''
Then userdata.usercount = user totalcpu
curuser = user
totalcpu = 0
End
totalcpu = totalcpu + tsecs
End
usercount = usercount + 1
If curuser <> ''
Then userdata.usercount = user totalcpu
userdata.0 = usercount
'PIPE stem userdata. | spec w2 1-10 right w1 15',
'| change 1-10 / /0/',
'| sort w1 D | take 10 | cons'
Assuming the EXEC is named TOPUSER EXEC, then issue the following command for April, 2011 data:
TOPUSER 201104*
This results in both a report showing the CPU seconds consumed and the user that consumed them on the 191 disk and it also puts the top ten users to the screen.
0000002473 LNXUWA02 0000001615 LNXUWA11 0000001413 LNXUWA03 0000000441 LNXQWA02 0000000415 LNXQWA01 0000000354 LNXDWA03 0000000309 LNXDWA04 0000000269 LNXPSP03 0000000217 LNXUWA15 0000000215 ESATCP
Back to Top
User CPU example
This example will extract user CPU data, including number of virtual CPUs, as well as show CPU per virtual cpu. Only data from prime shift is extracted.
EXTRACT: title=' STOP USERID NVCPU USERCPU USER' title=' TIME /CPU' title=' ------ --------- ----- ------ ------' columns = '7 9 5 8 7' X = 'STOPTIME' Y = 'USER' Y = 'USRCON.NVCPU' sig=3 Y = 'USEACT.USERPCT' sig=2 Y = 'USEACT.USERPCT / USRCON.NVCPU' sig=3 criteria='USEACT.USERPCT > 0.1' criteria='starttime > 075900' criteria='stoptime < 170100' interval='IN'
Create - Put the extract above into a file named fn ESAEXTR on the 191 disk
To run - ESAEXTR yyyymmdd (PARM fn
The following is a sample produced by this extract.
*HDR Date Time STOP USERID NVCPU USERCPU USER
* TIME /CPU
* ------ --------- ----- ------ ------
20110415 100000 100100 TCPIP 1 0.11 0.108
20110415 100000 100100 LNXUWA01 1 65.65 65.650
20110415 100000 100100 LNXUWA03 2 22.70 11.349
20110415 100000 100100 LNXUWA11 2 19.23 9.613
20110415 100000 100100 LNXUWA02 2 18.45 9.227
20110415 100000 100100 LNXQWA01 2 11.56 5.778
20110415 100000 100100 LNXDWA04 2 9.88 4.942
20110415 100000 100100 LNXPSP02 1 7.67 7.672
20110415 100000 100100 LNXUWA15 2 6.82 3.410
20110415 100000 100100 LNXDWA02 2 4.40 2.202
Back to Top
User Wait example
This example will extract user wait state data. When complete, the ESAPLOT process can be run to show how many virtual machines were in each of the different wait states for some period of time.
EXTRACT: INTERVAL = 'INTERVAL' X = 'SYTSCG.SRMCDISP' ; inqueue users Y = 'SYTUWT.CALCPURN' ; Percent in queue VMDBKs running Y = 'SYTUWT.CALCPUWT' ; Percent in queue VMDBKs in CPU Wait Y = 'SYTUWT.CALSIMWT' ; Percent in queue VMDBKs in Simulation Wait Y = 'SYTUWT.CALIOWT' ; Percent in queue VMDBKs in I/O wait Y = 'SYTUWT.CALWTPAG' ; Percent in queue VMDBKs in Page Wait Y = 'SYTUWT.CALTSVM' ; Percent VMDBKs in test idle and SVM wait Y = 'SYTUWT.CALCFWT' ; Percent in queue VMDBKs in Console Function Y = 'SYTUWT.CALOTHR' ; Percent in queue VMDBKs not in above states Y = 'SYTUWT.CALIOACT' ; Percent in queue VMDBKs with asynchronous I/O active
Create - Put the extract above into a file named fn ESAEXTR on the 191 disk
To run - ESAEXTR yyyymmdd (PARM fn or ESAEXTR yyyymm* (PARM fn - First is for one day, second is for all days in that month)
Once complete, use the file produced as input for ESAPLOT.
Create - Rename the file from yyyymmdd EXTRACT to (something meaningful like) USERWAIT EXTRACT on the 191 disk
To run - ESAPLOT fn (combine
The following is a sample produced by this plot.
Plot 1: 2022022 All cross-variables vs. Runnable users 0Base variable: SYTSCG.SRMCDISP Runnable users 0Sym Cross-variable Scale Origin Average Maximum Description --- --------------- ----- ------- ------- ------- ------------------------------------------------------------ * SYTUWT.CALCPURN 1 0.0 3.3 6.3 Percent VMDBKs running + SYTUWT.CALCPUWT 10 0.0 5.2 23.0 Percent VMDBKs in CPU Wait - SYTUWT.CALSIMWT 0.1 0.0 0.3 3.3 Percent VMDBKs in Simulation Wait # SYTUWT.CALWTPAG 1 0.0 0.0 3.8 Percent VMDBKs in Page Wait @ SYTUWT.CALTSVM 1 0.0 2.2 8.7 Percent VMDBKs in test idle and SVM wait > SYTUWT.CALCFWT 0.1 0.0 0.0 1.0 Percent VMDBKs in Console Function Wait = SYTUWT.CALOTHR 1 0.0 1.0 3.1 Percent VMDBKs in other wait state % SYTUWT.CALIOACT 1 0.0 0.8 3.9 Percent in-queue VMDBKs with asynchronous I/O active
Back to Top
LPAR Utilization
To extract CPU utilization by LPAR, use the following extract. This shows the total CPU, virtual CPU and overhead of the lpar. Note the report headings and column settings.
EXTRACT: Interval = 'SU' title = ' Stop LPAR vCPU <--CPU Utilization-->' title = ' time Name count Total Virtual Overhd' title = ' ------ ------- ---- ------ ------- ------' COLUMNS = '07 09 03 08 8 7 ' X = 'STOPTIME '; Y = 'SYTCUP.LCUPNAME '; Y = 'SYTCUP.LCUPCPCT ' sig = 0 Y = 'SYTCUP.LCUCACTM / 10000 / SECONDS ' sig = 2 ;Total Y = 'SYTCUP.LCUCLPTM / 10000 / SECONDS ' ;Logical CPU pct Y = 'SYTCUP.PCTOVHD '; ;Difference CRITERIA = 'SYTCUP.LCUCPUID > 63' ; TOTAL for LPAR */
Create - Put the extract above into a file named fn ESAEXTR on the 191 disk
To run - ESAEXTR yyyymmdd (PARM fn
This results in the following:
*HDR Date Time Stop LPAR vCPU <--CPU Utilization-->
* time Name count Total Virtual Overhd
* ------ ------- ---- ------ ------- ------
20110726 000000 010000 Totals: 5 84.32 84.22 0.10
20110726 000000 010000 VSIVM2 1 0.98 0.97 0.01
20110726 000000 010000 VSIVM1 2 4.89 4.88 0.01
20110726 000000 010000 VSIVM3 1 62.34 62.33 0.00
20110726 000000 010000 VSIVM4 1 16.11 16.03 0.08
20110726 010000 020000 Totals: 5 80.68 80.60 0.09
20110726 010000 020000 VSIVM2 1 1.05 1.05 0.01
20110726 010000 020000 VSIVM1 2 3.14 3.14 0.01
20110726 010000 020000 VSIVM3 1 62.28 62.27 0.00
20110726 010000 020000 VSIVM4 1 14.21 14.14 0.07
Linux CPU Utilization
To extract CPU utilization by Linux NODE, use the following extract. This shows the 'Nice, user, system CPU, as well as I/O time and kernel CPU. Note the report headings and column settings. Note also the CRITERIA allowing only one node to be extracted - this is currently commented out but can be used if desired by removing the ';'.
EXTRACT: ;CRITERIA='NODE=broblx1' title=' STOP NodeIO Kernel' title=' TIME Name Nice user SYST Time CPU ' title=' ------ -------- ------ ------ ------ ------ -------' columns = '7 9 7 7 7 7 7 7' INTERVAL='SU' X = 'STOPTIME' Y ='NODE' Y ='UCDSYS.NICEPCT' Y ='UCDSYS.USERPCT' Y ='UCDSYS.SYSTPCT' Y ='UCDSYS.IOWAIT ' Y ='UCDSYS.KRNLCPU'
Create - Put the extract above into a file named fn ESAEXTR on the 191 disk
To run - ESAEXTR yyyymmdd (PARM fn
An extract would produce the CPU time by server as in the following extract:
*HDR Date Time STOP NodeIO Kernel * TIME Name Nice user SYST Time CPU * ------ -------- ------ ------ ------ ------ ------- 20110726 000000 010000 *TheUsrs 0.42 4.25 5.55 1.25 0.92 20110726 000000 010000 vpnc 0.00 1.87 2.55 0.00 0.00 20110726 000000 010000 linux64 0.08 0.14 0.56 0.64 0.34 20110726 000000 010000 sles11 0.00 0.18 0.28 0.01 0.00 20110726 000000 010000 sles11v2 0.07 0.11 0.16 0.20 0.00 20110726 000000 010000 roblx2 0.22 0.01 0.46 0.03 0.00 20110726 000000 010000 sles11v 0.05 0.07 0.09 0.26 0.00 20110726 000000 010000 suselnx1 0.00 0.40 0.64 0.12 0.59 20110726 000000 010000 vpnz 0.00 1.48 0.81 0.00 0.00 20110726 010000 020000 *TheUsrs 0.41 4.35 5.56 1.25 0.95 20110726 010000 020000 vpnc 0.00 1.87 2.55 0.00 0.00 20110726 010000 020000 linux64 0.08 0.32 0.60 0.63 0.37 20110726 010000 020000 sles11 0.00 0.18 0.28 0.01 0.00 20110726 010000 020000 sles11v2 0.06 0.06 0.16 0.25 0.00 20110726 010000 020000 roblx2 0.22 0.01 0.46 0.02 0.00 20110726 010000 020000 sles11v 0.05 0.06 0.09 0.23 0.00 20110726 010000 020000 suselnx1 0.00 0.41 0.63 0.10 0.58 20110726 010000 020000 vpnz 0.00 1.46 0.79 0.00 0.00
