Basic Report and Data Flow Methodology

The Velocity methodology is a top down process. Below are some basics on how to determine where to look for performance issue causes. More details are found in the Flow Chart section.

Screen/Report General Information

• If ending in 'C' it contains configuration data.
• If ending in '1' it contains configuration data more relevant to performance.
• If ending in 'R' it contains raw data (CPU seconds).
• If ending in 'P' it contains data with the percent calculated (CPU seconds/time).
• On ESALPARS - the first LPAR is the reporting LPAR (where the report was run)
• On ESAUSR2/ESAUSP2/ESASTR1 - the unit of measurement can go between pages and megabytes.
• The default is changing from Pages to Megabytes
• To update it to your preference - in the ESAPARM FILE - Add the parm uspg_byMB = 'x'b - where 'x' is 1 = for megabytes and 0 = for pages.
• The different sources of data for Linux are Host (ESAHSTx), University of California Davis (ESAUCDx) and Velocity (ESALNXx).
• ESAHSTx screens/reports do not count deceased processes so the capture ratio may be low.
• ESAUCDx screens/reports show standard Linux system memory at a high level (what top shows).
• ESALNXx screens/reports will provide the most detailed and accurate information with low overhead.

General information flow:

In general, these are good things to investigate. All of these are explained in more details on the flow chart page:

• ESAHDR - Basic configuration:
• Note the master processor for reference. It could be over utilized.
• Is the system operating on IFL's? Seems obvious, but the system may have been configured incorrectly.
• Is SMT being used? This will make a difference in the performance numbers.
• ESAMAIN - Overall system performance:
• How does the system look overall?
• Did anything change drastically at the time the problem started?
• Did processor utilization go up? Did resident memory change? How about I/O?
• ESAXACT - Wait states (what resources are in low supply):
• SIM shows when the master processor is utilized. If waiting on SIM, there is an overload on the master processor.
• CPU shows when CPU is a bottleneck - users are waiting for CPU and not getting it.
• ESALPARS - LPAR and IFL utilization:
• What is the weight/polarization? Is it sufficient?
• How busy is each of the processors? Are they running over 80%?
• Is the system overhead time over 10%? That can be an indication of system thrashing.
• ESACPUU/ESACPUA - z/VM perspective:
• Did the system CPU utilization spike? If multiple engines, are they being utilized evenly?
• Shows the system overhead from the z/VM perspective.
• Shows where the master processor may be over utilized.
• ESAUSRC - z/VM SHARE settings
• Are there users/systems with incorrect SHARE settings?
• When adding a vCPU to a machine, be sure to adjust the SHARE setting accordingly.
• ESAUSP2/ESAUSP5- Individual virtual machine usage:
• Shows total amount of virtual cpu use.
• Shows the total to virtual ratio (T:V ratio) which indicates system overhead.
• Shows top users which could easily indicate a dominating server/user.
• ESAUSP5 - Shows the different ways of calculating user utilization with SMT active.

Common Tuning Issues

Problems will show up in many ways such as workload timeouts, applications running slowly, workloads/servers waiting or someone notices high steal time.
Many problems are caused by tuning issues such as (addressed on other pages):

1. LPAR tuning issues such as: LPAR Tuning
• LPAR weights vs utilization
• LPAR vCPU vs Assigned share
• LPAR set up inefficiently (extra cycles to spare but not able to be used).
• Parking CPUs on an LPAR (CPU/LPAR Parking)
2. Inefficient settings
   • SHARE settings incorrect Setting SHARE values for Virtual Machines (ESAUSRC)
   • Operating on the wrong engine type for the workload (GP engine vs IFL vs ZIIP). (ESAHDR)
   • HiperPAV turned off for paging. (ESADSD2)
   • No swap disk for Linux servers. (ESAUCD2)
   • SMT turned on when not needed. (ESALPAR/ESALPARS/ESACPUU)
   • Too many virtual CPUs (vCPUs) being allocated but not utilized. This can cause spin lock and/or processor cache issues. (ESALPAR/ESALPARS/ESACPUU)
3. Workload issues such as:
• Synchronized Cron jobs - 100 processes over 100 servers
• Spin locks (DIAG 44 vs DIAG 9C) and/or too many vCPUs
• Over usage of the master processor

Investigation Methodology - General to Specific:

• For Allocation Settings -
• CEC - has some number of engines
• LPAR - each LPAR has weights (becomes 'entitlement' divided equally by vCPU)
• Virtual Machine - each virtual machine has shares (becomes 'virtual entitlement')
• Process - Linux has priority/nice settings (gets virtual machine vCPU share)
• For Measuring Utilization -
• CEC - are the engines highly utilized?
• LPAR - do the LPARS have enough entitlement? Are there cycles to spare?
• z/VM - do virtual machines have enough/too much SHARE?
• Linux - are processes properly niced?
• Platform Considerations
• z/VM - if a problem is running in a virtual machine, check z/VM first
• Application - if not running on a virtual machine, then the application needs investigation
• Tuning Considerations
• Measure - have a baseline of the system's typical performance
• When making a change - change only one thing at a time
• Validate the change had the desired affect