CPU/LPAR Parking:
- CPU Definition
- Introduction to Parking
- Processor Cache
- Parking Summary/Benefits
- Things to keep in mind
- Helpful system commands
CPU Definition:
Before beginning, some clarification on CPU naming:
- The machine is equipped with physical cores - not engines, not processors, not CPUs, not IFLs, not CPs.
- Cores, whether physical or logical, come in different types: CP, IFL, etc. Logical cores and physical cores have a percent-busy metric called core utilization.
- For a logical core - this is the percent of time the logical core is dispatched on a physical core.
- For a physical core, this is the percent of time the physical core has a logical core dispatched upon it.
- Contained within a core, either physical or logical, are instruction execution units called processors. Physical cores contain physical processors. Logical cores contain logical processors.
- An IFL core-type can have either one or two processors contained in the core, depending upon the SMT level. SMT-2 has two processors in the core.
- Logical processors have a percent-busy metric called processor utilization. This is the percent of elapsed time the processor has a non-wait PSW loaded (this hasn't changed).
- Synonomous with processor utilization is processor busy, processor load, CPU utilization, CPU load and CPU busy.
Introduction to Parking:
First, in z/VM 6.3 there were Hypervisor changes that introduced polarization modes. The difference between horizontal and vertical polarization:
- Horizontal polarization - Each online (configured) logical CPU of the same type gets an equal portion of the partition's weight. z/VM tends to spread work evenly over those CPUs and tends to let virtual CPUs move freely from one logical CPU to another. This is the historical method of operating.
- Vertical polarization - Each of the partition's logical CPUs have different processing entitlements to physical CPUs. Entitlement can be to an entire physical CPU, some fraction or leftover/unconsumed resources. High entitlement CPUs are dispatched consistently, thus preserving accumulated cache contents. z/VM tends to run the work on fewer logical CPUs, keeping guests with the same cache topology together and does not tend to move virtual CPUs among logical CPUs. If the right balance is achieved, this can tend to improve system performance. This is now the default setting and mandatory for running SMT.
Second, introduced in the PTF for APAR VM66063 was the 'parking' of logical CPUs when the system has calculated there will be excess CPU power available. This allowed z/VM to determine the best number of CPUs for consuming the available power. Later support was provided to make changes to the defaults of when to 'park'. Parking decisions are made every two seconds. The amount of High, Medium and Low assignments are done by a system algorithm based on the number of engines and their weights, but the unparking level can be done by command.
Third, a few more specifics - understanding LPAR/CPU polarization and parking/unparking designations:
- CPU polarization:
- Horizontal - each CPU gets an equal portion of the LPAR weight.
- Vertical-High - based on LPAR weighting, vertical high CPUs have high entitlement. It is entitled to run 100% busy.
- Vertical-Medium - based on LPAR weighting, vertical medium CPUs have medium entitlement. It can have any entitlement from 0-100% based on the partition's total entitlement and the number of logical CPUs sharing that entitlement.
- Vertical-Low - based on LPAR weighting, vertical low CPUs have low entitlement. It is entitled to nothing.
- Unparking specifications:
- Unparking Large - (This was the default setting until z/VM 7.2)
- This unparks all of the entitled logical cores and some of the vertical-low cores.
- Unparking Medium - (This is the current default setting as of z/VM 7.3)
- This unparks all of the vertical-high logical cores, all of the vertical-medium logical cores and either
- All of the vertical-low logical cores (that z/VM predicts PR/SM will power) or
- All of the vertical-low logical cores (that z/VM predicts will be needed to cover the workload, plus the CPUPAD setting).
- Unparking Small -
- This unparks only the logical cores that are needed to cover the workload, plus the CPUPAD setting.
Helpful Screens:
ESAOPER - Shows in the operator log when a CPU has been parked and unparked. It will always show even numbers.
Processor Cache:
There are different levels of memory cache. Each CPU has L1 and L2 cache that is private to the CPU. L3/L4 cache is
shared by the CPUs at different levels of the machine. L1 cache must be populated with instructions and associated data
in order to be used. Performance is obviously better when all needed information is in L1 cache and processing can stay
in local cache. If other non-local cache is needed because the necessary data is unavailable in L1 cache, it can cause
delays in processing. When a core is unparked, its L1 cache must be refreshed before it can be used. This will also
cause delays in processing.
ESAMFC screen/report below shows the affects of unparking on cycles per instruction (CPI).
CPI is a measure of cache effectiveness. Watching this report will show impacts of parking/unparking over time and trends
can be determined. Since Linux tends to do consistent periodic polling, it creates a significant amount of cache use that
is not needed after the polling completes. This causes Linux workloads to not necessarily do well with vertical
polarization.
Helpful Screen:
ESAMFC - The screen/report shows the processor cache analysis for each minute (screen) and in 15 minute increments and a Summary for the day (report). This report will show if the cache is being used effectively. Note: You must have Measurement Facility turned on in the LPAR to collect the correct records for this screen/report - See Enabling CPUMFC Records
Parking Summary/Benefits:
For parking/unparking, each engine - based on LPAR weights (done dynamically - not assigned) - is provided a polarization
of high, medium or low. The system hypervisor will park LPAR engines, meaning will not dispatch those engines to the
LPAR based on requirements of the LPAR and total system utiilzation. In an SMT environment, both threads on that engine
are parked, z/VM is alerted and z/VM will no longer dispatch on those engines. High levels of parking on smaller systems
have been shown to be problematic with performance improving when the hypervisor is not allowed to park engines. SRM
settings and LPAR weightings can be used to adjust the parking/unparking of engines. For z/VM 7.2, the UNPARKING default
changed from LARGE to MEDIUM to reduce the tendency to use vertical-low logical cores and thereby potentially reduce the
PR/SM hypervisor overhead and improve the use of processor cache.
When this all can be done correctly, the following benefits can be expected:
- Efficiency - If the system is tuned correctly, parking/unparking allows the CPU cores/LPARs to work more efficently together thus utilizing cache and processing resources in a more productive manner.
- Performance/Capacity - If the the system is running more efficiently, there is better performance and more capacity can be gained from the same processing environment.
Helpful Screen:
ESACPUU - screen/report shows the CPU utilization and how often processors were parked in 15 minute increments and a summary for the day.
Things to keep in mind:
- Polarization must be set to vertical for CPU parking.
- CPUs can not be DEDICATE'd via command or the directory in vertical mode. And/or can not switch to vertical mode if there are any dedicated CPUs.
- Global Performance Data (GPD) must be set to ON on the hardware console for the LPAR.
- If GPD is on - z/VM decides the unparked configuration based on projected capacity.
- If GPD is off - z/VM decides the unparked configuration based on projected consumption.
- Parked CPUs remain in wait state - still varied on.
- Parking/Unparking is faster (and more automatic) than VARY OFF/ON.
- It is logical core utilization that z/VM examines in making unparking decisions.
- CPUPAD is taken into account when GPD is on if the MEDIUM or SMALL unparking setting is selected.
- The SRM setting EXCESSuse - None specifies that vertical-low logical cores should always be parked when GPD is ON. The dispatch contention can be forcibly reduced by using EXCESSuse None to force those LPARS to run only on their entitled logical cores. (If a problem condition has passed, it can be set back to MEDIUM or SMALL to resume unparking vertical-low cores).
- z/VM estimates future values that influence the decision of how many processors to unpark.
- First - it tracks CPU consumption in its own LPAR to project a ceiling/maximum value (consumption ceiling projection).
- Second - it tracks CPU usage and entitlement across the entire CPC to project a floor/minimum value (capacity floor projection).
- These are refreshed every two seconds and are used to decide how many processors to unpark.
- If GPD is disabled, z/VM cannot validate that PR/SM will power all of the in-use logical CPUs and projects its own utilization using the projected ceiling and the padded value.
- When a logical core is in dispatch, its two logical CPUs can independently move in and out of wait state. The concepts of CPU utilization and core utilization aren't exactly the same.
- Velocity's recommendations (see below for commands) -
- Set unparking to LARGE (which leaves all the high/medium and most of the low cores unparked.)
- Set EXCESSuse to IFL high (which specifies how aggressively the system should attempt to use unentitled CPU capacity.)
- Set CPUADD to 200%.
Helpful system commands:
+-TYPE--ALL------+
>>--Set--SRM--+-CPUPAD--+----------------+--qqqqq%--------------+-----------><
| +-TYPE--+-ALL--+-+ |
| +-CP---+ |
| +-ZIIP-+ |
| +-ZAAP-+ |
| +-IFL--+ |
| +-ICF--+ |
+-DSPWDMethod--+-REShuffle-+----------------------+
| +-REBalance-+ |
| +-TYPE--ALL------+ |
+-EXCESSuse--+----------------+--+-High---+-------+
| +-TYPE--+-ALL--+-+ +-Medium-+ |
| +-CP---+ +-Low----+ |
| +-ZIIP-+ +-None---+ |
| +-ZAAP-+ |
| +-IFL--+ |
| +-ICF--+ |
+-POLARization--+-VERTical---+--------------------+
| +-HORIZontal-+ |
+-UNPARKing--+-Large--+---------------------------+
+-Medium-+
+-Small--+
Commands for the recommendations above:
CP SET SRM UNPARKING LARGE
CP SET SRM EXCESSUSE TYPE IFL HIGH
CP SET SRM CPUPAD TYPE IFL 200%
+-ALL----------+
>>--Query--SRM--+--------------+--------------------------------------------><
+-CPUPAD-------+
+-DSPWDMethod--+
+-EXCESSuse----+
+-POLARization-+
+-UNPARKing----+
Example: (only relevant items are shown - all are defaults on a z/VM 7.1 system)
q srm POLARIZATION: VERTICAL GLOBAL PERFORMANCE DATA: ON EXCESSUSE: CP-MEDIUM ZAAP-MEDIUM IFL-MEDIUM ICF-MEDIUM ZIIP-MEDIUM CPUPAD: CP-100% ZAAP-100% IFL-100% ICF-100% ZIIP-100% DSPWDMETHOD: RESHUFFLE UNPARKING: LARGE
>>--Query--PROCessors--+----------+-----------------------------------------><
+-EXPanded-+
+-TOPOlogy-+
Examples:
Q PROCessor PROCESSOR 0000 MASTER IFL CORE 0000 PROCESSOR 0001 ALTERNATE IFL CORE 0000 PROCESSOR 0002 ALTERNATE IFL CORE 0001 PROCESSOR 0003 ALTERNATE IFL CORE 0001 PROCESSOR 0004 STANDBY IFL CORE 0002 PROCESSOR 0005 STANDBY IFL CORE 0002
Q PROCessor TOPOlogy
TOPOLOGY
NESTING LEVEL: 02 ID: 01
NESTING LEVEL: 01 ID: 02
PROCESSOR 0000 MASTER IFL VM 0000 CORE 0000
PROCESSOR 0001 ALTERNATE IFL VM 0000 CORE 0000
PROCESSOR 0002 ALTERNATE IFL VL 0001 CORE 0001
PROCESSOR 0003 ALTERNATE IFL VL 0001 CORE 0001
Back to top of page
