Storage Subsystem

Storage Subsystem Analysis

Storage Analysis

Storage (or memory) is a very important part of the system. Understanding storage utilization is critical and will depend greatly on workload. Paging happens when storage/memory is not available and data has to be written to disk. It is much more efficient to use storage as well as improving response times. Paging can cause severe system performance issues. Defining storage varies with traditional CMS workloads and Linux workloads. The amount of total storage, the size of the Dynamic Paging Area (DPA) and the size/type of paging disks all come into play for the most efficient performance of the system. One of the top reasons for a first-time installation having a z/VM outage is from lack of proper page space planning. If running out of page space, the system will take a PGT004 abend.

For more information about paging, see System Page analysis.

Linux and Storage:

Storage over-commit used to be a good way to reduce costs. Now, especially with Linux, it is no longer a valid metric.
Linux uses a 'round robin' style of paging - calling the least recently used first, however z/VM has most likely already moved that page to disk.
Linux tries to cache as much as possible. This is helpful on a server but not on z/VM. Managing the storage size for Linux systems is important (it doesn't need to be huge).
Linux uses storage for its own internal storage tables. When making a Linux server extra large, it needs that much more storage for those tables.
If Linux storage is an issue, consider using zVRM

For a presentation on the Storage/Paging environment and utilization, see z/VM Storage Analysis and Tuning

Helpful configuration settings:

Storage requirements should be reduced as much as possible to avoid unnecessary paging delays. Plan on 2GB of storage for z/VM, MDC and infrastructure (TCPIP/DIRMAINT/zVPS).
Review options for reducing storage requirements BEFORE analyzing or enhancing the paging subsystem. Often if storage is added when not necessarily needed, CPU will go up as work is not being delayed by paging. Watch batch windows for any work delays during high utilization.
Linux - the over-commit ratio is the planning target. IE - For 20 Linux servers at 1GB each with a target over-commit ratio of 2, 12GB will be required.
For WAS and Domino environments, an over-commit target of 1.5 is reasonable. For Oracle and other virtual friendly applications, an over-commit of 3 is reasonable.
(# of servers x server size / 2 plus 2GB for z/VM = storage requirement)
If adding more Linux servers, decrease the Linux server storage size until they start to swap. This is an easy tuning knob available to improve storage utilization!
It is recommended to set the values for the global aging list to help manage pageable storage. Set the size to 5%, earlywrites to yes and keepslot to yes.
SET AGELIST SIZE 5% EARLYWRITES YES KEEPSLOT YES
Recording account data uses a lot of storage - it can be turned off in the SYSTEM CONFIG or with CP RECORDING ACCOUNT OFF PURGE.
Another space saving setting is to limit the Trace Table size with CP SET TRACEFRAMES MASTER 100 ALTERNATE 75 PERCENT.
Minidisk Cache (MDC) defaults to 'all of it', set it down (ZVPS does use MDC!). SET MDC STORAGE 128M 1282M
Minidisks used for Linux file systems and for swap should have MDC turned OFF to avoid redundant levels of caching.
It is a very good idea to set up an alerts for page space in use and rate. Page space should alert if over 70%. Page rate will be based on your paging subsystem. (ie - if 1 device, rate of 1000, if 100 devices, 100,000). This needs quick action or the system will crash if it runs out of page space.

The storage subsystem analysis should be done top down:

z/VM - how the overall system is set up/running.
Virtual machines - how users/servers are set up/running.
VDISK/MDC/Address spaces - how the system processes are set up/running.
Linux servers/processes - how the Linux servers are running.

Helpful ESAMON screens/ESAMAP reports:

ESASTRC - Main Storage Configuration - Shows the configuration information for the main storage area.
ESASTR1 - Main Storage Analysis - Shows information about the dynamic paging area.
ESASTR2 - Main Storage DPA Analysis - Shows additional information about the dynamic paging area.
ESAXACT - Transaction delay analysis - shows an analysis of virtual machine states and wait states.
ESAUSPG - User Storage Analysis - Shows information about user storage utilization.

Using zVPS to find information for solving issues with the storage/paging level:

ESASTRC - Shows storage configuration information.

Offline storage - This shows if there main storage frames that are not online. If this number is not zero, investigation is needed.

ESASTR1 - Shows storage analysis information.

System Storage - This shows the amount of storage in the DPA area.

Available <2gb - This shows the amount of storage below the 2GB line. Many critical z/VM system tasks need to use storage below the 2GB line. Watch for large depletions.

Systm ExSpc - This is system execution space allocated (CP's virtual address space). Look for any large fluctuations. Fluctuations could be caused by multiple large virtual machines logging on. To drill down - use ESASXS screen.

User Resdnt - This shows the megabytes of user resident storage. This should be the major use. Look for any large fluctuations. To drill down - use ESAUSPG/ESAUSP2 screens.

NSS/DCSS Resident - This shows the megabytes of NSS and DCSS resident in storage. Again, look for large fluctuations. To drill down - use ESADCSS screen.

AddSpace - This shows the System and User owned resident space. To drill down - use ESAASPC screen.

VDISK Rsdnt - This shows the virtual disk storage in use. To drill down - use ESAVDSK/ESAUSPG screens.

MDC Rsdnt - This shows the minidisk cache resident storage. To drill down - use ESAMDC screen.

ESASTR2 - Shows additional storage analysis information.

Avail List Empty/Sec - This shows times the available list was empty. This means there were no pages in storage that were available to satisfy new storage requirements.

Demand Scans - This shows the number of times a demand scan could not obtain sufficient page frames. Both of these can indicate issues.

ESAXACT - Transaction delay analysis. This can show if users are waiting on paging operations. Helpful storage/paging information:

Pag - This shows the percentage of time a a user is in page wait. This would indicate a storage issue.

ESAUSPG - Shows user storage information. Both screen and report samples:

Total >2GB <2GB - This shows total storage above and below the 2GB line by group, then by members of the group. (This is user storage only so will show less than ESASTR1.) All of the Total storage is backed up by z/VM in real storage. This is helpful to show which users are consuming the most storage (especially <2G).

Paged out - This shows the amount of storage that is paged out, which means the data had to be written out to disk. Look for large fluctuations in these numbers.

VirtDisk - This is storage resident in virtual disk address spaces. Look for spikes. This means that swap is being used more. (This is only a problem if storage constrained) If this number goes up, maybe the workload is growing and a server needs to be reconfigured or a workload is out of control. It is advantageous to get a baseline for the usual amount of VirtDisk usage, then set an alert if the number jumps significantly.

Locked MegaBytes - This shows locked storage in MB both above and below 2GB. If the combined number is over 4000, there are too many pages locked. Use reserved pages instead.

Conclusions

Review the options for reducing storage requirements before analyzing or enhancing the paging subsystem. Many times storage requirements can be reduced so that paging requirements drop significantly. If this is the case, any time spent on the paging subsystem will be wasted. Spooling rarely impacts performance. Have sufficient number of spool volumes to keep the device busy at less than 20% during peak periods.