Linux Storage

Linux Storage Analysis

Linux Storage - Best Practices

Presentation on Linux configuration: Linux Storage Measurement and Tuning

Helpful Storage settings/information:

Storage issues:

Linux was designed for dedicated storage, which is very expensive.
Linux is LRU (Least Recently Used) which competes with VM's storage reference pattern.
When a Linux server logs on, it has a lot of initialization pages that won't be touched again. z/VM still has to back them up.
Linux tends to have timer pops that cause a large amount of data to be cached and then not referenced.
Spin locks can occur when storage that has a lock is paged out and another server is waiting, both having a large amount of storage in general. This can cause the system to start paging excessively.

Storage solutions:

Linux guest size - Do not define more storage than needed. Decrease the size of the machine until swapping occurs.
An example would be: Instead of 128M for a guest machine, define 24M and 100m swap.
If running Oracle (especially), defining HUGE pages is a benefit. Pages are by default 4K. HUGE pages are larger, usually 1M, 2M or 1G.
Linux swapping -

Use vdisk for swap (if using real disks, move to paging disks).
Virtual disks can be shared across virtual machines.
Start with 1 small vdisk (100M) and 1 larger vdisk (2G). Set the priority to favor the smaller vdisk.
Do not format the disks.
Set an alert if the larger vdisk is used.
Note: In Linux the lower number has the higher prioritization and will be used first.
Use $ swapon -s to verify the sizes of the two disks.
Check /etc/fstab to determine the priority of the two disks - the smaller should be set to pri=1.
Unused vdisk is paged out (goodness).
The cost of the swap daemon on Linux is about 10% but z/VM swapping to vdisk is very fast and low cost.
Use DIAG driver instead of FBA. FBA is 512 bytes. DIAG will group those into 4K pages which reduces I/O by a factor of 8.
Adjust the minidisk caching storage to SET MDC STORAGE 128M 128M
Set MDC OFF for minidisks being used for Linux file systems and swap disks to avoid redundant levels of caching.
Set the swappiness setting in linux to zero. Use $ cat /proc/sys/vm/swappiness to check the value.

Other helpful Storage information:

The Linux kernel tries to defer allocating dynamic memory to User Mode processes - those run in user space as opposed to kernel space.
Page faults take place when a process requests for an address that has been allocated to it but has not yet been assigned to it.
A minor page fault occurs when a process needs data that is in memory and is assigned to another process. Memory is shared between multiple processes - no additional data needs to be read from disk to memory.
A major page fault occurs when a process needs to read in data from disk to memory pages.
Swapping is when memory is unavailable for processes (vs disk cache).
Swappinness prefers pages of a user mode process over disk cache pages when reclaiming memory.
Inactive memory is the total amount of buffer or page cache memory that are free and available. Linux uses this cache to maintain data 'just in case'. Pages are reclaimed LRU (Least Recently Used). Inactive memory is 'extra', active storage is 'required'.
Anonymous pages are in User Mode heap. They have no named file system source. Swap size must be big enough to hold anonymous pages.
The kernel slab is memory being used for kernel mallocs. It helps eliminate fragmentation caused by allocations and deallocations.
With z/VM 6.3 and higher, page space utilization will be higher. Functional alerts on page space usage is advised.
VDISK has no overhead until 'touched'.
When a buffer is dirty, it means the data is stored in cache but needs to be written to the underlying storage device first because it has been modified.
Rule of thumb - if a server is not using swap, it is oversized. If a server is using some swap, it is sized correctly. If a server is using a lot of swap, it is undersized.

Helpful ESAMON screens/ESAMAP reports:

ESAUCD2 - Linux UCD Memory Analysis - Shows information about the real and virtual storage.
ESALNXR - Linux Memory Analysis - Shows information about Linux memory.
ESALNXP - Linux Usage by Process - Shows statistical information about the Linux host and processes.
ESAHST2 - Linux Host Storage Analysis - Shows information about the utilization of various types of storage.
ESAHST1 - Linux Host Software Analysis - Shows information about the software programs running.
ESAUSP2 - User Percent Utilization - Shows virtual machine utilization rate - main storage/paging.
ESAVDSK - VDISK Analysis - Shows VDISK paging activity.

If Linux storage is an issue, consider using zVRM

Using zVPS to find information for solving issues with the Linux storage:

ESAUCD2 - Shows the real and virtual storage for users.

Node/Goup - This shows the zTCP node name or node group.

Real Storage Avail - This shows available RAM not currently in use. If a server shows high availabilty of storage not in use, they may be over allocated (over 5%).

SWAP Storage Used - This shows SWAP space in use. Watch for high real storage availability with no swap.

Storage in Use Buffer - This shows storage in use for data buffers. Watch for greater than 20mb. If this is the case, may need to use CMM (see above) to change storage size on Linux.

Storage in Use Cache - This shows storage in use for page cache.

Error Message - This shows an error message if the threshold has been exceeded or some other error has occurred. (The mail server is a proprietary component and doesn't use swap space.)

ESALNXR - Shows the real and virtual storage for users.

Node/Goup - This shows the zTCP node name or node group.

Memory in Megabytes Cache Swap - This shows the amount of memory in swap cache.

Anonymous - This shows anonymous memory - pages in User Mode heap. (See above).

Kernel Slab - This shows kernel data structure use.

Page table - This shows the amount of memory used for page table structures.

Large Pages - This shows the memory for Linux Large Pages. Large Pages (also called Huge Pages) are beneficial to certain applications such as Oracle.

ESALNXP - Shows storage information for processes.

Storage Metrics Swap - This shows how much the process is using swap. If a process has a storage leak, it will show up here. Look for the proceses with the most storage swapped out and its storage keeps growing. It isn't giving anything back.

Storage Metrics PTbl - This shows the size of the Page Tables. If Large Pages is implemented on Linux, this number should go down and save a lot of storage.

ESAHST2 - Shows information about the utilization of various types of storage.

Node/Goup - This shows the zTCP node name or node group. Can click on a line (zview) or zoom (z/VM) to see the all the nodes in the group.

Utilization Pct Full - This shows the percent of storage area in use. If a file system is mostly read only and won't grow much, a large number here isn't a problem.

Err - This shows any failures on this storage. Watch for errors.

ESAHST1 - Shows information about the utilization of various types of storage.

Node/Group - This shows the zTCP node name where the process is running.

Name - This shows the name of the software package.

Type - This shows the type of program - OpSYS (Operating System), DevDrv (Device Driver), Applic (Application) or Unknown.

Status - This shows the status of the program - Running, ResWait (waiting on resources), Loaded (waiting for Interrupt) or Invalid.

Storage (K) Current - This shows the current amount of storage in use by this software.

ESAUSP2 - Shows more virtual machine main storage utilization information.

Resident Actv - This shows the active resident memory for each active user.

Pages/Sec - This shows how much paging is occuring. If the number of spool writes for a userid is over 100, this is excessive and should be reviewed.

ESAVDSK - Shows VDISK paging activity.

AddSpc Pages - This shows the average virtual disk space in pages that have been allocated.

Resident - This shows the number of address space pages that are resident. Watch the resident pages for large changes.

Conclusions

The Storage system has many performance knobs. It is worth the time to set it up properly and adjust early and often. Alerts and customized screen highlights are very helpful to quickly discover issues.