This page explains docker from a performance perspective. The mysterious containers are not so mysterious after looking at the numbers.
Docker containers reside inside Linux servers. What they perceive is "virtualized". But from the Linux server, everything running in the container is very visible.
The first step in understanding how it works is to measure it. A Docker container can be assigned an IP address - or a port from the host Linux server. To utilize zVPS to monitor and manage docker containers, snmp would be installed with the Velocity Software mib in each container, and then the container is monitored as any IP addressable server.
For the following anaysis, there is one docker container with node name "rksctnr1". This container lives in the "sles12" server. The sles12 server is a single virtual cpu server that also provides our OSA Data Capture. These two node names are the reporting entities in this analysis.
Inside the container runs snmp, and a looper process to drive load. From an overall system perspective, resources available to the server also appear to be available to the container. As load goes up on the container, we see exactly the same load on the server. In this analysis, the container was stopped for 45 minutes during the experiment to ensure the with and without pictures are clear
Report: ESALNXS LINUX VSI System Analysis Report Velocity Software Corporate ZMAP 4.3.0
--------------------------------------------------------------------------------------------------------------------------------
<-------------Rates per second-------------->
Node/ <---Load Numbers--> CPU >Processor Pct Util> NICE >CPU Overhead%> IO <-Blocks> <-Swaps-> >PageFlts>
Time Users Procs MaxProc NBR Total Syst User Idle Time Krnl IRQ Steal Wait IN OUT IN OUT MIN MAJ Intrpts CSwitch
-------- ----- ----- ------- --- ----- ---- ---- ---- ---- ---- ---- ----- ---- ----- ---- ---- --- ---- ---- ------- -------
11:00:00
rksctnr1 0 0 0 Tot 0.1 0.0 0.0 6.5 0 0.0 0.0 0.1 0.0 6.8 2.5 0 0 0 0 2.0 1.9
sles12 2 102 0 Tot 0.8 0.4 0.4 98.1 0 0.0 0.0 1.1 0.0 42.5 65.0 0 0 15.9 0 31.1 34.5
--------------------------------------------------------------------------------------------------------------------------------
11:15:00
rksctnr1 0 2 0 Tot 23.3 0.5 22.7 51.6 0 0.0 0.0 5.1 0.0 1.3 53.6 0 0 0.1 0 48.4 26.4
sles12 2 103 0 Tot 22.8 0.6 22.1 72.0 0 0.0 0.0 5.1 0.0 1.3 76.4 0 0 16.3 0 54.8 33.1
--------------------------------------------------------------------------------------------------------------------------------
11:30:00
sles12 2 98 0 Tot 1.0 0.6 0.4 97.7 0 0.0 0.0 1.3 0.0 0 34.4 0 0 16.1 0 29.2 27.1
--------------------------------------------------------------------------------------------------------------------------------
11:45:00
sles12 2 97 0 Tot 1.0 0.6 0.4 97.6 0 0.0 0.0 1.4 0.0 0 35.1 0 0 16.1 0 29.0 26.1
--------------------------------------------------------------------------------------------------------------------------------
12:00:00
sles12 2 100 0 Tot 7.4 1.2 6.1 90.3 0 0.0 0.1 2.2 0.2 151.1 656 0 0 16.1 0 56.5 80.8
--------------------------------------------------------------------------------------------------------------------------------
12:15:00
rksctnr1 0 3 0 Tot 13.2 0.5 12.7 56.7 0 0.0 0.0 3.4 0.0 5.8 47.7 0 0 1.7 0 36.7 23.7
sles12 2 102 0 Tot 13.2 0.7 12.4 83.0 0 0.0 0.0 3.7 0.1 36.2 116 0 0 16.0 0 46.4 34.4
--------------------------------------------------------------------------------------------------------------------------------
12:30:00
rksctnr1 0 5 0 Tot 21.4 0.7 20.7 72.9 0 0.0 0.0 5.7 0.0 0 40.5 0 0 2.5 0 53.0 30.1
sles12 2 104 0 Tot 21.8 0.7 21.1 72.5 0 0.0 0.0 5.7 0.0 0 40.4 0 0 16.1 0 53.0 30.0
--------------------------------------------------------------------------------------------------------------------------------
12:45:00
rksctnr1 0 4 0 Tot 20.5 0.6 19.8 67.8 0 0.0 0.0 5.1 0.0 11.0 74.4 0 0 2.9 0 51.5 33.7
sles12 2 105 0 Tot 19.5 0.7 18.8 75.4 0 0.0 0.0 5.0 0.0 10.7 89.5 0 0 16.1 0 53.5 37.4
--------------------------------------------------------------------------------------------------------------------------------
From a process perspective, it is interesting to see how processes are reported back. On the ESALNXP report, processes using less than .1% of the 15 minute period are not reported.
In looking at the processes at the 11:15 interval, the stresser process was running to drive a load. From the container perspective, this process is process ID 22. From the host perspective, this process is process ID 37173. After the break, the container calls the process ID 22 again, but from a host perspective, this is a new container and a new set of processes, and now the host sees the process as 38218.
What this proves is that each container can have it's own process ID, and several containers could run and all have the same process ID, but the host will assign each one a unique process ID to be managed at the host level.
The snmpd proces is running that the host level -
Report: ESALNXP LINUX HOST Process Statistics Report Velocity Software Corporate ZMAP 4.3.0 -------------------------------------------------------------------------------------------------------------------------------- node/ <-Process Ident-> Nice PRTY <------CPU Percents----> <---------------Storage Metrics (MB)----------> <-Faults/Second-> Name ID PPID GRP Valu Valu Tot sys user syst usrt Size RSS Peak Swap Data Stk EXEC Lib Lck PTbl min maj mint majt --------- ----- ----- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- --- --- ---- --- --- ---- ---- 11:00:00 sles12 0 0 0 0 0 1.05 0.49 0.39 0.04 0.14 3398 203 45K 0 36K 62.6 1154 2K 0.9 18.8 70K 0 14K 2 snmpd 1485 1 1482 -10 10 0.55 0.26 0.29 0 0 21 10 373 0 83.9 1.5 0.3 138 0 0.49 184 0 0 0 11:15:00 rksctnr1 0 0 0 0 0 19.0 0.03 0.02 0.03 18.9 31 11 396 0 154 4.4 13.4 192 0 0.80 12 0 21 0 stresser 22 1 1 0 20 18.9 0.00 0 0.03 18.9 4 2 32.3 0 2.11 1.2 5.6 22 0 0.12 1 0 1 sles12 0 0 0 0 0 20.4 0.72 0.99 0.04 18.6 3486 209 0 0 0 0 0 0 0 0 310 0 72 0 snmpd 1485 1 1482 -10 10 0.94 0.55 0.38 0 0 21 10 0 0 0 0 0 0 0 0 255 0 0 0 stresser 37173 37096 0 0 20 18.6 0.00 0 0.03 18.6 4 2 0 0 0 0 0 0 0 0 2 0 2 11:30:00 sles12 0 0 0 0 0 1.23 0.72 0.48 0.01 0.03 2916 181 0 0 0 0 0 0 0 0 302 0 53 0 snmpd 1485 1 1482 -10 10 1.07 0.62 0.44 0 0 21 10 0 0 0 0 0 0 0 0 299 0 0 0 11:45:00 sles12 0 0 0 0 0 1.21 0.73 0.48 0.00 0.00 2919 181 0 0 0 0 0 0 0 0 300 0 18 0 snmpd 1485 1 1482 -10 10 1.07 0.63 0.44 0 0 21 10 0 0 0 0 0 0 0 0 295 0 0 0 12:00:00 sles12 0 0 0 0 0 7.61 0.97 0.76 0.27 5.61 2923 193 0 0 0 0 0 0 0 0 347 0 461 0 systemd 1 1 1 0 20 1.84 0 0.00 0.00 1.83 1 1 0 0 0 0 0 0 0 0 0 0 0 0 containe 1289 1 1289 0 20 1.58 0.00 0.00 0.22 1.36 668 10 0 0 0 0 0 0 0 0 1 0 334 0 snmpd 1485 1 1482 -10 10 1.04 0.61 0.43 0 0 21 10 0 0 0 0 0 0 0 0 299 0 0 0 bash 32497 32494 32497 0 20 2.38 0.00 0.00 0.01 2.37 7 4 0 0 0 0 0 0 0 0 1 0 20 0 12:15:00 rksctnr1 0 0 0 0 0 10.8 0.03 0.02 0.02 10.7 34 12 310 0 111 4.6 16.4 169 0 0.69 12 0 9 0 stresser 22 17 1 0 20 10.7 0 0 0.02 10.7 4 2 28.7 0 1.91 1.1 5.0 19 0 0.09 0 0 sles12 0 0 0 0 0 14.2 1.93 1.25 0.03 11.0 3477 217 0 0 0 0 0 0 0 0 371 0 78 0 snmpd 1485 1 1482 -10 10 2.60 1.52 1.08 0 0 21 10 0 0 0 0 0 0 0 0 305 0 0 0 stresser 38218 38213 0 0 20 10.9 0 0 0.01 10.9 4 2 0 0 0 0 0 0 0 0 1 0 12:30:00 rksctnr1 0 0 0 0 0 20.6 0.04 2.57 0.03 18.0 37 12 560 0 187 10.1 28.8 317 0 1.34 23 0 15 0 stresser 22 17 1 0 20 18.0 0.00 0 0.03 18.0 4 2 53.8 0 3.57 2.0 9.3 36 0 0.18 1 0 1 stress 90 89 1 0 20 0.57 0 0.57 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 stress 99 98 1 0 20 0.54 0 0.54 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 stress 126 125 1 0 20 0.51 0 0.51 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 sles12 0 0 0 0 0 21.2 0.77 0.48 0.03 20.0 3758 224 0 0 0 0 0 0 0 0 321 0 30 0 snmpd 1485 1 1482 -10 10 1.01 0.60 0.42 0 0 21 10 0 0 0 0 0 0 0 0 295 0 0 0 stresser 38218 38213 0 0 20 20.0 0.00 0 0.03 20.0 4 2 0 0 0 0 0 0 0 0 1 0 1 12:45:00 rksctnr1 0 0 0 0 0 20.4 0.05 3.53 0.03 16.8 39 13 540 0 175 10.2 28.2 310 0 1.32 26 0 13 0 stresser 22 17 1 0 20 16.8 0.00 0.00 0.03 16.8 4 2 50.2 0 3.34 1.9 8.7 34 0 0.16 1 0 1 stress 135 134 1 0 20 0.50 0 0.50 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 stress 144 143 1 0 20 0.50 0 0.50 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 stress 172 171 1 0 20 0.58 0 0.58 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 stress 181 180 1 0 20 0.51 0 0.51 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 stress 204 203 1 0 20 0.58 0 0.58 0 0 2 0 2.5 0 0.05 0.1 0.0 2.2 0 0.01 0 0 0 0 sles12 0 0 0 0 0 19.5 0.76 0.52 0.04 18.2 3681 228 0 0 0 0 0 0 0 0 353 0 130 0 snmpd 1485 1 1482 -10 10 1.01 0.60 0.41 0 0 21 10 0 0 0 0 0 0 0 0 287 0 0 0 stresser 38218 38213 0 0 20 18.2 0.00 0.00 0.02 18.1 4 2 0 0 0 0 0 0 0 0 1 0 1
The process table inside the container is indeed very simple. This is the real time process analysis of the container, and how the host has mapped those processes. Note a one for one mapping of processes. This real time analysis is a different time period than the previous one and proces ID's have changed.
Screen: ESALNXC Velocity Software - VSIVM4 ESAMON 4.242 11/10 17:42-
1 of 3 Linux Process Configuration NODE RKSSCTNR1
<------ Process ------> Appl >Stor
Node Process Name ID PPID Group Appl Name Size
-------- ------------------------- ----- ----- ----- ----- -------- -----
rksctnr1 bash 1 1 1 1 bash .
snmpd 4560 1 4559 4560 snmpd 25419
stresser.sh 4574 1 1 4563 stresser 3663
sleep 6626 4574 1 6626 sleep .
====>
Screen: ESALNXC Velocity Software - VSIVM4 ESAMON 4.242 11/10 17:42-
1 of 3 Linux Process Configuration NODE SLES12
<------ Process ------> Appl >Stor
Node Process Name ID PPID Group Appl Name Size
-------- ------------------------- ----- ----- ----- ----- -------- -----
sles12 sshd 47050 1259 0 52635 sshd .
bash 47053 47050 0 52466 bash .
sshd 52635 1259 0 52635 sshd 12131
bash 52638 52635 0 52466 bash .
docker 59375 52638 0 59375 docker 293K
containerd 1289 1 1289 52452 containe 684K
containerd-shim 52830 1289 0 52452 containe .
bash 52844 52830 0 52466 bash .
snmpd 57662 52844 0 57662 snmpd 25415
stresser.sh 57683 52844 0 57669 stresser 3663
stress 61039 57683 52844 61040 stress
stress 61040 61039 52844 61040 stress 254
master 1489 1 1489 1489 master 529
qmgr 1524 1489 1489 1524 qmgr
pickup 52904 1489 1489 52326 pickup
docker 7195 1 7195 59375 docker 930
exe 52825 7195 7195 52447 exe
systemd 32388 1 32388 32388 systemd
(sd-pam) 32389 32388 32388 32389 (sd-pam)
snmpd 52318 1 0 57662 snmpd 1939
osasnmpd 52322 1 0 52094 osasnmpd
osasnmpd 52323 52322 0 52094 osasnmpd 1055
From a storage perspective, the metrics coming from both perspectives show little virtualization, the numbers appear to be identical. In reality, the container has access to the host resources in the same way any other process would, and there is no virtualization of storage. Containers "fill up" the same way the Linux server does, it "fills up" when the server itself is out of memory.
Report: ESAUCD2 LINUX UCD Memory Analysis Report Velocity Software Corporate ZMAP 4.3.0 -------------------------------------------------------------------------------------------------------------------------------- Node/ <-------------------------Storage Sizes (in MegaBytes)--------------------------> Time/ <--Real Storage--> <-----SWAP Storage----> Total <-------Storage in Use---------> Error Date Total Avail Used Total Avail Used MIN Avail CMM Buffer Cache Ovrhd Shared Message -------- ------ ----- ----- ----- ----- ----- ----- ----- ------ ------ ----- ----- ------ --------------- 11:00:00 rksctnr1 1897.6 139.4 1758 124.0 123.9 0.1 15.6 263.3 0 0.0 1501 257.1 0 sles12 1897.6 139.6 1758 124.0 123.9 0.1 15.6 263.5 0 0.0 1498 259.8 0 11:15:00 rksctnr1 1897.6 137.8 1760 124.0 123.9 0.1 15.6 261.6 0 0.0 1502 257.8 0 sles12 1897.6 145.8 1752 124.0 123.9 0.1 15.6 269.6 0 0.0 1498 253.3 0 11:30:00 sles12 1897.6 174.7 1723 124.0 123.9 0.1 15.6 298.5 0 0.0 1484 238.7 0 11:45:00 sles12 1897.6 176.4 1721 124.0 123.9 0.1 15.6 300.2 0 0.0 1484 237.0 0 12:00:00 sles12 1897.6 324.5 1573 124.0 123.9 0.1 15.6 448.4 0 0.0 1334 239.1 0 12:15:00 rksctnr1 1897.6 502.0 1396 124.0 123.9 0.1 15.6 625.9 0 0.0 1153 242.5 0 sles12 1897.6 511.6 1386 124.0 123.9 0.1 15.6 635.5 0 0.0 1149 237.3 0 12:30:00 rksctnr1 1897.6 507.6 1390 124.0 123.9 0.1 15.6 631.5 0 0.0 1153 236.9 0 sles12 1897.6 507.7 1390 124.0 123.9 0.1 15.6 631.5 0 0.0 1153 236.9 0 12:45:00 rksctnr1 1897.6 498.2 1399 124.0 123.9 0.1 15.6 622.1 0 0.0 1156 243.3 0 sles12 1897.6 501.7 1396 124.0 123.9 0.1 15.6 625.6 0 0.0 1154 241.9 0
From a file system perspective, all of the filesystems that the docker container sees as read/write are "virtualized", and have a different name at the host level.
Screen: ESAHST2 Velocity Software - VSIVM4 ESAMON 4.242 11/10 18:09-18:10
1 of 2 LINUX HOST Storage Analysis Report CLASS * NODE RKSCTN 2828 414C
<-Utilization->
Node/ Pct Alloc Storage
Time Group Size Used Full Units R/W Description
-------- -------- ----- ----- ---- ----- ---- --------------------------------
18:10:00 rksctnr1 124 0 0.1 1K Swap space
21 21 100 1K Shared memory
1376 1376 100 1K Cached memory
64 0 0 4K r/w /dev/shm
949 0 0 4K r/w /dev
1898 1 0.1 1K Memory buffers
2022 1615 79.9 1K Virtual memory
20276 11160 55.0 4K r/w /
1898 1615 85.1 1K Physical memory
F1=Help PF2=Zoom PF3=Quit PF4=Select PF5=Plot PF6=Reset
PF9=Sort PF10=Parms PF11=More PF12=Cancel
====>
Screen: ESAHST2 Velocity Software - VSIVM4 ESAMON 4.242 11/10 18:09-18:10
1 of 2 LINUX HOST Storage Analysis Report CLASS * NODE SLES12 2828 414C
<-Utilization->
Node/ Pct Alloc Storage
Time Group Size Used Full Units R/W Description
-------- -------- ----- ----- ---- ----- ---- --------------------------------
18:10:00 sles12 949 0 0.0 4K r/w /var/lib/docker/containers/7cfb1
64 0 0 4K r/w /var/lib/docker/containers/7cfb1
20276 11160 55.0 4K r/w /var/lib/pgsql
20276 11160 55.0 4K r/w /opt
20276 11160 55.0 4K r/w /var/opt
20276 11160 55.0 4K r/w /srv
20276 11160 55.0 4K r/w /tmp
20276 11160 55.0 4K r/w /var/lib/named
20276 11160 55.0 4K r/w /usr/local
When looking at multiple containers, the numbers add up. There are three containers, each have a different version of a stress program to create load. The sles12 node is the host.
Screen: ESALNXP Velocity Software - VSIVM4 ESAMON 4.242 11/15 20:33-20:34
1 of 3 VSI Linux Percent Usage by Process NODE RKS* 2828 414C7
<-Process Ident-> nice prty <-----CPU Percents----->
Time Node Name ID PPID GRP valu valu Tot sys user syst usrt
-------- -------- -------- ----- ----- ----- ---- ---- ---- ---- ---- ---- ----
20:34:00 rksctnr1 *Totals* 0 0 0 0 0 8.2 0.1 0.1 0.0 7.9
snmpd 26 1 25 0 20 0.2 0.1 0.1 0 0
stresser 28 1 28 0 20 7.9 0 0 0.0 7.9
stress 4495 4494 28 0 20 0.1 0 0.1 0 0
rksctnr2 *Totals* 0 0 0 0 0 8.1 0.0 0.0 4.7 3.3
snmpd 26 1 25 0 20 0.1 0.0 0.0 0 0
stresser 28 1 28 0 20 8.1 0 0 4.7 3.3
rksctnr3 *Totals* 0 0 0 0 0 9.7 0.9 0.1 8.7 0
snmpd 26 1 25 0 20 0.1 0.1 0.1 0 0
stresser 28 1 28 0 20 8.7 0 0 8.7 0
stress 4273 4272 28 0 20 0.8 0.8 0 0 0
====>
Screen: ESALNXP Velocity Software - VSIVM4 ESAMON 4.242 11/15 20:32-20:33
1 of 3 VSI Linux Percent Usage by Process NODE SLES12 2828 414C7
<-Process Ident-> nice prty <-----CPU Percents----->
Time Node Name ID PPID GRP valu valu Tot sys user syst usrt
-------- -------- -------- ----- ----- ----- ---- ---- ---- ---- ---- ---- ----
20:33:00 sles12 *Totals* 0 0 0 0 0 32.3 0.8 0.6 16.3 14.6
bash 1 1 1 0 20 0 0 0 0 0
snmpd 2774 1289 2773 0 20 0.3 0.2 0.1 0 0
stresser 2795 2781 2795 0 20 11.0 0 0 0.0 11.0
snmpd 3109 1289 3108 0 20 0.1 0.1 0.0 0 0
stresser 3168 3154 3168 0 20 8.8 0.0 0 5.2 3.6
snmpd 3475 1289 3474 0 20 0.1 0.1 0.1 0 0
stresser 3502 3488 3502 0 20 11.1 0.0 0.0 11.0 0.0
docker 7195 1 7195 0 20 0.0 0 0.0 0 0
kworker/ 16184 2 0 0 20 0.0 0.0 0 0 0
snmpd 63481 1 0 -10 10 0.7 0.3 0.4 0 0
osasnmpd 63485 1 0 0 20 0.1 0.1 0.0 0 0
Docker might seem mysterious at first, but it's really not rocket science, and it is easily understood when looking at how the host sees the resources, and how the container sees them. I'm sure there is more work to understand this, but no different than understanding linux performance and capacity planning in general.
