Many IBM presentations on IBM Secure Container have left the impression that this was a black box - "Black box" meaning that if there was a performance problem or a lack of resource or other things that could impact the workload, there was no mechanism for diagnosing the problem. Customers rarely want to trust important workload to a black box - as jobs are at risk when applications fail to meet requirements due to bad architectural decisions.
Velocity Software, in conjunction with the IBM Secure Container team, have developed an architecture and solution to this problem. The architecture of the IBM Secure Container does not allow for bi-directional requests as zVPS does to support Linux using snmp. However, an open source package, collectd, can be installed and tailored to send data to an IP address using the UDP protocol. zVPS has been enhanced to support this data feed, and now provides full performance management for IBM Secure Container using the collectd open source package.
zVPS will provide operational alerts, historical data, capacity planning information, and performance diagnostic capability to the IBM Secure Container using this architecture.
Installations wishing to test this function may actually point their collectd collector at the Velocity Software cloud and see their data - send a note to Barton if interested, and the correct IP address and port will be provided for your testing.
The collectd package is also useful for monitoring other docker container architectures as well. This is in addition to the docker mib as currently provided by Velocity Software.
The following examples show some of the data provided by collectd and some of the reporting provided by zVPS, the Velocity Software Performance Suite:
CPU Reporting:
1Report: ESASSCC Secure Container Report Velocity Software Corporate ZMAP 5.1.0 05/30/19 Page 1
Monitor initialized: 05/27/19 at 15:30:01 on 2828 serial 0314C7 First record analyzed: 05/27/19 15:31:00
--------------------------------------------------------------------------------------------------------------------------------
Container <-------------CPU Percents-----------> <-----Memory Used (GB)------> <-----Memory Used (Percent-->
User SYST Nice Intr SIRQ STL Wait Idle Used Buff Cach Free URcl Recl Used Buff Cach Free URcl Recl
----------------------- ---- ---- ---- ---- ---- --- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
15:32:00
cpca-mon1 2.6 2.8 0.0 1.3 1.2 1.3 0.0 3.3 28.7 17.7 37.6 40.0 24.8 30.3 2.3 1.4 3.0 3.2 1.9 2.5
sysdig_collectd_us_ash_c 1.8 1.5 0.0 1.3 0.6 0.0 0.0 3.4 18.5 0.0 16.4 22.8 13.5 14.0 2.8 0.0 2.5 3.3 1.9 2.0
--------------------------------------------------------------------------------------------------------------------------------
15:33:00
cpca-mon1 2.6 2.8 0.0 1.4 1.3 1.6 1.0 3.3 28.7 17.7 37.6 40.0 24.8 30.3 2.3 1.4 3.0 3.2 1.9 2.5
sysdig_collectd_us_ash_c 1.8 1.6 0.0 0.0 0.0 0.0 0.0 3.4 18.5 0.0 16.4 22.8 13.5 14.0 2.8 0.0 2.5 3.3 1.9 2.0
--------------------------------------------------------------------------------------------------------------------------------
15:34:00
cpca-mon1 2.6 2.8 0.0 1.4 1.4 1.6 0.0 3.3 28.7 17.7 37.6 40.0 24.8 30.3 2.3 1.4 3.0 3.2 1.9 2.5
sysdig_collectd_us_ash_c 1.7 1.5 0.0 1.3 0.0 1.3 0.0 3.4 18.5 0.0 16.4 22.8 13.5 14.0 2.8 0.0 2.5 3.3 1.9 2.0
--------------------------------------------------------------------------------------------------------------------------------
Process Reporting:
1Report: ESASSCH Secure Container Process Report Velocity Software Corporate ZMAP 5.1.0 05/30/19 Page 1 Monitor initialized: 05/27/19 at 15:30:01 on 2828 serial 0314C7 First record analyzed: 05/27/19 15:31:00 -------------------------------------------------------------------------------------------------------------------------------- Node <---Percents-------> <------------Time----------> Process core Dispatch Thread Mgmt Dispatch Thread Mgmt Online --------- ---- -------- ------ ---- -------- ------ ---- ------- 15:32:00 cpca-mon1 CSDEV1 0 1.89 1.90 1.24 0 0 0 1.00 CSSTAG1 0 1.97 1.99 1.65 1313.00 0 4.3G 1.00 CS1 0 1.39 1.39 1.03 4294967K 0 0 1.00 DBCIM1 0 1.76 1.78 1.24 0 0 0 1.00 DBCIW1 0 2.01 2.03 1.43 0 1.00 0 1.00 DBCIW3 0 2.01 2.04 1.36 0 0 0 1.00 DBM1 0 1.91 1.93 1.36 0 0 0 1.00 DBPFM1 0 1.59 1.63 1.16 0 0 0 1.00 DBPFW1 0 1.54 1.55 1.16 0 0 0 1.00 DBPFW3 0 1.56 1.56 1.03 4294967K 4295M 0 1.00 DBSREM1 0 1.92 2.02 1.43 4294967K 4295M 0 1.00 DBSREM2 0 1.90 1.91 1.32 0 0 0 1.00 DBSREW1 0 2.94 3.00 1.88 0 0 0 1.00 DBSREW2 0 3.03 3.21 1.90 0 1.00 0 1.00 DBSREW2 1 2.96 3.10 1.86 0 0 0 0 DBSTAGM1 0 1.52 1.57 1.03 0 0 0 1.00 DBTESTD 0 1.66 1.67 1.16 0 0 0 1.00 DBTESTM1 0 1.63 1.65 1.24 0 0 0 1.00 DBTESTW1 0 2.63 2.65 1.93 0 1.00 0 1.00 DBW1 0 1.49 1.50 1.03 0 0 0 1.00 MGMT1 0 1.73 1.76 1.16 0 4295M 0 1.00 MGMT2 0 1.89 1.89 1.16 22.00 27.00 0 1.00 MON1 0 3.05 3.10 1.98 0 0 4.3G 1.00 SRETEST 0 1.58 1.63 1.03 0 0 0 1.00 VMPF1 0 0 0 0 0 0 0 0 VMPF2 0 1.64 1.67 1.03 0 0 0 1.00 VMPF3 0 1.28 1.51 1.03 0 0 0 1.00 -------------------------------------------------------------------------------------------------------------------------------- 15:33:00 cpca-mon1 CSDEV1 0 1.66 1.67 1.16 0 0 2.00 1.00 CSSTAG1 0 1.92 1.93 1.32 0 0 0 1.00 CS1 0 1.41 1.41 1.03 0 0 0 1.00 DBCIM1 0 1.87 1.88 1.45 0 0 0 1.00 DBCIW1 0 2.62 2.68 1.76 0 0 0 1.00 DBCIW3 0 3.07 3.15 1.94 0 1.00 0 1.00 DBM1 0 2.04 2.06 1.45 0 0 0 1.00 DBPFM1 0 1.32 1.45 0.00 4294967K 0 0 1.00 DBPFW1 0 2.96 2.97 2.14 61.00 0 0 1.00 DBPFW3 0 1.43 1.47 1.03 0 0 0 1.00 DBSREM1 0 2.02 2.09 1.39 0 19.00 0 1.00 DBSREM2 0 1.92 1.94 1.52 0 0 0 1.00 DBSREW1 0 0 0 0 0 0 0 0 DBSREW2 0 1.97 2.01 1.47 4294967K 4295M 0 1.00 DBSREW2 1 2.96 3.10 1.86 0 0 0 0 DBSTAGM1 0 1.63 1.66 1.03 0 0 0 1.00 DBTESTD 0 1.70 1.74 1.16 0 0 0 1.00 DBTESTM1 0 0 0 0 0 0 0 0 DBTESTW1 0 1.63 1.64 1.16 0 0 0 1.00 DBW1 0 1.65 1.65 1.24 14.00 0 0 1.00 MGMT1 0 1.91 1.91 1.24 0 0 0 1.00 MGMT2 0 1.88 1.88 1.16 0 0 0 1.00 MON1 0 2.07 2.26 1.43 0 0 2.00 1.00 SRETEST 0 1.67 1.70 1.03 0 0 0 1.00 VMPF1 0 1.36 1.39 0.00 0 0 0 1.00 VMPF2 0 0 0 0 0 0 0 0 VMPF3 0 1.55 1.57 1.03 0 0 0 1.00 --------------------------------------------------------------------------------------------------------------------------------
Disk Reporting:
1Report: ESASSCD Secure Container Disk Report Velocity Software Corporate ZMAP 5.1.0 05/30/19 Page 1 Monitor initialized: 05/27/19 at 15:30:01 on 2828 serial 0314C7 First record analyzed: 05/27/19 15:31:00 -------------------------------------------------------------------------------------------------------------------------------- Node <----I/O---><--Time----> <--DiskIO---> DISKName Read Write Read Write Read Write Read Write -------- ----- ----- ----- ------ ----- ------ ------ ------ 15:32:00 cpca-mon1 sdc1 4295M 4291M 0.9 0.5 0.0 1.0 0 0 sdd 0 48.0 0 56149 0 0 0 0 sdd1 0 48.0 0 56149 0 0 0 0 dm-0 0 134.0 0 33074 0 0.1 0 0 o 0 15.0 0 0 0 0 0 0 dm-1 0 50.0 0 4096.0 0 1.2 10.0 1.0 dm-2 0 111.0 0 38082 0 0 0 0 dm-3 4295M 370M 0 0.0 0 0 0 0 dm-4 0 111.0 0 38082 0 0 0 0 dm-5 0 272.0 0 58157 0 0.0 20.0 0.1 sde 0 22.0 0 34071 0 0 0 0 sde1 0 3.0 0 4096.0 0 0 0 0 sde2 0 19.0 0 38804 0 0 0 0 sdh 0 39.0 0 69632 0 0.5 0 0 sdh1 0 39.0 0 69632 0 0.5 0 0 sdi 0 23.0 0 35617 0 0 0 0 sdi1 0 4.0 0 8192.0 0 0 0 0 sdi2 0 19.0 0 41391 0 0 0 0 sdl1 0 50.0 0 52019 0 0.5 0 0 sdm 0 24.0 0 37035 0 0 0 0 sdm1 0 5.0 0 7372.8 0 0 0 0 sdp1 0 45.0 0 59984 0 0 0 0 sds 0 23.0 0 27782 0 0 0 0 sds2 0 18.0 0 32540 0 0 0 0 sdr 0 45.0 0 58800 0 0 0 0 sdr1 0 45.0 0 58800 0 0 0 0 sdu 0 20.0 0 26419 0 0 0 0 sdu1 0 3.0 0 5461.3 0 0 0 0 sdu2 0 17.0 0 30118 0 0 0 0 sdx 0 45.0 0 54704 0 0 0 0 dm-7 0 272.0 0 58157 0 0.0 20.0 0.1 dm-8 0 272.0 0 58157 0 0.0 20.0 0.1 sda 0 37.0 0 33100 0 1.2 0 0 -------------------------------------------------------------------------------------------------------------------------------- 15:33:00 cpca-mon1 sda1 0 7.0 0 9362.3 0 21.4 0 0 sdd 0 24.0 0 62976 0 0 0 0 sdd1 0 24.0 0 62976 0 0 0 0 dm-0 0 159.0 0 27513 0 0.0 0 0 o 0 15.0 0 0 0 0 0 0 dm-1 0 46.0 0 4096.0 0 0.3 10.0 0.2 dm-2 0 140.0 0 29901 0 0 0 0 dm-3 13968 3925M 0 0.0 0 0 0 0 dm-4 0 140.0 0 29901 0 0 0 0 dm-5 0 141.0 0 59348 0 0.0 10.0 0.1 sde 0 26.0 0 30878 0 0 0 0 sde2 0 24.0 0 33109 0 0 0 0 sdh 0 25.0 0 52265 0 0 0 0 sdh1 0 25.0 0 52265 0 0 0 0 sdi 0 27.0 0 20480 0 0 0 0 sdi1 0 4.0 0 8192.0 0 0 0 0 sdi2 0 23.0 0 22617 0 0 0 0 sdl 0 22.0 0 59578 0 0 0 0 sdl1 0 22.0 0 59578 0 0 0 0 sdm 0 26.0 0 30247 0 1.0 0 0 sdm1 0 4.0 0 12288 0 37.5 0 0 sdm2 0 22.0 0 33513 0 0 0 0 sdp 0 24.0 0 64512 0 0.5 0 0 sdp1 0 24.0 0 64512 0 0.5 0 0 sds 0 27.0 0 22300 0 0 0 0 sds2 0 22.0 0 25507 0 0 0 0 sdr 0 23.0 0 61440 0 0.6 0 0 sdr1 0 23.0 0 61440 0 0.6 0 0 sds1 0 15.0 0 2730.7 0 0 0 0 sdu 0 26.0 0 30090 0 0 0 0 sdu1 0 3.0 0 15019 0 0 0 0 sdu2 0 23.0 0 32056 0 0 0 0 sdx 0 23.0 0 55563 0 0 0 0 dm-7 0 143.0 0 58518 0 0.0 10.0 0.1 dm-8 0 143.0 0 58518 0 0.0 10.0 0.1 sda 0 27.0 0 31403 0 0 0 0 --------------------------------------------------------------------------------------------------------------------------------
File System Reporting:
1Report: ESASSCF Secure Container File Systems Velocity Software Corporate ZMAP 5.1.0 05/30/19 Page 1 Monitor initialized: 05/27/19 at 15:30:01 on 2828 serial 0314C7 First record analyzed: 05/27/19 15:31:00 -------------------------------------------------------------------------------------------------------------------------------- Container <-----------Storage (GB) -------------> File System Size Free Reserved Used PCT ------------- ------ ------- -------- ------- ------- 15:32:00 cpca-mon1 hostfs 7.7 3.4 1.9 2.4 0.6 hostfs-v 0 0 0 0 0 -------------------------------------------------------------------------------------------------------------------------------- 15:33:00 cpca-mon1 hostfs 7.7 3.4 1.9 2.4 0.6 hostfs-v 8.1 3.2 1.9 3.1 0.6 -------------------------------------------------------------------------------------------------------------------------------- 15:34:00 cpca-mon1 hostfs 7.7 0 0 0 0 hostfs-v 8.1 3.2 1.9 3.1 0.6 --------------------------------------------------------------------------------------------------------------------------------
As always, Velocity Software is open to suggestions and requirements on managing your z performance.
