Hi Frank, We had a zoom call with Al Bailey to troubleshoot the issues that we are observing. The bug where a single CPU was taking all of the workload is resolved. What we observed seems to be an issue with memory exhaust, additional information was gathered an added to this bug for further troubleshooting: https://bugs.launchpad.net/starlingx/+bug/1826308 If additional information is required, please, just let us know. Thanks & Regards, Cristopher Lemus From: "Miller, Frank" <Frank.Miller@windriver.com> Date: Thursday, April 25, 2019 at 8:24 AM To: "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424 Maria: It looks like the commit referenced yesterday [1] is not addressing the issue in your BM labs. Can you set up a live debug session so that some container SMEs can investigate? Frank [1] https://review.opendev.org/#/c/655240/ From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com] Sent: Thursday, April 25, 2019 12:12 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-24 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/>) Status: RED =========================================== Sanity Test is executed in a Containers – Bare Metal Environment AIO - Simplex Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [FAIL]| 40 TCs FAIL Sanity Platform 07 TCs [FAIL]| 07 TCs FAIL TOTAL: 57 TCS [Fail : 47] AIO – Duplex Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [FAIL] | 42 TCs FAIL Sanity Platform 05 TCs [FAIL] | 05 TCs FAIL TOTAL: 57 TCS [Fail : 47 TCs] Standard - Local Storage (2+2) Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: 57 TCS PASS Standard - Dedicated Storage (2+2+2) Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 05 TCs [PASS] TOTAL: 57 TCS PASS Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] Standard – Local Storage Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] Standard – Dedicated Storage Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] * some pods are failing during BM sanity execution. https://bugs.launchpad.net/starlingx/+bug/1826308 * Sanity Bare metal was tested with : http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145... * Sanity Virtual was tested with : http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T013... * Tomorrow in sanity virtual we will perform a double check with the latest ISO that includes the fixes. For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack Regards Maria G.