[Starlingx-discuss] [Containers] Sanity Test - ISO 20190424

Cordoba Malibran, Erich erich.cordoba.malibran at intel.com
Fri Apr 26 02:59:27 UTC 2019


Hi, 

In this case we have: 

HugePages_Total: 34104
HugePages_Free: 34104
HugePages_Rsvd: 0
HugePages_Surp: 0

So, I'm not sure if it can be related with 1825814.

Also, for people not seeing this issue, how much memory do you have in your baremetal systems? What's the minimum required memory for running an AIO system. Our failing system have 97 GB and free -h shows. 

                    total        used        free      shared  buff/cache   available
Mem:            93G         84G        3.2G         66M        5.6G        4.8G
Swap:            0B          0B          0B


A couple months ago I reported a similar issue[0], in that case after three days in stand-by the system started to throw Out of Memory errors. Does anyone has performed a longevity test for some days? Maybe the working systems might fail after a while if the memory usage keeps increasing over time. 

-Erich 

[0] http://lists.starlingx.io/pipermail/starlingx-discuss/2019-February/002923.html

 

From: "Li, Cheng1" <cheng1.li at intel.com>
Date: Thursday, April 25, 2019 at 8:29 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>, "Miller, Frank" <Frank.Miller at windriver.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com>, "starlingx-discuss at lists.starlingx.io" <starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424

Actually, I had also reported the memory issue[1] days ago. 
Memory exhaust happens because so little 4K memory is allocated for system/software load.
 
[1] https://bugs.launchpad.net/starlingx/+bug/1825814
 
Thanks,
Cheng
 
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com] 
Sent: Friday, April 26, 2019 1:50 AM
To: Miller, Frank <Frank.Miller at windriver.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra at intel.com>; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424
 
Hi Frank,
 
We had a zoom call with Al Bailey to troubleshoot the issues that we are observing. The bug where a single CPU was taking all of the workload is resolved. 
 
What we observed seems to be an issue with memory exhaust, additional information was gathered an added to this bug for further troubleshooting: https://bugs.launchpad.net/starlingx/+bug/1826308
 
If additional information is required, please, just let us know.
 
Thanks & Regards,
 
Cristopher Lemus
 
From: "Miller, Frank" <mailto:Frank.Miller at windriver.com>
Date: Thursday, April 25, 2019 at 8:24 AM
To: "Perez Ibarra, Maria G" <mailto:maria.g.perez.ibarra at intel.com>, "mailto:starlingx-discuss at lists.starlingx.io" <mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424
 
Maria:
 
It looks like the commit referenced yesterday [1] is not addressing the issue in your BM labs.  Can you set up a live debug session so that some container SMEs can investigate?
 
Frank
[1] https://review.opendev.org/#/c/655240/
 
From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com] 
Sent: Thursday, April 25, 2019 12:12 AM
To: mailto:starlingx-discuss at lists.starlingx.io
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424
 
Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-24 (http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/)
 
Status: RED
 
===========================================
 
Sanity Test is executed in a Containers – Bare Metal Environment
 
AIO - Simplex
 
Setup             Manual [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  49 TCs [FAIL]| 40 TCs FAIL
Sanity Platform   07 TCs [FAIL]| 07 TCs FAIL 
 
TOTAL: 57 TCS [Fail : 47]
 
AIO – Duplex
 
Setup             Manual [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  52 TCs [FAIL] | 42 TCs FAIL
Sanity Platform   05 TCs [FAIL] | 05 TCs FAIL
 
TOTAL: 57 TCS [Fail : 47 TCs]
 
Standard - Local Storage (2+2)
 
Setup             Manual [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  49 TCs [PASS] 
Sanity Platform   07 TCs [PASS]
 
TOTAL: 57 TCS PASS
 
Standard - Dedicated Storage (2+2+2)
 
Setup             Manual [PASS]
Provisioning      01 TCs [PASS] 
Sanity OpenStack  52 TCs [PASS] 
Sanity Platform   05 TCs [PASS]
 
TOTAL: 57 TCS PASS 
 
 
 
Sanity Test is executed in a Containers - Virtual Environment
 
AIO - Simplex
 
Setup             04 TCs [PASS]  
Provisioning      01 TCs [FAIL]
Sanity OpenStack  49 TCs [FAIL] 
Sanity Platform   07 TCs [FAIL]
 
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]
 
 
AIO - Duplex
 
Setup             04 TCs [PASS]  
Provisioning      01 TCs [FAIL]
Sanity OpenStack  49 TCs [FAIL] 
Sanity Platform   07 TCs [FAIL]
 
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]
 
 
Standard – Local Storage
 
Setup             04 TCs [PASS]  
Provisioning      01 TCs [FAIL]
Sanity OpenStack  49 TCs [FAIL] 
Sanity Platform   07 TCs [FAIL]
 
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]
 
 
Standard – Dedicated Storage
 
Setup             04 TCs [PASS]  
Provisioning      01 TCs [FAIL]
Sanity OpenStack  49 TCs [FAIL] 
Sanity Platform   07 TCs [FAIL]
 
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]
 
- some pods are failing during BM sanity execution. https://bugs.launchpad.net/starlingx/+bug/1826308
- Sanity Bare metal was tested with : http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/
- Sanity Virtual was tested with : http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T013000Z/ 
- Tomorrow in sanity virtual we will perform a double check with the latest ISO that includes the fixes. 
 
For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack
 
 
Regards
Maria G.
 
 
 



More information about the Starlingx-discuss mailing list