<div dir="auto">Can we consider the track of vm used by the running proces from /proc? we can work on a script using psstop(0) or other similar tool,what do you think. This might help us to find the process is consuming the memory over the time <div dir="auto"><div dir="auto"><br></div><div dir="auto">I also see the same problem of consuming almost 90% of the memory not only in all in one systems but also in duplex</div><div dir="auto"><br></div><div dir="auto">(0) <a href="https://github.com/clearlinux/psstop">https://github.com/clearlinux/psstop</a></div><div dir="auto"><br></div><div dir="auto">Regards </div><div dir="auto">Victor Rodriguez </div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 25, 2019, 21:59 Cordoba Malibran, Erich <<a href="mailto:erich.cordoba.malibran@intel.com">erich.cordoba.malibran@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, <br>
<br>
In this case we have: <br>
<br>
HugePages_Total: 34104<br>
HugePages_Free: 34104<br>
HugePages_Rsvd: 0<br>
HugePages_Surp: 0<br>
<br>
So, I'm not sure if it can be related with 1825814.<br>
<br>
Also, for people not seeing this issue, how much memory do you have in your baremetal systems? What's the minimum required memory for running an AIO system. Our failing system have 97 GB and free -h shows. <br>
<br>
total used free shared buff/cache available<br>
Mem: 93G 84G 3.2G 66M 5.6G 4.8G<br>
Swap: 0B 0B 0B<br>
<br>
<br>
A couple months ago I reported a similar issue[0], in that case after three days in stand-by the system started to throw Out of Memory errors. Does anyone has performed a longevity test for some days? Maybe the working systems might fail after a while if the memory usage keeps increasing over time. <br>
<br>
-Erich <br>
<br>
[0] <a href="http://lists.starlingx.io/pipermail/starlingx-discuss/2019-February/002923.html" rel="noreferrer noreferrer" target="_blank">http://lists.starlingx.io/pipermail/starlingx-discuss/2019-February/002923.html</a><br>
<br>
<br>
<br>
From: "Li, Cheng1" <<a href="mailto:cheng1.li@intel.com" target="_blank" rel="noreferrer">cheng1.li@intel.com</a>><br>
Date: Thursday, April 25, 2019 at 8:29 PM<br>
To: "Lemus Contreras, Cristopher J" <<a href="mailto:cristopher.j.lemus.contreras@intel.com" target="_blank" rel="noreferrer">cristopher.j.lemus.contreras@intel.com</a>>, "Miller, Frank" <<a href="mailto:Frank.Miller@windriver.com" target="_blank" rel="noreferrer">Frank.Miller@windriver.com</a>>, "Perez Ibarra, Maria G" <<a href="mailto:maria.g.perez.ibarra@intel.com" target="_blank" rel="noreferrer">maria.g.perez.ibarra@intel.com</a>>, "<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">starlingx-discuss@lists.starlingx.io</a>" <<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">starlingx-discuss@lists.starlingx.io</a>><br>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424<br>
<br>
Actually, I had also reported the memory issue[1] days ago. <br>
Memory exhaust happens because so little 4K memory is allocated for system/software load.<br>
<br>
[1] <a href="https://bugs.launchpad.net/starlingx/+bug/1825814" rel="noreferrer noreferrer" target="_blank">https://bugs.launchpad.net/starlingx/+bug/1825814</a><br>
<br>
Thanks,<br>
Cheng<br>
<br>
From: Lemus Contreras, Cristopher J [mailto:<a href="mailto:cristopher.j.lemus.contreras@intel.com" target="_blank" rel="noreferrer">cristopher.j.lemus.contreras@intel.com</a>] <br>
Sent: Friday, April 26, 2019 1:50 AM<br>
To: Miller, Frank <<a href="mailto:Frank.Miller@windriver.com" target="_blank" rel="noreferrer">Frank.Miller@windriver.com</a>>; Perez Ibarra, Maria G <<a href="mailto:maria.g.perez.ibarra@intel.com" target="_blank" rel="noreferrer">maria.g.perez.ibarra@intel.com</a>>; <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">starlingx-discuss@lists.starlingx.io</a><br>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424<br>
<br>
Hi Frank,<br>
<br>
We had a zoom call with Al Bailey to troubleshoot the issues that we are observing. The bug where a single CPU was taking all of the workload is resolved. <br>
<br>
What we observed seems to be an issue with memory exhaust, additional information was gathered an added to this bug for further troubleshooting: <a href="https://bugs.launchpad.net/starlingx/+bug/1826308" rel="noreferrer noreferrer" target="_blank">https://bugs.launchpad.net/starlingx/+bug/1826308</a><br>
<br>
If additional information is required, please, just let us know.<br>
<br>
Thanks & Regards,<br>
<br>
Cristopher Lemus<br>
<br>
From: "Miller, Frank" <mailto:<a href="mailto:Frank.Miller@windriver.com" target="_blank" rel="noreferrer">Frank.Miller@windriver.com</a>><br>
Date: Thursday, April 25, 2019 at 8:24 AM<br>
To: "Perez Ibarra, Maria G" <mailto:<a href="mailto:maria.g.perez.ibarra@intel.com" target="_blank" rel="noreferrer">maria.g.perez.ibarra@intel.com</a>>, "mailto:<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">starlingx-discuss@lists.starlingx.io</a>" <mailto:<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">starlingx-discuss@lists.starlingx.io</a>><br>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424<br>
<br>
Maria:<br>
<br>
It looks like the commit referenced yesterday [1] is not addressing the issue in your BM labs. Can you set up a live debug session so that some container SMEs can investigate?<br>
<br>
Frank<br>
[1] <a href="https://review.opendev.org/#/c/655240/" rel="noreferrer noreferrer" target="_blank">https://review.opendev.org/#/c/655240/</a><br>
<br>
From: Perez Ibarra, Maria G [mailto:<a href="mailto:maria.g.perez.ibarra@intel.com" target="_blank" rel="noreferrer">maria.g.perez.ibarra@intel.com</a>] <br>
Sent: Thursday, April 25, 2019 12:12 AM<br>
To: mailto:<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">starlingx-discuss@lists.starlingx.io</a><br>
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190424<br>
<br>
Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-24 (<a href="http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/" rel="noreferrer noreferrer" target="_blank">http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/</a>)<br>
<br>
Status: RED<br>
<br>
===========================================<br>
<br>
Sanity Test is executed in a Containers – Bare Metal Environment<br>
<br>
AIO - Simplex<br>
<br>
Setup Manual [PASS]<br>
Provisioning 01 TCs [PASS]<br>
Sanity OpenStack 49 TCs [FAIL]| 40 TCs FAIL<br>
Sanity Platform 07 TCs [FAIL]| 07 TCs FAIL <br>
<br>
TOTAL: 57 TCS [Fail : 47]<br>
<br>
AIO – Duplex<br>
<br>
Setup Manual [PASS]<br>
Provisioning 01 TCs [PASS]<br>
Sanity OpenStack 52 TCs [FAIL] | 42 TCs FAIL<br>
Sanity Platform 05 TCs [FAIL] | 05 TCs FAIL<br>
<br>
TOTAL: 57 TCS [Fail : 47 TCs]<br>
<br>
Standard - Local Storage (2+2)<br>
<br>
Setup Manual [PASS]<br>
Provisioning 01 TCs [PASS]<br>
Sanity OpenStack 49 TCs [PASS] <br>
Sanity Platform 07 TCs [PASS]<br>
<br>
TOTAL: 57 TCS PASS<br>
<br>
Standard - Dedicated Storage (2+2+2)<br>
<br>
Setup Manual [PASS]<br>
Provisioning 01 TCs [PASS] <br>
Sanity OpenStack 52 TCs [PASS] <br>
Sanity Platform 05 TCs [PASS]<br>
<br>
TOTAL: 57 TCS PASS <br>
<br>
<br>
<br>
Sanity Test is executed in a Containers - Virtual Environment<br>
<br>
AIO - Simplex<br>
<br>
Setup 04 TCs [PASS] <br>
Provisioning 01 TCs [FAIL]<br>
Sanity OpenStack 49 TCs [FAIL] <br>
Sanity Platform 07 TCs [FAIL]<br>
<br>
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]<br>
<br>
<br>
AIO - Duplex<br>
<br>
Setup 04 TCs [PASS] <br>
Provisioning 01 TCs [FAIL]<br>
Sanity OpenStack 49 TCs [FAIL] <br>
Sanity Platform 07 TCs [FAIL]<br>
<br>
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]<br>
<br>
<br>
Standard – Local Storage<br>
<br>
Setup 04 TCs [PASS] <br>
Provisioning 01 TCs [FAIL]<br>
Sanity OpenStack 49 TCs [FAIL] <br>
Sanity Platform 07 TCs [FAIL]<br>
<br>
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]<br>
<br>
<br>
Standard – Dedicated Storage<br>
<br>
Setup 04 TCs [PASS] <br>
Provisioning 01 TCs [FAIL]<br>
Sanity OpenStack 49 TCs [FAIL] <br>
Sanity Platform 07 TCs [FAIL]<br>
<br>
TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs]<br>
<br>
- some pods are failing during BM sanity execution. <a href="https://bugs.launchpad.net/starlingx/+bug/1826308" rel="noreferrer noreferrer" target="_blank">https://bugs.launchpad.net/starlingx/+bug/1826308</a><br>
- Sanity Bare metal was tested with : <a href="http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/" rel="noreferrer noreferrer" target="_blank">http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T145954Z/</a><br>
- Sanity Virtual was tested with : <a href="http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T013000Z/" rel="noreferrer noreferrer" target="_blank">http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190424T013000Z/</a> <br>
- Tomorrow in sanity virtual we will perform a double check with the latest ISO that includes the fixes. <br>
<br>
For more detail of the tests: <a href="https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack" rel="noreferrer noreferrer" target="_blank">https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack</a><br>
<br>
<br>
Regards<br>
Maria G.<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank" rel="noreferrer">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" rel="noreferrer noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote></div>