<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I *think* that documentation is stale since we are now using containerd instead of docker on StarlingX, but I might be wrong. Anybody have some thoughts about that?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for opening that bug.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thiago</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> open infra <openinfradn@gmail.com><br>
<b>Sent:</b> Wednesday, September 15, 2021 2:33 AM<br>
<b>To:</b> Brito, Thiago <Thiago.Brito@windriver.com><br>
<b>Cc:</b> starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io><br>
<b>Subject:</b> Re: [Starlingx-discuss] Openstack image upload failed with evicted pods</font>
<div> </div>
</div>
<div>
<p></p>
<div style="border:solid #9C6500 1.0pt; padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="x_MsoNormal" style="line-height:12.0pt; background:#FFEB9C"><b><span style="font-size:10.0pt; color:#9C6500"></span></b><span style="font-size:10.0pt; color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]</span></p>
</div>
<div>
<div dir="ltr">Hi Thiago,
<div><br>
</div>
<div>I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0].</div>
<div><a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809328/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmwKyMsji$">https://paste.opendev.org/show/809328/</a><br>
</div>
<div>Disk usage of kubelet controller-0: [1] and controller-1: [2]</div>
<div>Disk usage of controller-0 [3] and controller-1 [4].</div>
<div><br>
</div>
<div>There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9].</div>
<div>
<pre style="line-height:16.25px; margin-top:0px; margin-bottom:0px; padding:5px 0px; font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px; color:rgb(0,0,0)">The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6]</pre>
<pre style="line-height:16.25px; margin-top:0px; margin-bottom:0px; padding:5px 0px; font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px; color:rgb(0,0,0)"><br></pre>
</div>
<div>I was able to upload the image after increasing kubelet size to 50GB.</div>
<div><br>
</div>
<div>[0] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809328/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmwKyMsji$">
https://paste.opendev.org/show/809328/</a><br>
</div>
<div>[1] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809325/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmxg0Jhmj$">
https://paste.opendev.org/show/809325/</a></div>
<div>[2] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809324/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm41DIss_$">
https://paste.opendev.org/show/809324/</a></div>
<div>[3] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809326/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_HpCpom$">
https://paste.opendev.org/show/809326/</a><br>
</div>
<div>[4] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809323/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_SwV79i$">
https://paste.opendev.org/show/809323/</a><br>
</div>
<div><br>
</div>
<div>[5] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809320/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm8xkqOPE$">
https://paste.opendev.org/show/809320/</a><br>
</div>
<div>[6] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809319/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_cngqi-$">
https://paste.opendev.org/show/809319/</a></div>
<div>[7] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809318/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm8JC1vgQ$">
https://paste.opendev.org/show/809318/</a><br>
[8] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809321/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm5PvdFrb$">
https://paste.opendev.org/show/809321/</a><br>
</div>
<div>[9] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809322/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_tcuKJR$">
https://paste.opendev.org/show/809322/</a><br>
</div>
<div><br>
</div>
<div>I have created a LP as at least the stx documentation should advice to increase the kubelet size.</div>
<div>The documentation suggest to increase docker size set to 60GB instead of default 30GB.</div>
<div><a href="https://urldefense.com/v3/__https://bugs.launchpad.net/starlingx/*bug/1943674__;Kw!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm50bMs3j$">https://bugs.launchpad.net/starlingx/+bug/1943674</a><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Danishka</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <<a href="mailto:Thiago.Brito@windriver.com">Thiago.Brito@windriver.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Danishka,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
2) Check the eviction reason by looking at kubectl describe on the evicted pods</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Let me know.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thiago</div>
<div id="x_gmail-m_8425453646194598897appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_8425453646194598897divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>><br>
<b>Sent:</b> Tuesday, September 14, 2021 1:08 PM<br>
<b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">
starlingx-discuss@lists.starlingx.io</a> <<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> [Starlingx-discuss] Openstack image upload failed with evicted pods</font>
<div> </div>
</div>
<div>
<p></p>
<div style="border:1pt solid rgb(156,101,0); padding:2pt">
<p style="line-height:12pt; background:rgb(255,235,156)"><b><span style="font-size:10pt; color:rgb(156,101,0)"></span></b><span style="font-size:10pt; color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]</span></p>
</div>
<div>
<div dir="ltr">
<div><br>
</div>
<div>Hi,</div>
<div><br>
</div>
<div>In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. </div>
<div><br>
</div>
<div>Ceph cluster has enough space and health is ok [3]. </div>
<div><br>
</div>
<div>I can upload smaller files without any issue.</div>
<div><br>
</div>
<div>I am still trying to understand what went wrong. </div>
<div><br>
</div>
<div>[1] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$" target="_blank">
https://paste.opendev.org/show/809313/</a><br>
</div>
[2] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$" target="_blank">
https://paste.opendev.org/show/809312/</a><br>
<div>[3] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$" target="_blank">
https://paste.opendev.org/show/809314/</a><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Danishka</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm9mPx9SB$" rel="noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote>
</div>
</div>
</div>
</body>
</html>