<div dir="ltr">Hi Thiago,<div><br></div><div>I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0].</div><div><a href="https://paste.opendev.org/show/809328/">https://paste.opendev.org/show/809328/</a><br></div><div>Disk usage of kubelet controller-0: [1] and controller-1: [2]</div><div>Disk usage of controller-0 [3] and controller-1 [4].</div><div><br></div><div>There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9].</div><div><pre style="line-height:16.25px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6]</pre><pre style="line-height:16.25px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)"><br></pre></div><div>I was able to upload the image after increasing kubelet size to 50GB.</div><div><br></div><div>[0] <a href="https://paste.opendev.org/show/809328/">https://paste.opendev.org/show/809328/</a><br></div><div>[1] <a href="https://paste.opendev.org/show/809325/">https://paste.opendev.org/show/809325/</a></div><div>[2] <a href="https://paste.opendev.org/show/809324/">https://paste.opendev.org/show/809324/</a></div><div>[3] <a href="https://paste.opendev.org/show/809326/">https://paste.opendev.org/show/809326/</a><br></div><div>[4] <a href="https://paste.opendev.org/show/809323/">https://paste.opendev.org/show/809323/</a><br></div><div><br></div><div>[5] <a href="https://paste.opendev.org/show/809320/">https://paste.opendev.org/show/809320/</a><br></div><div>[6] <a href="https://paste.opendev.org/show/809319/">https://paste.opendev.org/show/809319/</a></div><div>[7] <a href="https://paste.opendev.org/show/809318/">https://paste.opendev.org/show/809318/</a><br>[8] <a href="https://paste.opendev.org/show/809321/">https://paste.opendev.org/show/809321/</a><br></div><div>[9] <a href="https://paste.opendev.org/show/809322/">https://paste.opendev.org/show/809322/</a><br></div><div><br></div><div>I have created a LP as at least the stx documentation should advice to increase the kubelet size.</div><div>The documentation suggest to increase docker size set to 60GB instead of default 30GB.</div><div><a href="https://bugs.launchpad.net/starlingx/+bug/1943674">https://bugs.launchpad.net/starlingx/+bug/1943674</a><br></div><div><br></div><div>Regards,</div><div>Danishka</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <<a href="mailto:Thiago.Brito@windriver.com">Thiago.Brito@windriver.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Danishka,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
2) Check the eviction reason by looking at kubectl describe on the evicted pods</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Let me know.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thiago</div>
<div id="gmail-m_8425453646194598897appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_8425453646194598897divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>><br>
<b>Sent:</b> Tuesday, September 14, 2021 1:08 PM<br>
<b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a> <<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> [Starlingx-discuss] Openstack image upload failed with evicted pods</font>
<div> </div>
</div>
<div>
<p></p>
<div style="border:1pt solid rgb(156,101,0);padding:2pt">
<p style="line-height:12pt;background:rgb(255,235,156)"><b><span style="font-size:10pt;color:rgb(156,101,0)"></span></b><span style="font-size:10pt;color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]</span></p>
</div>
<div>
<div dir="ltr">
<div><br>
</div>
<div>Hi,</div>
<div><br>
</div>
<div>In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. </div>
<div><br>
</div>
<div>Ceph cluster has enough space and health is ok [3]. </div>
<div><br>
</div>
<div>I can upload smaller files without any issue.</div>
<div><br>
</div>
<div>I am still trying to understand what went wrong. </div>
<div><br>
</div>
<div>[1] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$" target="_blank">
https://paste.opendev.org/show/809313/</a><br>
</div>
[2] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$" target="_blank">
https://paste.opendev.org/show/809312/</a><br>
<div>[3] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$" target="_blank">
https://paste.opendev.org/show/809314/</a><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Danishka</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" rel="noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote></div>