[Starlingx-discuss] Openstack image upload failed with evicted pods

[open infra]

Hi Thiago,

I have increased both docker (150GB) and kublet (25GB) size in both
controllers while working on this issue. And also increase the ceph mon
size. [0].
https://paste.opendev.org/show/809328/
Disk usage of kubelet controller-0: [1] and controller-1: [2]
Disk usage of controller-0 [3] and controller-1 [4].

There were couple of pods failed with the following error and few failed
with DiskPressure [7][8][9].

The node was low on resource: ephemeral-storage. Container horizon was
using 19361853, which exceeds its request of 0. [5][6]


I was able to upload the image after increasing kubelet size to 50GB.

[0] https://paste.opendev.org/show/809328/
[1] https://paste.opendev.org/show/809325/
[2] https://paste.opendev.org/show/809324/
[3] https://paste.opendev.org/show/809326/
[4] https://paste.opendev.org/show/809323/

[5] https://paste.opendev.org/show/809320/
[6] https://paste.opendev.org/show/809319/
[7] https://paste.opendev.org/show/809318/
[8] https://paste.opendev.org/show/809321/
[9] https://paste.opendev.org/show/809322/

I have created a LP as at least the stx documentation should advice to
increase the kubelet size.
The documentation suggest to increase docker size set to 60GB instead of
default 30GB.
https://bugs.launchpad.net/starlingx/+bug/1943674

Regards,
Danishka

On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <Thiago.Brito at windriver.com>
wrote:

> Hi Danishka,
>
> We noticed rather recently that in order to update larger images, we have
> to increase the kubelet-lv size so the system doesn't get starved on disk
> during image conversion/upload to Glance due to the high usage of
> "emptyDir" volumes. Could you please:
> 1) Post the size and %Used of your kubelet-lv (df -h on the controller
> nodes)
> 2) Check the eviction reason by looking at kubectl describe on the evicted
> pods
> 3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?
>
> Let me know.
>
> Thiago
> ------------------------------
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Tuesday, September 14, 2021 1:08 PM
> *To:* starlingx-discuss at lists.starlingx.io <
> starlingx-discuss at lists.starlingx.io>
> *Subject:* [Starlingx-discuss] Openstack image upload failed with evicted
> pods
>
>
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> Hi,
>
> In a freshly installed stx release 5 (standard dedicated storage) system,
> I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of
> the pods are evicted [2].
>
> Ceph cluster has enough space and health is ok [3].
>
> I can upload smaller files without any issue.
>
> I am still trying to understand what went wrong.
>
> [1] https://paste.opendev.org/show/809313/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$>
> [2] https://paste.opendev.org/show/809312/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$>
> [3] https://paste.opendev.org/show/809314/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$>
>
> Regards,
> Danishka
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210915/6f6f0870/attachment-0001.htm>