[Starlingx-discuss] Openstack image upload failed with evicted pods

[Brito, Thiago]

I *think* that documentation is stale since we are now using containerd instead of docker on StarlingX, but I might be wrong. Anybody have some thoughts about that?

Thanks for opening that bug.

Thiago
________________________________
From: open infra <openinfradn at gmail.com>
Sent: Wednesday, September 15, 2021 2:33 AM
To: Brito, Thiago <Thiago.Brito at windriver.com>
Cc: starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Openstack image upload failed with evicted pods


[Please note: This e-mail is from an EXTERNAL e-mail address]

Hi Thiago,

I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0].
https://paste.opendev.org/show/809328/<https://urldefense.com/v3/__https://paste.opendev.org/show/809328/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmwKyMsji$>
Disk usage of kubelet controller-0: [1] and controller-1: [2]
Disk usage of controller-0 [3] and controller-1 [4].

There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9].

The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6]


I was able to upload the image after increasing kubelet size to 50GB.

[0] https://paste.opendev.org/show/809328/<https://urldefense.com/v3/__https://paste.opendev.org/show/809328/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmwKyMsji$>
[1] https://paste.opendev.org/show/809325/<https://urldefense.com/v3/__https://paste.opendev.org/show/809325/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmxg0Jhmj$>
[2] https://paste.opendev.org/show/809324/<https://urldefense.com/v3/__https://paste.opendev.org/show/809324/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm41DIss_$>
[3] https://paste.opendev.org/show/809326/<https://urldefense.com/v3/__https://paste.opendev.org/show/809326/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_HpCpom$>
[4] https://paste.opendev.org/show/809323/<https://urldefense.com/v3/__https://paste.opendev.org/show/809323/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_SwV79i$>

[5] https://paste.opendev.org/show/809320/<https://urldefense.com/v3/__https://paste.opendev.org/show/809320/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm8xkqOPE$>
[6] https://paste.opendev.org/show/809319/<https://urldefense.com/v3/__https://paste.opendev.org/show/809319/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_cngqi-$>
[7] https://paste.opendev.org/show/809318/<https://urldefense.com/v3/__https://paste.opendev.org/show/809318/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm8JC1vgQ$>
[8] https://paste.opendev.org/show/809321/<https://urldefense.com/v3/__https://paste.opendev.org/show/809321/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm5PvdFrb$>
[9] https://paste.opendev.org/show/809322/<https://urldefense.com/v3/__https://paste.opendev.org/show/809322/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_tcuKJR$>

I have created a LP as at least the stx documentation should advice to increase the kubelet size.
The documentation suggest to increase docker size set to 60GB instead of default 30GB.
https://bugs.launchpad.net/starlingx/+bug/1943674<https://urldefense.com/v3/__https://bugs.launchpad.net/starlingx/*bug/1943674__;Kw!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm50bMs3j$>

Regards,
Danishka

On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <Thiago.Brito at windriver.com<mailto:Thiago.Brito at windriver.com>> wrote:
Hi Danishka,

We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please:
1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes)
2) Check the eviction reason by looking at kubectl describe on the evicted pods
3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?

Let me know.

Thiago
________________________________
From: open infra <openinfradn at gmail.com<mailto:openinfradn at gmail.com>>
Sent: Tuesday, September 14, 2021 1:08 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Subject: [Starlingx-discuss] Openstack image upload failed with evicted pods


[Please note: This e-mail is from an EXTERNAL e-mail address]

Hi,

In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2].

Ceph cluster has enough space and health is ok [3].

I can upload smaller files without any issue.

I am still trying to understand what went wrong.

[1] https://paste.opendev.org/show/809313/<https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$>
[2] https://paste.opendev.org/show/809312/<https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$>
[3] https://paste.opendev.org/show/809314/<https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$>

Regards,
Danishka
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss<https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm9mPx9SB$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210915/0d7175cf/attachment.htm>