[Starlingx-discuss] Openstack image upload failed with evicted pods
Hi, In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. Ceph cluster has enough space and health is ok [3]. I can upload smaller files without any issue. I am still trying to understand what went wrong. [1] https://paste.opendev.org/show/809313/ [2] https://paste.opendev.org/show/809312/ [3] https://paste.opendev.org/show/809314/ Regards, Danishka
Hi Danishka, We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please: 1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes) 2) Check the eviction reason by looking at kubectl describe on the evicted pods 3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again? Let me know. Thiago ________________________________ From: open infra <openinfradn@gmail.com> Sent: Tuesday, September 14, 2021 1:08 PM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Openstack image upload failed with evicted pods [Please note: This e-mail is from an EXTERNAL e-mail address] Hi, In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. Ceph cluster has enough space and health is ok [3]. I can upload smaller files without any issue. I am still trying to understand what went wrong. [1] https://paste.opendev.org/show/809313/<https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$> [2] https://paste.opendev.org/show/809312/<https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$> [3] https://paste.opendev.org/show/809314/<https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$> Regards, Danishka
Hi Thiago, I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0]. https://paste.opendev.org/show/809328/ Disk usage of kubelet controller-0: [1] and controller-1: [2] Disk usage of controller-0 [3] and controller-1 [4]. There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9]. The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6] I was able to upload the image after increasing kubelet size to 50GB. [0] https://paste.opendev.org/show/809328/ [1] https://paste.opendev.org/show/809325/ [2] https://paste.opendev.org/show/809324/ [3] https://paste.opendev.org/show/809326/ [4] https://paste.opendev.org/show/809323/ [5] https://paste.opendev.org/show/809320/ [6] https://paste.opendev.org/show/809319/ [7] https://paste.opendev.org/show/809318/ [8] https://paste.opendev.org/show/809321/ [9] https://paste.opendev.org/show/809322/ I have created a LP as at least the stx documentation should advice to increase the kubelet size. The documentation suggest to increase docker size set to 60GB instead of default 30GB. https://bugs.launchpad.net/starlingx/+bug/1943674 Regards, Danishka On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <Thiago.Brito@windriver.com> wrote:
Hi Danishka,
We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please: 1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes) 2) Check the eviction reason by looking at kubectl describe on the evicted pods 3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?
Let me know.
Thiago ------------------------------ *From:* open infra <openinfradn@gmail.com> *Sent:* Tuesday, September 14, 2021 1:08 PM *To:* starlingx-discuss@lists.starlingx.io < starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Openstack image upload failed with evicted pods
[Please note: This e-mail is from an EXTERNAL e-mail address]
Hi,
In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2].
Ceph cluster has enough space and health is ok [3].
I can upload smaller files without any issue.
I am still trying to understand what went wrong.
[1] https://paste.opendev.org/show/809313/ <https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$> [2] https://paste.opendev.org/show/809312/ <https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$> [3] https://paste.opendev.org/show/809314/ <https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$>
Regards, Danishka _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
I *think* that documentation is stale since we are now using containerd instead of docker on StarlingX, but I might be wrong. Anybody have some thoughts about that? Thanks for opening that bug. Thiago ________________________________ From: open infra <openinfradn@gmail.com> Sent: Wednesday, September 15, 2021 2:33 AM To: Brito, Thiago <Thiago.Brito@windriver.com> Cc: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Openstack image upload failed with evicted pods [Please note: This e-mail is from an EXTERNAL e-mail address] Hi Thiago, I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0]. https://paste.opendev.org/show/809328/<https://urldefense.com/v3/__https://paste.opendev.org/show/809328/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmwKyMsji$> Disk usage of kubelet controller-0: [1] and controller-1: [2] Disk usage of controller-0 [3] and controller-1 [4]. There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9]. The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6] I was able to upload the image after increasing kubelet size to 50GB. [0] https://paste.opendev.org/show/809328/<https://urldefense.com/v3/__https://paste.opendev.org/show/809328/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmwKyMsji$> [1] https://paste.opendev.org/show/809325/<https://urldefense.com/v3/__https://paste.opendev.org/show/809325/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nmxg0Jhmj$> [2] https://paste.opendev.org/show/809324/<https://urldefense.com/v3/__https://paste.opendev.org/show/809324/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm41DIss_$> [3] https://paste.opendev.org/show/809326/<https://urldefense.com/v3/__https://paste.opendev.org/show/809326/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_HpCpom$> [4] https://paste.opendev.org/show/809323/<https://urldefense.com/v3/__https://paste.opendev.org/show/809323/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_SwV79i$> [5] https://paste.opendev.org/show/809320/<https://urldefense.com/v3/__https://paste.opendev.org/show/809320/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm8xkqOPE$> [6] https://paste.opendev.org/show/809319/<https://urldefense.com/v3/__https://paste.opendev.org/show/809319/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_cngqi-$> [7] https://paste.opendev.org/show/809318/<https://urldefense.com/v3/__https://paste.opendev.org/show/809318/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm8JC1vgQ$> [8] https://paste.opendev.org/show/809321/<https://urldefense.com/v3/__https://paste.opendev.org/show/809321/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm5PvdFrb$> [9] https://paste.opendev.org/show/809322/<https://urldefense.com/v3/__https://paste.opendev.org/show/809322/__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm_tcuKJR$> I have created a LP as at least the stx documentation should advice to increase the kubelet size. The documentation suggest to increase docker size set to 60GB instead of default 30GB. https://bugs.launchpad.net/starlingx/+bug/1943674<https://urldefense.com/v3/__https://bugs.launchpad.net/starlingx/*bug/1943674__;Kw!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm50bMs3j$> Regards, Danishka On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <Thiago.Brito@windriver.com<mailto:Thiago.Brito@windriver.com>> wrote: Hi Danishka, We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please: 1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes) 2) Check the eviction reason by looking at kubectl describe on the evicted pods 3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again? Let me know. Thiago ________________________________ From: open infra <openinfradn@gmail.com<mailto:openinfradn@gmail.com>> Sent: Tuesday, September 14, 2021 1:08 PM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] Openstack image upload failed with evicted pods [Please note: This e-mail is from an EXTERNAL e-mail address] Hi, In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. Ceph cluster has enough space and health is ok [3]. I can upload smaller files without any issue. I am still trying to understand what went wrong. [1] https://paste.opendev.org/show/809313/<https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$> [2] https://paste.opendev.org/show/809312/<https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$> [3] https://paste.opendev.org/show/809314/<https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$> Regards, Danishka _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss<https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!LZVYa0P6xkz5oFXyZpkcDyf1Wn8VMY5xuWVn2YQ4uuC1rG-pUGaX5Rc0SW5nm9mPx9SB$>
participants (2)
-
Brito, Thiago
-
open infra