Hi Thiago, I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0]. https://paste.opendev.org/show/809328/ Disk usage of kubelet controller-0: [1] and controller-1: [2] Disk usage of controller-0 [3] and controller-1 [4]. There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9]. The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6] I was able to upload the image after increasing kubelet size to 50GB. [0] https://paste.opendev.org/show/809328/ [1] https://paste.opendev.org/show/809325/ [2] https://paste.opendev.org/show/809324/ [3] https://paste.opendev.org/show/809326/ [4] https://paste.opendev.org/show/809323/ [5] https://paste.opendev.org/show/809320/ [6] https://paste.opendev.org/show/809319/ [7] https://paste.opendev.org/show/809318/ [8] https://paste.opendev.org/show/809321/ [9] https://paste.opendev.org/show/809322/ I have created a LP as at least the stx documentation should advice to increase the kubelet size. The documentation suggest to increase docker size set to 60GB instead of default 30GB. https://bugs.launchpad.net/starlingx/+bug/1943674 Regards, Danishka On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <Thiago.Brito@windriver.com> wrote:
Hi Danishka,
We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please: 1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes) 2) Check the eviction reason by looking at kubectl describe on the evicted pods 3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?
Let me know.
Thiago ------------------------------ *From:* open infra <openinfradn@gmail.com> *Sent:* Tuesday, September 14, 2021 1:08 PM *To:* starlingx-discuss@lists.starlingx.io < starlingx-discuss@lists.starlingx.io> *Subject:* [Starlingx-discuss] Openstack image upload failed with evicted pods
[Please note: This e-mail is from an EXTERNAL e-mail address]
Hi,
In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2].
Ceph cluster has enough space and health is ok [3].
I can upload smaller files without any issue.
I am still trying to understand what went wrong.
[1] https://paste.opendev.org/show/809313/ <https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$> [2] https://paste.opendev.org/show/809312/ <https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$> [3] https://paste.opendev.org/show/809314/ <https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$>
Regards, Danishka _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss