[Starlingx-discuss] Openstack image upload failed with evicted pods

[Brito, Thiago]

Hi Danishka,

We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please:
1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes)
2) Check the eviction reason by looking at kubectl describe on the evicted pods
3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?

Let me know.

Thiago
________________________________
From: open infra <openinfradn at gmail.com>
Sent: Tuesday, September 14, 2021 1:08 PM
To: starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Openstack image upload failed with evicted pods


[Please note: This e-mail is from an EXTERNAL e-mail address]

Hi,

In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2].

Ceph cluster has enough space and health is ok [3].

I can upload smaller files without any issue.

I am still trying to understand what went wrong.

[1] https://paste.opendev.org/show/809313/<https://urldefense.com/v3/__https://paste.opendev.org/show/809313/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHEYx6De1$>
[2] https://paste.opendev.org/show/809312/<https://urldefense.com/v3/__https://paste.opendev.org/show/809312/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHN8J2O6S$>
[3] https://paste.opendev.org/show/809314/<https://urldefense.com/v3/__https://paste.opendev.org/show/809314/__;!!AjveYdw8EvQ!LqJ04-tWXdE2kt6JfD5R7VU_pZsgv8NZTzZi1tEhg0av3cD_OcPfRvqIn8aWHMcF5oVF$>

Regards,
Danishka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210914/ab3c964f/attachment.htm>