Hi Danishka,

We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please:
1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes)
2) Check the eviction reason by looking at kubectl describe on the evicted pods
3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?

Let me know.

Thiago
From: open infra <openinfradn@gmail.com>
Sent: Tuesday, September 14, 2021 1:08 PM
To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: [Starlingx-discuss] Openstack image upload failed with evicted pods
 

[Please note: This e-mail is from an EXTERNAL e-mail address]


Hi,

In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. 

Ceph cluster has enough space and health is ok [3]. 

I can upload smaller files without any issue.

I am still trying to understand what went wrong. 

[1] https://paste.opendev.org/show/809313/
[2] https://paste.opendev.org/show/809312/
[3] https://paste.opendev.org/show/809314/

Regards,
Danishka