Hi Thiago,

I have increased both docker (150GB) and kublet (25GB) size in both controllers while working on this issue. And also increase the ceph mon size. [0].
https://paste.opendev.org/show/809328/
Disk usage of kubelet controller-0: [1] and controller-1: [2]
Disk usage of controller-0 [3] and controller-1 [4].

There were couple of pods failed with the following error and few failed with DiskPressure [7][8][9].
The node was low on resource: ephemeral-storage. Container horizon was using 19361853, which exceeds its request of 0. [5][6]

I was able to upload the image after increasing kubelet size to 50GB.

[0] https://paste.opendev.org/show/809328/
[1] https://paste.opendev.org/show/809325/
[2] https://paste.opendev.org/show/809324/
[3] https://paste.opendev.org/show/809326/
[4] https://paste.opendev.org/show/809323/

[5] https://paste.opendev.org/show/809320/
[6] https://paste.opendev.org/show/809319/
[7] https://paste.opendev.org/show/809318/
[8] https://paste.opendev.org/show/809321/
[9] https://paste.opendev.org/show/809322/

I have created a LP as at least the stx documentation should advice to increase the kubelet size.
The documentation suggest to increase docker size set to 60GB instead of default 30GB.
https://bugs.launchpad.net/starlingx/+bug/1943674

Regards,
Danishka

On Tue, Sep 14, 2021 at 10:01 PM Brito, Thiago <Thiago.Brito@windriver.com> wrote:
Hi Danishka,

We noticed rather recently that in order to update larger images, we have to increase the kubelet-lv size so the system doesn't get starved on disk during image conversion/upload to Glance due to the high usage of "emptyDir" volumes. Could you please:
1) Post the size and %Used of your kubelet-lv (df -h on the controller nodes)
2) Check the eviction reason by looking at kubectl describe on the evicted pods
3) maybe try to extend kubelet-lv a couple of dozen of GBs and try again?

Let me know.

Thiago
From: open infra <openinfradn@gmail.com>
Sent: Tuesday, September 14, 2021 1:08 PM
To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: [Starlingx-discuss] Openstack image upload failed with evicted pods
 

[Please note: This e-mail is from an EXTERNAL e-mail address]


Hi,

In a freshly installed stx release 5 (standard dedicated storage) system, I tried to upload 41GB of qcow2 image but failed [1]. Noticed that some of the pods are evicted [2]. 

Ceph cluster has enough space and health is ok [3]. 

I can upload smaller files without any issue.

I am still trying to understand what went wrong. 

[2] https://paste.opendev.org/show/809312/

Regards,
Danishka
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss