[Starlingx-discuss] [glance] Analysis of patch c73ab48 (Enhanced-error-messages) for StartlingX upstreaming

Dinescu, Stefan Stefan.Dinescu at windriver.com
Tue Nov 13 15:41:02 UTC 2018


I think you are missing a glance_store patch for starlingx: https://github.com/openstack/stx-upstream/blob/master/openstack/python-glance-store/centos/patches/0001-Check-ceph-cluster-free-space-before-creating-image.patch

This patch added aditional checks for ceph space requirments.

I am saying this due to how you described the "self._create_image function" where it is missing a parameter added by the above patch.

In our use-case, the image-size was always passed to the rbd driver code, so the resize function (which is only called when the size is zero) was never called.

How the algorithm should work:
- grab the size of the image that is about to be uploaded (we tested this through cli and horizon)
- before uploading any data, check that ceph actually has space for that data
- only after the check passes, we actually begin writing data to ceph

We did this, because ceph actually blocked almost completely when a pool was full, requiring manual intervention to recover.

The LVM backend behaved properly in such cases, so no fix was written for that.

________________________________
From: Fang, Liang A [liang.a.fang at intel.com]
Sent: Tuesday, November 13, 2018 5:23 PM
To: Dinescu, Stefan; Badea, Daniel
Cc: starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] [glance] Analysis of patch c73ab48 (Enhanced-error-messages) for StartlingX upstreaming

Hi Stefan

I take a look at this issue today, the hang happened inside ceph. So I modified ceph code (function resize) to just raise exception rbd.NoSpace at the entry of the function. This issue can be simulated now.

Regarding this patch, as I said previously, rbd (ceph) is just one type of the multiple types of backend, so I prefer to catch the rbd.NoSpace in glance_store (rbd.py), handle the exception there, and then re-raise a more common exception (StorageFull) to glance. Do you think this makes sense? If so, I will close the review in glance, and create a new review in glance_store.

Regards
Liang
From: Dinescu, Stefan [mailto:Stefan.Dinescu at windriver.com]
Sent: Monday, November 12, 2018 10:13 PM
To: Fang, Liang A <liang.a.fang at intel.com>; Badea, Daniel <Daniel.Badea at windriver.com>
Cc: starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] [glance] Analysis of patch c73ab48 (Enhanced-error-messages) for StartlingX upstreaming

Hello,

I havent' tested that part of the code in some time now, but from what I remember it was part of a larger set of changes to prevent ceph from hanging if not enough space was available.

How it should work (but as I said, I haven't tested this in some time):
-  decrease the glance image pool quota (this is just so it's easier to test)
-  upload images to glance until space if almost full
- the image that would have filled the pool should fail to upload entirely.

It seems you are doing the correct procedure from what you described, so this probably requires a more in-depth investigation to see why it fails.
________________________________
From: Fang, Liang A [liang.a.fang at intel.com]
Sent: Saturday, November 10, 2018 11:23 AM
To: Badea, Daniel; Dinescu, Stefan
Cc: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [glance] Analysis of patch c73ab48 (Enhanced-error-messages) for StartlingX upstreaming
Hi Daniel and Stefan

Please refer to https://review.openstack.org/#/c/616880/
This patch is to provide detailed exception info when no space in rbd. But in my environment, if rbd has no space, the function call to rbd will not return, there’s no exception raised.

Details:
In /usr/local/lib/python2.7/dist-packages/glance_store/_drivers/rbd.py

-          Function call will hang at line 484 (_create_image) if no space to create image at the beginning
[cid:image001.png at 01D47B9B.E0DB9020]

-          Function call will hang at line 506 (image.resize) if no space at the middle of uploading.

[cid:image002.png at 01D47B9B.E0DB9020]

Could you please give some clue to reproduce this patch (raise rbd.NoSpace)?
Couple of other thought:

1.       This patch is to catch exception at v1 api, but v1 is about to be removed by community.

2.       rbd is just one type of the multiple types of backend, so I think catch specific backend’s exception at a common code is not so good. glance_store/rbd.py may be the better place to catch this exception. In fact rbd.py already did similar thing, see:

[cid:image003.png at 01D47B9B.E0DB9020]

Thanks
Liang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181113/d0e3c9db/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 2947 bytes
Desc: image001.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181113/d0e3c9db/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1524 bytes
Desc: image002.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181113/d0e3c9db/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 8559 bytes
Desc: image003.png
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181113/d0e3c9db/attachment-0005.png>


More information about the Starlingx-discuss mailing list