[Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming

Mihai, Irina Irina.Mihai at windriver.com
Wed Nov 28 02:29:21 UTC 2018


Hi Liang,

Please find bellow the testcase initially used for this issue, together with the seen Cinder error.

Irina

+++++++++++++++++++++++

Testcase
1. Create an image with 8gb rootfs, using a qcow2 Centos7 image (I've used the following image:
image: /home/wrsroot/CentOS-7-x86_64-GenericCloud.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 857M
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16
).
2. Occupy the conversions partition until you remain with 5-6GB free (I used the CentOS-7-x86_64-GenericCloud.qcow2 to fill the space)
3. Launch 4 VMs using the centos7 image and choose the "Create New Volume" option

Error
ERROR cinder.volume.flows.manager.create_volume [req-d3ec7b32-2599-4a14-93bc-f8051e74c52f d57645567e284958b293ffe92c391024 75602a3f79b343cb891da84d1519aaef - - -] Failed to copy image 7c662b9b-fa75-4cb1-ae8f-b38a95094fb1 to volume: 1468a764-81cb-45b2-af4a-66d006cd9401, error: qemu-img: error while writing sector 587904: No space left on device


________________________________
From: Fang, Liang A [liang.a.fang at intel.com]
Sent: Tuesday, November 27, 2018 8:24 PM
To: Mihai, Irina
Cc: 'starlingx-discuss at lists.starlingx.io'
Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming

Hi Irina

The image I’m using is small (~20MB), but added debug code (sleep) to help simulate and analyze. Free disk space for image_conversion_dir is about 30MB.
Do you remember which line of code raising the IOError exception in this case?

Thanks
Liang
From: Mihai, Irina [mailto:Irina.Mihai at windriver.com]
Sent: Wednesday, November 28, 2018 5:45 AM
To: Fang, Liang A <liang.a.fang at intel.com>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming

Hey Liang,

When we saw the IO error, we had created 20 boot from volume instances at once with the following flavor: 8gb rootfs, 8gb ephemeral and 512 meg swap, using a qcow2 Centos7 image (quite large, not sure what you've been using in your tests).

Irina

________________________________
From: Fang, Liang A [liang.a.fang at intel.com]
Sent: Tuesday, November 27, 2018 5:36 AM
To: Mihai, Irina
Cc: 'starlingx-discuss at lists.starlingx.io'
Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming
Hi Irina

Thanks for the info. Some status:

-          I can reproduce the issue that solved in _convert_image(). The issue I reproduced may be not the same one as the original issue this patch to fix. Reproduced by: uploading a volume to image, but with insufficient space in image_conversion_dir.

-          But I still cannot reproduce the issue solved in _create_from_image_cache_or_download() even with multiple volume creating requests. Take two requests as example, the second request will be rejected by image_utils.check_available_space() at the beginning of _create_from_image_cache_or_download(). If I add debug code - sleep(60) after the line of image_utils.check_available_space, so that both two requests can pass the check, but the second request will raise exception(ImageTooBig) and be caught in the line 777, 810: “image_utils.TemporaryImages.fetch”. And I take a look of other code following, don’t find any chance of raising “IOError”.

o   verify_glance_image_signature

o   volume_glance_metadata_bulk_create

o   qemu_img_info

o   check_virtual_size

o   _create_from_image_download: This will re-use image file. Even insufficient space happen, the exception be re-raised is exception.InsufficientConversionSpace, not “IOError”.

If you can get the original reproduce steps, that would be great!

Some info related:
Change-Id: Ie5eba32f92eb1740554bfc44baf612e326d6fde5, submitted in this year, with 4 commits, shows dest will be a volume. So unlikely be the same as image_conversion_dir.

Thanks,
Liang
From: Mihai, Irina [mailto:Irina.Mihai at windriver.com]
Sent: Tuesday, November 27, 2018 1:25 AM
To: Fang, Liang A <liang.a.fang at intel.com<mailto:liang.a.fang at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming

Hi Liang,

This patch was covering the scenario when we're trying to create multiple volumes from image at the same time.
In this situation, multiple downloads/convertions would be allowed to start since there would still be conversion space left, but eventually the operation would fail because of the disk being full. This is not caught by any of the "ImageTooBig" or "ImageUnacceptable" exceptions.

Could you please help to address one doubt: why image is converted in the same location with image itself? This patch is to create a volume from image, right?
Right, but I'm not aware of the reasoning behind this. However, if the used image is qcow2 then it needs to be converted to raw before being saved to the volume.

Please let me know if I can help in any other way.

Thanks,
Irina

________________________________
From: Fang, Liang A [liang.a.fang at intel.com]
Sent: Sunday, November 25, 2018 11:44 AM
To: Mihai, Irina
Cc: 'starlingx-discuss at lists.starlingx.io'
Subject: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming
Hi Irina

Code review: https://review.openstack.org/#/c/618976/

I’m working of upstreaming this patch. Could you please help to address one doubt: why image is converted in the same location with image itself? This patch is to create a volume from image, right?

I take a look of related code, the destination location would be the newly created volume and the source location is configured in cinder.conf, something like:
image_conversion_dir = /mnt/d100m/image_conversion_dir

I did an experiment, in the latest master branch of cinder upstream repo:

-          If image_conversion_dir has no space to store the downloading image, the exception ImageTooBig will be caught in the beginning of _create_from_image_cache_or_download().

-          If has space to store the image, but dest volume size is smaller than image’s virtual size, an exception “ImageUnacceptable” will raised by check_virtual_size.
So it seems no chance to go to the situation this patch aimed to solve.

Could you please help to give more background of this patch? Or if you think this patch is invalid now, please just tell me, I will mark this patch as abandoned.

Thanks
Liang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20181128/37cf860d/attachment-0001.html>


More information about the Starlingx-discuss mailing list