Hi Irina Thank you that you gave the initial test case and the error log. I think I have found the error point now. The error message is raised in _copy_image_to_volume, meanwhile the cinder backend must be rbd. Rbd driver did one more action: convert the image to a temp file, and then import the temp file to volume inside rbd. Thank you for the support! Thanks Liang From: Mihai, Irina [mailto:Irina.Mihai@windriver.com] Sent: Wednesday, November 28, 2018 10:29 AM To: Fang, Liang A <liang.a.fang@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming Hi Liang, Please find bellow the testcase initially used for this issue, together with the seen Cinder error. Irina +++++++++++++++++++++++ Testcase 1. Create an image with 8gb rootfs, using a qcow2 Centos7 image (I've used the following image: image: /home/wrsroot/CentOS-7-x86_64-GenericCloud.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 857M cluster_size: 65536 Format specific information: compat: 0.10 refcount bits: 16 ). 2. Occupy the conversions partition until you remain with 5-6GB free (I used the CentOS-7-x86_64-GenericCloud.qcow2 to fill the space) 3. Launch 4 VMs using the centos7 image and choose the "Create New Volume" option Error ERROR cinder.volume.flows.manager.create_volume [req-d3ec7b32-2599-4a14-93bc-f8051e74c52f d57645567e284958b293ffe92c391024 75602a3f79b343cb891da84d1519aaef - - -] Failed to copy image 7c662b9b-fa75-4cb1-ae8f-b38a95094fb1 to volume: 1468a764-81cb-45b2-af4a-66d006cd9401, error: qemu-img: error while writing sector 587904: No space left on device ________________________________ From: Fang, Liang A [liang.a.fang@intel.com] Sent: Tuesday, November 27, 2018 8:24 PM To: Mihai, Irina Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming Hi Irina The image I'm using is small (~20MB), but added debug code (sleep) to help simulate and analyze. Free disk space for image_conversion_dir is about 30MB. Do you remember which line of code raising the IOError exception in this case? Thanks Liang From: Mihai, Irina [mailto:Irina.Mihai@windriver.com] Sent: Wednesday, November 28, 2018 5:45 AM To: Fang, Liang A <liang.a.fang@intel.com<mailto:liang.a.fang@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming Hey Liang, When we saw the IO error, we had created 20 boot from volume instances at once with the following flavor: 8gb rootfs, 8gb ephemeral and 512 meg swap, using a qcow2 Centos7 image (quite large, not sure what you've been using in your tests). Irina ________________________________ From: Fang, Liang A [liang.a.fang@intel.com] Sent: Tuesday, November 27, 2018 5:36 AM To: Mihai, Irina Cc: 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming Hi Irina Thanks for the info. Some status: - I can reproduce the issue that solved in _convert_image(). The issue I reproduced may be not the same one as the original issue this patch to fix. Reproduced by: uploading a volume to image, but with insufficient space in image_conversion_dir. - But I still cannot reproduce the issue solved in _create_from_image_cache_or_download() even with multiple volume creating requests. Take two requests as example, the second request will be rejected by image_utils.check_available_space() at the beginning of _create_from_image_cache_or_download(). If I add debug code - sleep(60) after the line of image_utils.check_available_space, so that both two requests can pass the check, but the second request will raise exception(ImageTooBig) and be caught in the line 777, 810: "image_utils.TemporaryImages.fetch". And I take a look of other code following, don't find any chance of raising "IOError". o verify_glance_image_signature o volume_glance_metadata_bulk_create o qemu_img_info o check_virtual_size o _create_from_image_download: This will re-use image file. Even insufficient space happen, the exception be re-raised is exception.InsufficientConversionSpace, not "IOError". If you can get the original reproduce steps, that would be great! Some info related: Change-Id: Ie5eba32f92eb1740554bfc44baf612e326d6fde5, submitted in this year, with 4 commits, shows dest will be a volume. So unlikely be the same as image_conversion_dir. Thanks, Liang From: Mihai, Irina [mailto:Irina.Mihai@windriver.com] Sent: Tuesday, November 27, 2018 1:25 AM To: Fang, Liang A <liang.a.fang@intel.com<mailto:liang.a.fang@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming Hi Liang, This patch was covering the scenario when we're trying to create multiple volumes from image at the same time. In this situation, multiple downloads/convertions would be allowed to start since there would still be conversion space left, but eventually the operation would fail because of the disk being full. This is not caught by any of the "ImageTooBig" or "ImageUnacceptable" exceptions. Could you please help to address one doubt: why image is converted in the same location with image itself? This patch is to create a volume from image, right? Right, but I'm not aware of the reasoning behind this. However, if the used image is qcow2 then it needs to be converted to raw before being saved to the volume. Please let me know if I can help in any other way. Thanks, Irina ________________________________ From: Fang, Liang A [liang.a.fang@intel.com] Sent: Sunday, November 25, 2018 11:44 AM To: Mihai, Irina Cc: 'starlingx-discuss@lists.starlingx.io' Subject: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming Hi Irina Code review: https://review.openstack.org/#/c/618976/ I'm working of upstreaming this patch. Could you please help to address one doubt: why image is converted in the same location with image itself? This patch is to create a volume from image, right? I take a look of related code, the destination location would be the newly created volume and the source location is configured in cinder.conf, something like: image_conversion_dir = /mnt/d100m/image_conversion_dir I did an experiment, in the latest master branch of cinder upstream repo: - If image_conversion_dir has no space to store the downloading image, the exception ImageTooBig will be caught in the beginning of _create_from_image_cache_or_download(). - If has space to store the image, but dest volume size is smaller than image's virtual size, an exception "ImageUnacceptable" will raised by check_virtual_size. So it seems no chance to go to the situation this patch aimed to solve. Could you please help to give more background of this patch? Or if you think this patch is invalid now, please just tell me, I will mark this patch as abandoned. Thanks Liang