<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style>
<!--
@font-face
        {font-family:SimSun}
@font-face
        {font-family:"Cambria Math"}
@font-face
        {font-family:Calibri}
@font-face
        {font-family:Tahoma}
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
a:link, span.MsoHyperlink
        {color:#0563C1;
        text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
        {color:#954F72;
        text-decoration:underline}
p
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman",serif}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {margin-top:0in;
        margin-right:0in;
        margin-bottom:0in;
        margin-left:.5in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.msochpdefault, li.msochpdefault, div.msochpdefault
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Calibri",sans-serif}
span.emailstyle17
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.emailstyle21
        {font-family:"Calibri",sans-serif;
        color:#1F497D}
span.EmailStyle22
        {font-family:"Calibri",sans-serif;
        color:#1F497D}
.MsoChpDefault
        {font-size:10.0pt;
        font-family:"Calibri",sans-serif}
@page WordSection1
        {margin:1.0in 1.0in 1.0in 1.0in}
-->
</style><style type="text/css" id="owaParaStyle"></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi Liang,
<div><br>
</div>
<div>Please find bellow the testcase initially used for this issue, together with the seen Cinder error.</div>
<div><br>
</div>
<div>Irina</div>
<div><br>
</div>
<div>+++++++++++++++++++++++</div>
<div><br>
</div>
<div><b>Testcase</b></div>
<div>
<div><span style="font-size: 13.3333px;">1. Create an image with 8gb rootfs, using a qcow2 Centos7 image (I've used the following image: </span></div>
<div><span style="font-size: 13.3333px;">image: /home/wrsroot/CentOS-7-x86_64-GenericCloud.qcow2 </span></div>
<div><span style="font-size: 13.3333px;">file format: qcow2 </span></div>
<div><span style="font-size: 13.3333px;">virtual size: 8.0G (8589934592 bytes) </span></div>
<div><span style="font-size: 13.3333px;">disk size: 857M </span></div>
<div><span style="font-size: 13.3333px;">cluster_size: 65536 </span></div>
<div><span style="font-size: 13.3333px;">Format specific information: </span></div>
<div><span style="font-size: 13.3333px;">    compat: 0.10 </span></div>
<div><span style="font-size: 13.3333px;">    refcount bits: 16 </span></div>
<div><span style="font-size: 13.3333px;">). </span></div>
<div><span style="font-size: 13.3333px;">2. Occupy the conversions partition until you remain with 5-6GB free (I used the CentOS-7-x86_64-GenericCloud.qcow2 to fill the space) </span></div>
<div><span style="font-size: 13.3333px;">3. Launch 4 VMs using the centos7 image and choose the "Create New Volume" option</span></div>
</div>
<div><span style="font-size: 13.3333px;"><br>
</span></div>
<div><span style="font-size: 13.3333px;"><b>Error</b></span></div>
<div>
<div><span style="font-size: 13.3333px;">ERROR cinder.volume.flows.manager.create_volume [req-d3ec7b32-2599-4a14-93bc-f8051e74c52f d57645567e284958b293ffe92c391024 75602a3f79b343cb891da84d1519aaef - - -] Failed to copy image 7c662b9b-fa75-4cb1-ae8f-b38a95094fb1
 to volume: 1468a764-81cb-45b2-af4a-66d006cd9401, error: qemu-img: error while writing sector 587904: No space left on device </span></div>
<div><span style="font-size: 13.3333px;"><br>
</span></div>
<div><br>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF915266" style="direction: ltr;"><font face="Tahoma" size="2" color="#000000"><b>From:</b> Fang, Liang A [liang.a.fang@intel.com]<br>
<b>Sent:</b> Tuesday, November 27, 2018 8:24 PM<br>
<b>To:</b> Mihai, Irina<br>
<b>Cc:</b> 'starlingx-discuss@lists.starlingx.io'<br>
<b>Subject:</b> RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming<br>
</font><br>
</div>
<div></div>
<div>
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi Irina</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="color:#1F497D">The image I’m using is small (~20MB), but added debug code (sleep) to help simulate and analyze. Free disk space for image_conversion_dir is about 30MB.</span></p>
<p class="MsoNormal"><span style="color:#1F497D">Do you remember which line of code raising the
</span><span style="color:#1F497D">IOError exception in this case?</span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks</span><span style="color:#1F497D"></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D">Liang</span></a></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> Mihai, Irina [mailto:Irina.Mihai@windriver.com]
<br>
<b>Sent:</b> Wednesday, November 28, 2018 5:45 AM<br>
<b>To:</b> Fang, Liang A <liang.a.fang@intel.com><br>
<b>Cc:</b> 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io><br>
<b>Subject:</b> RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming</p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Hey Liang,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">When we saw the IO error, we had created 20 boot from volume instances at once with the following flavor: 8gb rootfs, 8gb ephemeral and 512 meg swap, using a qcow2
 Centos7 image (quite large, not sure what you've been using in your tests).</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Irina</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span></p>
</div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt; font-family:"Times New Roman",serif; color:black">
<hr size="2" width="100%" align="center">
</span></div>
<div id="divRpF832001">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">From:</span></b><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> Fang, Liang A [liang.a.fang@intel.com]<br>
<b>Sent:</b> Tuesday, November 27, 2018 5:36 AM<br>
<b>To:</b> Mihai, Irina<br>
<b>Cc:</b> 'starlingx-discuss@lists.starlingx.io'<br>
<b>Subject:</b> RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming</span><span style="font-size:12.0pt; font-family:"Times New Roman",serif; color:black"></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:#1F497D">Hi Irina</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks for the info. Some status:</span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">-</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">         
</span><span style="color:#1F497D">I can reproduce the issue that solved in _convert_image(). The issue I reproduced may be not the same one as the original issue this patch to fix. Reproduced by: uploading a volume to image, but with insufficient space in
 image_conversion_dir. </span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:#1F497D">-</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">         
</span><span style="color:#1F497D">But I still cannot reproduce the issue solved in _create_from_image_cache_or_download() even with multiple volume creating requests. Take two requests as example, the second request will be rejected by image_utils.check_available_space()
 at the beginning of _create_from_image_cache_or_download(). If I add debug code - sleep(60) after the line of image_utils.check_available_space, so that both two requests can pass the check, but the second request will raise exception(ImageTooBig) and be caught
 in the line 777, 810: “image_utils.TemporaryImages.fetch”. And I take a look of other code following, don’t find any chance of raising “IOError”.</span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in; text-indent:-.25in"><span style="font-family:"Courier New"; color:#1F497D">o</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">  
</span><span style="color:#1F497D">verify_glance_image_signature</span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in; text-indent:-.25in"><span style="font-family:"Courier New"; color:#1F497D">o</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">  
</span><span style="color:#1F497D">volume_glance_metadata_bulk_create</span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in; text-indent:-.25in"><span style="font-family:"Courier New"; color:#1F497D">o</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">  
</span><span style="color:#1F497D">qemu_img_info</span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in; text-indent:-.25in"><span style="font-family:"Courier New"; color:#1F497D">o</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">  
</span><span style="color:#1F497D">check_virtual_size</span><span style="color:black"></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in; text-indent:-.25in"><span style="font-family:"Courier New"; color:#1F497D">o</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:#1F497D">  
</span><span style="color:#1F497D">_create_from_image_download: This will re-use image file. Even insufficient space happen, the exception be re-raised is exception.InsufficientConversionSpace, not “IOError”.</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D">If you can get the original reproduce steps, that would be great!</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Some info related:</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Change-Id: Ie5eba32f92eb1740554bfc44baf612e326d6fde5, submitted in this year, with 4 commits, shows dest will be a volume. So unlikely be the same as image_conversion_dir.</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Liang</span><span style="color:black"></span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Mihai, Irina [<a href="mailto:Irina.Mihai@windriver.com" target="_blank" rel="noopener noreferrer">mailto:Irina.Mihai@windriver.com</a>]
<br>
<b>Sent:</b> Tuesday, November 27, 2018 1:25 AM<br>
<b>To:</b> Fang, Liang A <<a href="mailto:liang.a.fang@intel.com" target="_blank" rel="noopener noreferrer">liang.a.fang@intel.com</a>><br>
<b>Cc:</b> 'starlingx-discuss@lists.starlingx.io' <<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank" rel="noopener noreferrer">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> RE: [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming</span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> </span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Hi Liang,</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">This patch was covering the scenario when we're trying to create multiple volumes from image at the same time.</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">In this situation, multiple downloads/convertions would be allowed to start since there would still be conversion space left, but eventually the operation would
 fail because of the disk being full. This is not caught by any of the "ImageTooBig" or "ImageUnacceptable" exceptions.</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Courier New"; color:black">Could you please help to address one doubt: why image is converted in the same location with image itself? This patch is to create a volume from image, right?
</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Right, but I'm not aware of the reasoning behind this. However, if the used image is qcow2 then it needs to be converted to raw before being saved to the volume.</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Please let me know if I can help in any other way.</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Thanks,</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">Irina</span><span style="color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> </span><span style="color:black"></span></p>
</div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt; font-family:"Times New Roman",serif; color:black">
<hr size="2" width="100%" align="center">
</span></div>
<div id="divRpF77892">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black">From:</span></b><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"> Fang, Liang A [liang.a.fang@intel.com]<br>
<b>Sent:</b> Sunday, November 25, 2018 11:44 AM<br>
<b>To:</b> Mihai, Irina<br>
<b>Cc:</b> 'starlingx-discuss@lists.starlingx.io'<br>
<b>Subject:</b> [Starlingx-discuss] [cinder] Analysis of patch c45bb0e for StartlingX upstreaming</span><span style="color:black"></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Hi Irina</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">Code review: <a href="https://review.openstack.org/#/c/618976/" target="_blank" rel="noopener noreferrer">
https://review.openstack.org/#/c/618976/</a></span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">I’m working of upstreaming this patch. Could you please help to address one doubt: why image is converted in the same location with image itself? This patch is to create a volume from image, right?
</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">I take a look of related code, the destination location would be the newly created volume and the source location is configured in cinder.conf, something like:</span></p>
<p class="MsoNormal"><span style="color:black">image_conversion_dir = /mnt/d100m/image_conversion_dir</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">I did an experiment, in the latest master branch of cinder upstream repo:</span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:black">-</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:black">         
</span><span style="color:black">If image_conversion_dir has no space to store the downloading image, the exception ImageTooBig will be caught in the beginning of _create_from_image_cache_or_download().
</span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="color:black">-</span><span style="font-size:7.0pt; font-family:"Times New Roman",serif; color:black">         
</span><span style="color:black">If has space to store the image, but dest volume size is smaller than image’s virtual size, an exception “ImageUnacceptable” will raised by check_virtual_size.
</span></p>
<p class="MsoNormal"><span style="color:black">So it seems no chance to go to the situation this patch aimed to solve.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">Could you please help to give more background of this patch? Or if you think this patch is invalid now, please just tell me, I will mark this patch as abandoned.</span></p>
<p class="MsoNormal"><span style="color:black"> </span></p>
<p class="MsoNormal"><span style="color:black">Thanks</span></p>
<p class="MsoNormal"><span style="color:black">Liang</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>