[Starlingx-discuss] About EB verification for openstack train upgrade

Martinez Monroy, Elio elio.martinez.monroy at intel.com
Mon Nov 4 17:30:28 UTC 2019 

Christopher is on vacations today. I Will provide the image ASAP.

-----Original Message-----
From: Hu, Yong <yong.hu at intel.com> 
Sent: Monday, November 4, 2019 7:56 AM
To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; Liu, ZhipengS <zhipengs.liu at intel.com>; Xie, Cindy <cindy.xie at intel.com>; '朱博祥' <zhu.boxiang at 99cloud.net>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Christopher,
A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX?

In addition, @Zhipeng, we might get this image to dig out the root-cause further.

Regards,
Yong


On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
> Hello,
> 
> Using the EB that Zhipeng provided, and replacing 
> “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, 
> Sanity status is Green.
> 
> *AIO – Simplex*
> 
> **
> 
> Setup             04 TCs [PASS]
> 
> Provisioning      01 TCs [PASS]
> 
> Sanity OpenStack  49 TCs [PASS]
> 
> Sanity Platform   07 TCs [PASS]
> 
> TOTAL: [ 61 TCs PASS ]
> 
> *AIO – Duplex*
> 
> **
> 
> Setup             04 TCs [PASS]
> 
> Provisioning      01 TCs [PASS]
> 
> Sanity OpenStack  52 TCs [PASS]
> 
> Sanity Platform   07 TCs [PASS]
> 
> TOTAL: [ 64 TCs PASS ]
> 
> *Standard - Local Storage (2+2)*
> 
> Setup             04 TCs [PASS]
> 
> Provisioning      01 TCs [PASS]
> 
> Sanity OpenStack  52 TCs [PASS]
> 
> Sanity Platform   08 TCs [PASS]
> 
> TOTAL: [ 65 TCs PASS ]
> 
> *Standard - Dedicated Storage (2+2+2)*
> 
> Setup             04 TCs [PASS]
> 
> Provisioning      01 TCs [PASS]
> 
> Sanity OpenStack  52 TCs [PASS]
> 
> Sanity Platform   09 TCs [PASS]
> 
> TOTAL: [ 66 TCs PASS ]
> 
> ===========================================
> 
> Just the cirros image was replaced. Centos file is still a .qcow2.
> 
> Please, let me know if you need further details.
> 
> Thanks & Regards,
> 
> Cristopher Lemus
> 
> *From: *"Liu, ZhipengS" <zhipengs.liu at intel.com>
> *Date: *Friday, November 1, 2019 at 12:02 AM
> *To: *"Xie, Cindy" <cindy.xie at intel.com>, "Lemus Contreras, Cristopher 
> J" <cristopher.j.lemus.contreras at intel.com>, '朱博祥'
> <zhu.boxiang at 99cloud.net>, "'starlingx-discuss at lists.starlingx.io'" 
> <starlingx-discuss at lists.starlingx.io>
> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Hu, Yong" 
> <yong.hu at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>, "Jones, 
> Bruce E" <bruce.e.jones at intel.com>, "Martinez Monroy, Elio"
> <elio.martinez.monroy at intel.com>
> *Subject: *RE: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> Hi all,
> 
> After using new image file (cirros-0.4.0-x86_64-disk.img), we can 
> launch instance on standard external Storage (2+2+2) now.
> 
> It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by 
> Cristopher Lemus causes qemu-img detecting wrong format after format 
> convert from qcow2to raw in Nova code.  We are not sure how/where we 
> got this image file.
> 
> Thanks!
> 
> Zhipeng
> 
> *From:* Liu, ZhipengS
> *Sent:* 2019年10月31日 17:31
> *To:* Xie, Cindy <cindy.xie at intel.com>; Lemus Contreras, Cristopher J 
> <cristopher.j.lemus.contreras at intel.com>; '朱博祥'
> <zhu.boxiang at 99cloud.net>; 'starlingx-discuss at lists.starlingx.io' 
> <starlingx-discuss at lists.starlingx.io>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong 
> <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, 
> Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio 
> <elio.martinez.monroy at intel.com>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> Hi Boxiang and all,
> 
> For remote storage configuration,  we have overridden below 
> configurations in helm/nova.py.
> 
>       if remote_storage:
> 
> default_config.update({'force_raw_images': True})
> 
>              libvirt_config.update({'images_type': 'rbd',
> 
>                                     'images_rbd_pool': rbd_pool,
> 
>                                     'images_rbd_ceph_conf': 
> rbd_ceph_conf})
> 
>          else:
> 
>              libvirt_config.update({'images_type': 'default'})
> 
> Next step:
> 
> The only issue we need to figure out is why image format is still 
> qcow2 after image format convert by qemu-img in nova code.
> 
> I tried to reproduce it in my virtualized multi-node environment, but 
> no issue found, VM created normally.
> 
> (I copied yellow code block to else branch)
> 
> In /var/lib/nova/instances/_base/ folder,
> 
> I can see one qcow2 image and one raw image which should be converted 
> by nova.
> 
> [root at compute-1 _base]# ls
> 
> 3cb926303b665709318d2c099722b914025023b4
> 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
> 
> [root at compute-1 _base]# qemu-img info
> 3cb926303b665709318d2c099722b914025023b4
> 
> image: 3cb926303b665709318d2c099722b914025023b4
> 
> file format: qcow2
> 
> virtual size: 44M (46137344 bytes)
> 
> disk size: 12M
> 
> cluster_size: 65536
> 
> Format specific information:
> 
>      compat: 1.1
> 
>      lazy refcounts: false
> 
>      refcount bits: 16
> 
>      corrupt: false
> 
> [root at compute-1 _base]# qemu-img info 
> 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
> 
> image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
> 
> file format: raw
> 
> virtual size: 44M (46137344 bytes)
> 
> disk size: 13M
> 
> Any comment?
> 
> Thanks!
> 
> zhipeng
> 
> *From:* Liu, ZhipengS <zhipengs.liu at intel.com 
> <mailto:zhipengs.liu at intel.com>>
> *Sent:* 2019年10月31日 10:15
> *To:* Liu, ZhipengS <zhipengs.liu at intel.com 
> <mailto:zhipengs.liu at intel.com>>; Xie, Cindy <cindy.xie at intel.com 
> <mailto:cindy.xie at intel.com>>; Lemus Contreras, Cristopher J 
> <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; '朱博祥' 
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> 'starlingx-discuss at lists.starlingx.io' 
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com 
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com 
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com 
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E 
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez 
> Monroy, Elio <elio.martinez.monroy at intel.com 
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> Hi Boxiang and all,
> 
> In fetch_to_raw(),
> 
> After force convert image from qcow2 to raw,  the format is still not 
> raw,  then throw below error.
> 
> It seems qemu-image convert image failed.
> 
> Zhipeng
> 
> *From:* Liu, ZhipengS <zhipengs.liu at intel.com 
> <mailto:zhipengs.liu at intel.com>>
> *Sent:* 2019年10月31日 9:35
> *To:* Xie, Cindy <cindy.xie at intel.com <mailto:cindy.xie at intel.com>>; 
> Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; '朱博祥' 
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> 'starlingx-discuss at lists.starlingx.io' 
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com 
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com 
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com 
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E 
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez 
> Monroy, Elio <elio.martinez.monroy at intel.com 
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> *From:* Liu, ZhipengS
> *Sent:* 2019年10月30日 23:24
> *To:* Xie, Cindy <cindy.xie at intel.com <mailto:cindy.xie at intel.com>>; 
> Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; 朱博祥
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com 
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com 
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com 
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E 
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez 
> Monroy, Elio <elio.martinez.monroy at intel.com 
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> Hi Boxiang,
> 
> It should be an issue in nova train.
> 
> Error log is below:
> 
> build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 
> 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to 
> raw, but format is now qcow2'
> 
> You tried to fix this issue before in nova LP:
> 
> https://bugs.launchpad.net/nova/+bug/1816686
> 
> In your patch: https://review.opendev.org/#/c/640271 (This patch is 
> already in train release now)
> 
> You added below commit message in the patch.
> 
> “When we import image into rbd, check the format of cache
> 
> images. If the format is not raw, remove it first and
> 
> fetch it again. It will be raw format now.”
> 
> However, it seems not work!  Below code will throw error when image 
> format is not raw and block the VM creation.
> 
> In nova/virt/images.py
> 
> def fetch_to_raw() è
> 
> if data.file_format != "raw":
> 
>                        raise
> exception.ImageUnacceptable(image_id=image_href,
> 
>                          reason=_("Converted to raw, but format is now 
> %s")
> 
> Not sure if your patch fixed this issue or is there any regression 
> introduced in nova code later.
> 
> Any comment from you?
> 
> I will check nova code further tomorrow and sync with you.
> 
> We may need a patch for nova☹
> 
> Thanks!
> 
> Zhipeng
> 
> *From:* Xie, Cindy <cindy.xie at intel.com <mailto:cindy.xie at intel.com>>
> *Sent:* 2019年10月30日 22:49
> *To:* Lemus Contreras, Cristopher J
> <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; Liu, ZhipengS 
> <zhipengs.liu at intel.com <mailto:zhipengs.liu at intel.com>>; 朱博祥
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com 
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com 
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com 
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E 
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez 
> Monroy, Elio <elio.martinez.monroy at intel.com 
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> + mailing list in case anybody can help on debug the issue.
> 
> *From:* Lemus Contreras, Cristopher J 
> <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>
> *Sent:* Wednesday, October 30, 2019 7:45 PM
> *To:* Liu, ZhipengS <zhipengs.liu at intel.com 
> <mailto:zhipengs.liu at intel.com>>; 朱博祥 <zhu.boxiang at 99cloud.net 
> <mailto:zhu.boxiang at 99cloud.net>>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com 
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com 
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com 
> <mailto:ada.cabrales at intel.com>>; Xie, Cindy <cindy.xie at intel.com 
> <mailto:cindy.xie at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com 
> <mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio 
> <elio.martinez.monroy at intel.com 
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack 
> train upgrade
> 
> Hi Zhipeng,
> 
> The results are OK for Simplex, Duplex and Standard Local Storage, all 
> Test Cases PASSED.
> 
> For Standard External Storage (2+2+2) there is a high quantity of Test 
> Cases failing because we cannot launch instances. Here’s a sample
> output: http://paste.openstack.org/show/785641/ with the error that 
> matches what is described on the launchpad.
> 
> Sanity summary for this specific configuration: 
> http://paste.openstack.org/show/785655/
> 
> Please, let us know if you need additional details.
> 
> Thanks & Regards,
> 
> Cristopher Lemus
>

More information about the Starlingx-discuss mailing list