[Starlingx-discuss] About EB verification for openstack train upgrade
Liu, ZhipengS
zhipengs.liu at intel.com
Tue Nov 5 02:39:27 UTC 2019
Hi Yong and all,
The root cause should be below.
Download image from
https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
It is already qcow2, if we use below command to convert to qcow2 again, it will be added with an additional qcow2 header.
This is the problem!!!
controller-0:~$ qemu-img convert -f raw -O qcow2 cirros-0.4.0-x86_64-disk.img cirros-0.4.0-x86_64-disk.qcow2
So, if we run test case in nova train release. It will just like below.
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.qcow2 test
controller-0:~$ qemu-img info test
image: test
file format: qcow2 //it is still qcow2, just removed additional qcow header.
virtual size: 44M (46137344 bytes)
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 test test22
controller-0:~$ qemu-img info test22
image: test22
file format: raw //after second convert, it becomes raw!!!
virtual size: 44M (46137344 bytes)
That's why we use the cirros-0.4.0-x86_64-disk.img downloaded from official website can pass sanity test.
Thanks!
Zhipeng
-----Original Message-----
From: Hu, Yong <yong.hu at intel.com>
Sent: 2019年11月4日 21:56
To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; Liu, ZhipengS <zhipengs.liu at intel.com>; Xie, Cindy <cindy.xie at intel.com>; '朱博祥' <zhu.boxiang at 99cloud.net>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Christopher,
A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX?
In addition, @Zhipeng, we might get this image to dig out the root-cause further.
Regards,
Yong
On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
> Hello,
>
> Using the EB that Zhipeng provided, and replacing
> “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”,
> Sanity status is Green.
>
> *AIO – Simplex*
>
> **
>
> Setup 04 TCs [PASS]
>
> Provisioning 01 TCs [PASS]
>
> Sanity OpenStack 49 TCs [PASS]
>
> Sanity Platform 07 TCs [PASS]
>
> TOTAL: [ 61 TCs PASS ]
>
> *AIO – Duplex*
>
> **
>
> Setup 04 TCs [PASS]
>
> Provisioning 01 TCs [PASS]
>
> Sanity OpenStack 52 TCs [PASS]
>
> Sanity Platform 07 TCs [PASS]
>
> TOTAL: [ 64 TCs PASS ]
>
> *Standard - Local Storage (2+2)*
>
> Setup 04 TCs [PASS]
>
> Provisioning 01 TCs [PASS]
>
> Sanity OpenStack 52 TCs [PASS]
>
> Sanity Platform 08 TCs [PASS]
>
> TOTAL: [ 65 TCs PASS ]
>
> *Standard - Dedicated Storage (2+2+2)*
>
> Setup 04 TCs [PASS]
>
> Provisioning 01 TCs [PASS]
>
> Sanity OpenStack 52 TCs [PASS]
>
> Sanity Platform 09 TCs [PASS]
>
> TOTAL: [ 66 TCs PASS ]
>
> ===========================================
>
> Just the cirros image was replaced. Centos file is still a .qcow2.
>
> Please, let me know if you need further details.
>
> Thanks & Regards,
>
> Cristopher Lemus
>
> *From: *"Liu, ZhipengS" <zhipengs.liu at intel.com>
> *Date: *Friday, November 1, 2019 at 12:02 AM
> *To: *"Xie, Cindy" <cindy.xie at intel.com>, "Lemus Contreras, Cristopher
> J" <cristopher.j.lemus.contreras at intel.com>, '朱博祥'
> <zhu.boxiang at 99cloud.net>, "'starlingx-discuss at lists.starlingx.io'"
> <starlingx-discuss at lists.starlingx.io>
> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Hu, Yong"
> <yong.hu at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>, "Jones,
> Bruce E" <bruce.e.jones at intel.com>, "Martinez Monroy, Elio"
> <elio.martinez.monroy at intel.com>
> *Subject: *RE: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> Hi all,
>
> After using new image file (cirros-0.4.0-x86_64-disk.img), we can
> launch instance on standard external Storage (2+2+2) now.
>
> It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by
> Cristopher Lemus causes qemu-img detecting wrong format after format
> convert from qcow2to raw in Nova code. We are not sure how/where we
> got this image file.
>
> Thanks!
>
> Zhipeng
>
> *From:* Liu, ZhipengS
> *Sent:* 2019年10月31日 17:31
> *To:* Xie, Cindy <cindy.xie at intel.com>; Lemus Contreras, Cristopher J
> <cristopher.j.lemus.contreras at intel.com>; '朱博祥'
> <zhu.boxiang at 99cloud.net>; 'starlingx-discuss at lists.starlingx.io'
> <starlingx-discuss at lists.starlingx.io>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong
> <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones,
> Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio
> <elio.martinez.monroy at intel.com>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> Hi Boxiang and all,
>
> For remote storage configuration, we have overridden below
> configurations in helm/nova.py.
>
> if remote_storage:
>
> default_config.update({'force_raw_images': True})
>
> libvirt_config.update({'images_type': 'rbd',
>
> 'images_rbd_pool': rbd_pool,
>
> 'images_rbd_ceph_conf':
> rbd_ceph_conf})
>
> else:
>
> libvirt_config.update({'images_type': 'default'})
>
> Next step:
>
> The only issue we need to figure out is why image format is still
> qcow2 after image format convert by qemu-img in nova code.
>
> I tried to reproduce it in my virtualized multi-node environment, but
> no issue found, VM created normally.
>
> (I copied yellow code block to else branch)
>
> In /var/lib/nova/instances/_base/ folder,
>
> I can see one qcow2 image and one raw image which should be converted
> by nova.
>
> [root at compute-1 _base]# ls
>
> 3cb926303b665709318d2c099722b914025023b4
> 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
>
> [root at compute-1 _base]# qemu-img info
> 3cb926303b665709318d2c099722b914025023b4
>
> image: 3cb926303b665709318d2c099722b914025023b4
>
> file format: qcow2
>
> virtual size: 44M (46137344 bytes)
>
> disk size: 12M
>
> cluster_size: 65536
>
> Format specific information:
>
> compat: 1.1
>
> lazy refcounts: false
>
> refcount bits: 16
>
> corrupt: false
>
> [root at compute-1 _base]# qemu-img info
> 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
>
> image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
>
> file format: raw
>
> virtual size: 44M (46137344 bytes)
>
> disk size: 13M
>
> Any comment?
>
> Thanks!
>
> zhipeng
>
> *From:* Liu, ZhipengS <zhipengs.liu at intel.com
> <mailto:zhipengs.liu at intel.com>>
> *Sent:* 2019年10月31日 10:15
> *To:* Liu, ZhipengS <zhipengs.liu at intel.com
> <mailto:zhipengs.liu at intel.com>>; Xie, Cindy <cindy.xie at intel.com
> <mailto:cindy.xie at intel.com>>; Lemus Contreras, Cristopher J
> <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; '朱博祥'
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> 'starlingx-discuss at lists.starlingx.io'
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez
> Monroy, Elio <elio.martinez.monroy at intel.com
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> Hi Boxiang and all,
>
> In fetch_to_raw(),
>
> After force convert image from qcow2 to raw, the format is still not
> raw, then throw below error.
>
> It seems qemu-image convert image failed.
>
> Zhipeng
>
> *From:* Liu, ZhipengS <zhipengs.liu at intel.com
> <mailto:zhipengs.liu at intel.com>>
> *Sent:* 2019年10月31日 9:35
> *To:* Xie, Cindy <cindy.xie at intel.com <mailto:cindy.xie at intel.com>>;
> Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; '朱博祥'
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> 'starlingx-discuss at lists.starlingx.io'
> <starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez
> Monroy, Elio <elio.martinez.monroy at intel.com
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> *From:* Liu, ZhipengS
> *Sent:* 2019年10月30日 23:24
> *To:* Xie, Cindy <cindy.xie at intel.com <mailto:cindy.xie at intel.com>>;
> Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; 朱博祥
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez
> Monroy, Elio <elio.martinez.monroy at intel.com
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> Hi Boxiang,
>
> It should be an issue in nova train.
>
> Error log is below:
>
> build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image
> 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to
> raw, but format is now qcow2'
>
> You tried to fix this issue before in nova LP:
>
> https://bugs.launchpad.net/nova/+bug/1816686
>
> In your patch: https://review.opendev.org/#/c/640271 (This patch is
> already in train release now)
>
> You added below commit message in the patch.
>
> “When we import image into rbd, check the format of cache
>
> images. If the format is not raw, remove it first and
>
> fetch it again. It will be raw format now.”
>
> However, it seems not work! Below code will throw error when image
> format is not raw and block the VM creation.
>
> In nova/virt/images.py
>
> def fetch_to_raw() è
>
> if data.file_format != "raw":
>
> raise
> exception.ImageUnacceptable(image_id=image_href,
>
> reason=_("Converted to raw, but format is now
> %s")
>
> Not sure if your patch fixed this issue or is there any regression
> introduced in nova code later.
>
> Any comment from you?
>
> I will check nova code further tomorrow and sync with you.
>
> We may need a patch for nova☹
>
> Thanks!
>
> Zhipeng
>
> *From:* Xie, Cindy <cindy.xie at intel.com <mailto:cindy.xie at intel.com>>
> *Sent:* 2019年10月30日 22:49
> *To:* Lemus Contreras, Cristopher J
> <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>; Liu, ZhipengS
> <zhipengs.liu at intel.com <mailto:zhipengs.liu at intel.com>>; 朱博祥
> <zhu.boxiang at 99cloud.net <mailto:zhu.boxiang at 99cloud.net>>;
> starlingx-discuss at lists.starlingx.io
> <mailto:starlingx-discuss at lists.starlingx.io>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com
> <mailto:ada.cabrales at intel.com>>; Jones, Bruce E
> <bruce.e.jones at intel.com <mailto:bruce.e.jones at intel.com>>; Martinez
> Monroy, Elio <elio.martinez.monroy at intel.com
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> + mailing list in case anybody can help on debug the issue.
>
> *From:* Lemus Contreras, Cristopher J
> <cristopher.j.lemus.contreras at intel.com
> <mailto:cristopher.j.lemus.contreras at intel.com>>
> *Sent:* Wednesday, October 30, 2019 7:45 PM
> *To:* Liu, ZhipengS <zhipengs.liu at intel.com
> <mailto:zhipengs.liu at intel.com>>; 朱博祥 <zhu.boxiang at 99cloud.net
> <mailto:zhu.boxiang at 99cloud.net>>
> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso at intel.com
> <mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com
> <mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com
> <mailto:ada.cabrales at intel.com>>; Xie, Cindy <cindy.xie at intel.com
> <mailto:cindy.xie at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com
> <mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio
> <elio.martinez.monroy at intel.com
> <mailto:elio.martinez.monroy at intel.com>>
> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack
> train upgrade
>
> Hi Zhipeng,
>
> The results are OK for Simplex, Duplex and Standard Local Storage, all
> Test Cases PASSED.
>
> For Standard External Storage (2+2+2) there is a high quantity of Test
> Cases failing because we cannot launch instances. Here’s a sample
> output: http://paste.openstack.org/show/785641/ with the error that
> matches what is described on the launchpad.
>
> Sanity summary for this specific configuration:
> http://paste.openstack.org/show/785655/
>
> Please, let us know if you need additional details.
>
> Thanks & Regards,
>
> Cristopher Lemus
>
More information about the Starlingx-discuss
mailing list