Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX? In addition, @Zhipeng, we might get this image to dig out the root-cause further. Regards, Yong On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
Hello,
Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.
*AIO – Simplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
*AIO – Duplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
*Standard - Local Storage (2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
*Standard - Dedicated Storage (2+2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Just the cirros image was replaced. Centos file is still a .qcow2.
Please, let me know if you need further details.
Thanks & Regards,
Cristopher Lemus
*From: *"Liu, ZhipengS" <zhipengs.liu@intel.com> *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" <cindy.xie@intel.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, '朱博祥' <zhu.boxiang@99cloud.net>, "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com> *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.
It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks!
Zhipeng
*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py.
if remote_storage:
default_config.update({'force_raw_images': True})
libvirt_config.update({'images_type': 'rbd',
'images_rbd_pool': rbd_pool,
'images_rbd_ceph_conf': rbd_ceph_conf})
else:
libvirt_config.update({'images_type': 'default'})
Next step:
The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.
(I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder,
I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls
3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4
image: 3cb926303b665709318d2c099722b914025023b4
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M
Any comment?
Thanks!
zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(),
After force convert image from qcow2 to raw, the format is still not raw, then throw below error.
It seems qemu-image convert image failed.
Zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.
“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py
def fetch_to_raw() è
if data.file_format != "raw":
raise exception.ImageUnacceptable(image_id=image_href,
reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹
Thanks!
Zhipeng
*From:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus