Hi all,

After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.

It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code.  We are not sure how/where we got this image file.

Thanks!

Zhipeng

From: Liu, ZhipengS
Sent: 2019年10月31日 17:31
To: Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

For remote storage configuration,  we have overridden below configurations in helm/nova.py.

     if remote_storage:

default_config.update({'force_raw_images': True})

            libvirt_config.update({'images_type': 'rbd',

                                   'images_rbd_pool': rbd_pool,

                                   'images_rbd_ceph_conf': rbd_ceph_conf})

        else:

            libvirt_config.update({'images_type': 'default'}) 

Next step:

The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.

I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.

(I copied yellow code block to else branch)

In /var/lib/nova/instances/_base/ folder,

I can see one qcow2 image and one raw image which should be converted by nova.

[root@compute-1 _base]# ls

3cb926303b665709318d2c099722b914025023b4  6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4

image: 3cb926303b665709318d2c099722b914025023b4

file format: qcow2

virtual size: 44M (46137344 bytes)

disk size: 12M

cluster_size: 65536

Format specific information:

    compat: 1.1

    lazy refcounts: false

    refcount bits: 16

    corrupt: false

[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

file format: raw

virtual size: 44M (46137344 bytes)

disk size: 13M

Any comment?

Thanks!

zhipeng

From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: 2019年10月31日 10:15
To: Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

In fetch_to_raw(),

After force convert image from qcow2 to raw,  the format is still not raw,  then throw below error.

It seems qemu-image convert image failed.

Zhipeng

From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: 2019年10月31日 9:35
To: Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

From: Liu, ZhipengS
Sent: 2019年10月30日 23:24
To: Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

It should be an issue in nova train.

Error log is below:

build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'

You tried to fix this issue before in nova LP:

https://bugs.launchpad.net/nova/+bug/1816686

In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)

You added below commit message in the patch.

“When we import image into rbd, check the format of cache

images. If the format is not raw, remove it first and

fetch it again. It will be raw format now.”

However, it seems not work!  Below code will throw error when image format is not raw and block the VM creation.

In nova/virt/images.py

def fetch_to_raw()  è

if data.file_format != "raw":

                      raise exception.ImageUnacceptable(image_id=image_href,

                        reason=_("Converted to raw, but format is now %s")

Not sure if your patch fixed this issue or is there any regression introduced in nova code later.

Any comment from you? 

I will check nova code further tomorrow and sync with you.

We may need a patch for nova☹

Thanks!

Zhipeng

From: Xie, Cindy <cindy.xie@intel.com>
Sent: 2019年10月30日 22:49
To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

+ mailing list in case anybody can help on debug the issue.

From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>
Sent: Wednesday, October 30, 2019 7:45 PM
To: Liu, ZhipengS <zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.

For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.

Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/

Please, let us know if you need additional details.

Thanks & Regards,

Cristopher Lemus