[Starlingx-discuss] About EB verification for openstack train upgrade

Lemus Contreras, Cristopher J cristopher.j.lemus.contreras at intel.com
Fri Nov 1 20:53:45 UTC 2019 

Hello,

Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.

AIO – Simplex

Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  49 TCs [PASS]
Sanity Platform   07 TCs [PASS]

TOTAL: [ 61 TCs PASS ]

AIO – Duplex

Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  52 TCs [PASS]
Sanity Platform   07 TCs [PASS]

TOTAL: [ 64 TCs PASS ]

Standard - Local Storage (2+2)

Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  52 TCs [PASS]
Sanity Platform   08 TCs [PASS]

TOTAL: [ 65 TCs PASS ]

Standard - Dedicated Storage (2+2+2)

Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  52 TCs [PASS]
Sanity Platform   09 TCs [PASS]

TOTAL: [ 66 TCs PASS ]

===========================================

Just the cirros image was replaced. Centos file is still a .qcow2.

Please, let me know if you need further details.

Thanks & Regards,

Cristopher Lemus

From: "Liu, ZhipengS" <zhipengs.liu at intel.com>
Date: Friday, November 1, 2019 at 12:02 AM
To: "Xie, Cindy" <cindy.xie at intel.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>, '朱博祥' <zhu.boxiang at 99cloud.net>, "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io>
Cc: "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "Hu, Yong" <yong.hu at intel.com>, "Cabrales, Ada" <ada.cabrales at intel.com>, "Jones, Bruce E" <bruce.e.jones at intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi all,

After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.
It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code.  We are not sure how/where we got this image file.

Thanks!
Zhipeng
From: Liu, ZhipengS
Sent: 2019年10月31日 17:31
To: Xie, Cindy <cindy.xie at intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; '朱博祥' <zhu.boxiang at 99cloud.net>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

For remote storage configuration,  we have overridden below configurations in helm/nova.py.
     if remote_storage:
            default_config.update({'force_raw_images': True})
            libvirt_config.update({'images_type': 'rbd',
                                   'images_rbd_pool': rbd_pool,
                                   'images_rbd_ceph_conf': rbd_ceph_conf})
        else:
            libvirt_config.update({'images_type': 'default'})

Next step:
The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.

I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.
(I copied yellow code block to else branch)

In /var/lib/nova/instances/_base/ folder,
I can see one qcow2 image and one raw image which should be converted by nova.

[root at compute-1 _base]# ls
3cb926303b665709318d2c099722b914025023b4  6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
[root at compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4
image: 3cb926303b665709318d2c099722b914025023b4
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
[root at compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M

Any comment?

Thanks!
zhipeng
From: Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>
Sent: 2019年10月31日 10:15
To: Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>; '朱博祥' <zhu.boxiang at 99cloud.net<mailto:zhu.boxiang at 99cloud.net>>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com<mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

In fetch_to_raw(),
After force convert image from qcow2 to raw,  the format is still not raw,  then throw below error.
It seems qemu-image convert image failed.

Zhipeng
From: Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>
Sent: 2019年10月31日 9:35
To: Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>; '朱博祥' <zhu.boxiang at 99cloud.net<mailto:zhu.boxiang at 99cloud.net>>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com<mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

From: Liu, ZhipengS
Sent: 2019年10月30日 23:24
To: Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>; 朱博祥 <zhu.boxiang at 99cloud.net<mailto:zhu.boxiang at 99cloud.net>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com<mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'

You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.

“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”

However, it seems not work!  Below code will throw error when image format is not raw and block the VM creation.

In nova/virt/images.py
def fetch_to_raw()  ==>
if data.file_format != "raw":
                      raise exception.ImageUnacceptable(image_id=image_href,
                        reason=_("Converted to raw, but format is now %s")

Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹

Thanks!
Zhipeng
From: Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>
Sent: 2019年10月30日 22:49
To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>; Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>; 朱博祥 <zhu.boxiang at 99cloud.net<mailto:zhu.boxiang at 99cloud.net>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com<mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

+ mailing list in case anybody can help on debug the issue.

From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>
Sent: Wednesday, October 30, 2019 7:45 PM
To: Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>; 朱博祥 <zhu.boxiang at 99cloud.net<mailto:zhu.boxiang at 99cloud.net>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com<mailto:juan.carlos.alonso at intel.com>>; Hu, Yong <yong.hu at intel.com<mailto:yong.hu at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.

For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.

Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/

Please, let us know if you need additional details.

Thanks & Regards,

Cristopher Lemus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20191101/b3672ee5/attachment-0001.html>

More information about the Starlingx-discuss mailing list