[Starlingx-discuss] About EB verification for openstack train upgrade

Thu Oct 31 03:37:33 UTC 2019

From the nova code, in nova/virt/images.py
def fetch_to_raw():  ->
    convert_image(path_tmp, staged, fmt, 'raw') ->
        cmd = ('qemu-img', 'convert', '-t', cache_mode, '-O', out_format)
So if the format of image is not raw, it will call qemu-img command to convert it to raw.
But now we got  

--
Thanks,
Boxiang

发件人："Liu, ZhipengS" <zhipengs.liu at intel.com>
发送日期：2019-10-31 11:11:37
收件人："朱博祥" <zhu.boxiang at 99cloud.net>,"Xie, Cindy" <cindy.xie at intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>,"'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io>
抄送人："Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>,"Hu, Yong" <yong.hu at intel.com>,"Cabrales, Ada" <ada.cabrales at intel.com>,"Jones, Bruce E" <bruce.e.jones at intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy at intel.com>
主题：RE: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,

Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in
stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently.
https://review.opendev.org/#/c/691802/
I will modify my patch.
However, it is not related to current issue since this flag is set to true in the test.
The issue is just as error log mentioned below.
“ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ”
We need check this point further.

Thanks!
Zhipeng

From: 朱博祥 <zhu.boxiang at 99cloud.net>
 Sent: 2019年10月31日 10:41
 To: Liu, ZhipengS <zhipengs.liu at intel.com>; Xie, Cindy <cindy.xie at intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
 Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
 Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage).

In my nova.conf file, the section of libvirt is as followed:

[libvirt]

connection_uri = qemu+tcp://172.16.140.14/system

images_type = rbd

images_rbd_pool = vms

images_rbd_ceph_conf = /etc/ceph/ceph.conf

rbd_user = nova

disk_cachemodes = network=writeback

hw_disk_discard = unmap

rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9

virt_type = kvm

The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file.

I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder.

It's my image's info:

(nova-compute)[root at zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img

image: cirros-0.4.0-x86_64-disk.img

file format: qcow2

virtual size: 44M (46137344 bytes)

disk size: 12M

cluster_size: 65536

Format specific information:

    compat: 1.1

    lazy refcounts: false

    refcount bits: 16

    corrupt: false

So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX?

Are there other error logs in nova-compute or other nova-* services?

--
Thanks,

Boxiang

发件人："Liu, ZhipengS" <zhipengs.liu at intel.com>
 发送日期：2019-10-31 10:14:44
 收件人："Liu, ZhipengS" <zhipengs.liu at intel.com>,"Xie, Cindy" <cindy.xie at intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com>,"'朱博祥'" <zhu.boxiang at 99cloud.net>,"'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io>
 抄送人："Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>,"Hu, Yong" <yong.hu at intel.com>,"Cabrales, Ada" <ada.cabrales at intel.com>,"Jones, Bruce E" <bruce.e.jones at intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy at intel.com>
 主题：RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

In fetch_to_raw(),
After force convert image from qcow2 to raw,  the format is still not raw,  then throw below error.
It seems qemu-image convert image failed.

Zhipeng
From: Liu, ZhipengS <zhipengs.liu at intel.com>
 Sent: 2019年10月31日 9:35
 To: Xie, Cindy <cindy.xie at intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; '朱博祥' <zhu.boxiang at 99cloud.net>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
 Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
 Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

From: Liu, ZhipengS
 Sent: 2019年10月30日 23:24
 To: Xie, Cindy <cindy.xie at intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; 朱博祥 <zhu.boxiang at 99cloud.net>; starlingx-discuss at lists.starlingx.io
 Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
 Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'

You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.

“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”

However, it seems not work!  Below code will throw error when image format is not raw and block the VM creation.

In nova/virt/images.py
def fetch_to_raw()  è
if data.file_format != "raw":
                      raise exception.ImageUnacceptable(image_id=image_href,
                        reason=_("Converted to raw, but format is now %s")

Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you? 
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹

Thanks!
Zhipeng
From: Xie, Cindy <cindy.xie at intel.com>
 Sent: 2019年10月30日 22:49
 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>; Liu, ZhipengS <zhipengs.liu at intel.com>; 朱博祥 <zhu.boxiang at 99cloud.net>; starlingx-discuss at lists.starlingx.io
 Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
 Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

+ mailing list in case anybody can help on debug the issue.

From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras at intel.com>
 Sent: Wednesday, October 30, 2019 7:45 PM
 To: Liu, ZhipengS <zhipengs.liu at intel.com>; 朱博祥 <zhu.boxiang at 99cloud.net>
 Cc: Alonso, Juan Carlos <juan.carlos.alonso at intel.com>; Hu, Yong <yong.hu at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Xie, Cindy <cindy.xie at intel.com>; Jones, Bruce E <bruce.e.jones at intel.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>
 Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.

For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.

Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/

Please, let us know if you need additional details.

Thanks & Regards,

Cristopher Lemus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20191031/884fde8c/attachment-0001.html>