Re: [Starlingx-discuss] About EB verification for openstack train upgrade

30 Oct 2019

      Boxiang,

Yes, I just want to check with you about images_type.
Do we really need to change “images_type” to rbd from default, is it related to format convert?
You patch code should already be hit, as we failed with EB307 with below error msg you added in your patch.

if CONF.libvirt.images_type in ('rbd',):
            if not CONF.force_raw_images:
                msg = _("'[DEFAULT]/force_raw_images = False' is not "
                        "allowed with '[libvirt]/images_type = rbd'. "
                        "Please check the two configs and if you really "
                        "do want to use rbd as images_type, set "
                        "force_raw_images to True.")
                raise exception.InvalidConfiguration(msg)

Below command is sent in my virtual multi-node environment, not in EB test environment.
[root@compute-0 instances]# rbd ls -p vms
rbd: error opening pool 'vms': (2) No such file or directory
rbd: list: (2) No such file or directory
[root@compute-0 instances]# ls
compute_nodes  locks  lost+found

Unfortunately, we have no whole test log.
@Lemus Contreras, Cristopher J<mailto:cristopher.j.lemus.contreras@intel.com>  Need your help to provide complete logs tested by EB308

Thanks!
Zhipeng

From: 朱博祥 <zhu.boxiang@99cloud.net>
Sent: 2019年10月31日 12:06
To: Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; starlingx-discuss@lists.starlingx.io
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>
Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng and All,

First of all, I found in your nova.conf, images_type = default
So I think you just use qcow2 imagebackend and the root disk is on your nova-compute node
not in vms pool of ceph storage. You can check that with command 'rbd ls -p vms'.

So that the creating vm process will not trigger my code in this patch
https://review.opendev.org/#/c/640271

From the nova code, in nova/virt/images.py
def fetch_to_raw():  ->
    convert_image(path_tmp, staged, fmt, 'raw') ->
        cmd = ('qemu-img', 'convert', '-t', cache_mode, '-O', out_format)
So if the format of image is not raw, it will call qemu-img command to convert it to raw.
But after converting and check the format of image again, it is still not raw, so it raise
exception.

So we can check the nova-compute logs whether other error msgs exist or not.
BTW, can you check the base image's format under '/var/lib/nova/instances/_base'?
You can find the image in nova-compute container.

--
Thanks,
Boxiang

From: "Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>
Date: 2019-10-31 11:38:36
To: "Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>,"朱博祥" <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>,"Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>,"starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>
Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>,"Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>,"Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>,"Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Below is current test config.
[libvirt]
connection_uri = qemu+tcp://127.0.0.1/system
cpu_mode = host-model
disk_cachemodes = network=writeback
hw_disk_discard = unmap
images_rbd_ceph_conf = /etc/ceph/ceph.conf
images_rbd_pool = vms
images_type = default
live_migration_completion_timeout = 180
live_migration_inbound_addr = 192.168.206.150
live_migration_permit_auto_converge = true
mem_stats_period_seconds = 0
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
rbd_user = cinder
remove_unused_resized_minimum_age_seconds = 86400
virt_type = qemu

I also did below convert command manually, it seems OK.
controller-0:~$ qemu-img convert -t none -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test1.img
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test2.img
controller-0:~$ qemu-img info test1.img
image: test1.img
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M
controller-0:~$ qemu-img info test2.img
image: test2.img
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M

Need further check why in the code after convert, the image format is still qcow2.

Zhipeng
From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>
Sent: 2019年10月31日 11:12
To: 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in
stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently.
https://review.opendev.org/#/c/691802/
I will modify my patch.
However, it is not related to current issue since this flag is set to true in the test.
The issue is just as error log mentioned below.
“ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ”
We need check this point further.

Thanks!
Zhipeng

From: 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>
Sent: 2019年10月31日 10:41
To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage).
In my nova.conf file, the section of libvirt is as followed:
[libvirt]
connection_uri = qemu+tcp://172.16.140.14/system
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
disk_cachemodes = network=writeback
hw_disk_discard = unmap
rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9
virt_type = kvm

The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file.
I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder.
It's my image's info:
(nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img
image: cirros-0.4.0-x86_64-disk.img
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX?
Are there other error logs in nova-compute or other nova-* services?

--
Thanks,
Boxiang

发件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>
发送日期：2019-10-31 10:14:44
收件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>,"Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>,"'朱博祥'" <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>
抄送人："Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>,"Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>,"Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>,"Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
主题：RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,

In fetch_to_raw(),
After force convert image from qcow2 to raw,  the format is still not raw,  then throw below error.
It seems qemu-image convert image failed.

Zhipeng
From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>
Sent: 2019年10月31日 9:35
To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

From: Liu, ZhipengS
Sent: 2019年10月30日 23:24
To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'

You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.

“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”

However, it seems not work!  Below code will throw error when image format is not raw and block the VM creation.

In nova/virt/images.py
def fetch_to_raw()  ==>
if data.file_format != "raw":
                      raise exception.ImageUnacceptable(image_id=image_href,
                        reason=_("Converted to raw, but format is now %s")

Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹

Thanks!
Zhipeng
From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>
Sent: 2019年10月30日 22:49
To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade

+ mailing list in case anybody can help on debug the issue.

From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>
Sent: Wednesday, October 30, 2019 7:45 PM
To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>
Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.

For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.

Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/

Please, let us know if you need additional details.

Thanks & Regards,

Cristopher Lemus