Hi Boxiang and all, In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2' You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch. “When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.” However, it seems not work! Below code will throw error when image format is not raw and block the VM creation. In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s") Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹ Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade + mailing list in case anybody can help on debug the issue. From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED. For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad. Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/ Please, let us know if you need additional details. Thanks & Regards, Cristopher Lemus

Hi Boxiang, Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further. Thanks! Zhipeng From: 朱博祥 <zhu.boxiang@99cloud.net> Sent: 2019年10月31日 10:41 To: Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services? -- Thanks, Boxiang 发件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> 发送日期：2019-10-31 10:14:44 收件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>,"Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>,"'朱博祥'" <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> 抄送人："Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>,"Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>,"Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>,"Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> 主题：RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all, In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2' You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch. “When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.” However, it seems not work! Below code will throw error when image format is not raw and block the VM creation. In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s") Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹ Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade + mailing list in case anybody can help on debug the issue. From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED. For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad. Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/ Please, let us know if you need additional details. Thanks & Regards, Cristopher Lemus

Below is current test config. [libvirt] connection_uri = qemu+tcp://127.0.0.1/system cpu_mode = host-model disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = default live_migration_completion_timeout = 180 live_migration_inbound_addr = 192.168.206.150 live_migration_permit_auto_converge = true mem_stats_period_seconds = 0 rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337 rbd_user = cinder remove_unused_resized_minimum_age_seconds = 86400 virt_type = qemu I also did below convert command manually, it seems OK. controller-0:~$ qemu-img convert -t none -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test1.img controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test2.img controller-0:~$ qemu-img info test1.img image: test1.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M controller-0:~$ qemu-img info test2.img image: test2.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M Need further check why in the code after convert, the image format is still qcow2. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com> Sent: 2019年10月31日 11:12 To: 朱博祥 <zhu.boxiang@99cloud.net>; Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further. Thanks! Zhipeng From: 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Sent: 2019年10月31日 10:41 To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services? -- Thanks, Boxiang 发件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> 发送日期：2019-10-31 10:14:44 收件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>,"Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>,"'朱博祥'" <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> 抄送人："Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>,"Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>,"Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>,"Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> 主题：RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all, In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2' You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch. “When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.” However, it seems not work! Below code will throw error when image format is not raw and block the VM creation. In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s") Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹ Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade + mailing list in case anybody can help on debug the issue. From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED. For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad. Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/ Please, let us know if you need additional details. Thanks & Regards, Cristopher Lemus

Boxiang, Yes, I just want to check with you about images_type. Do we really need to change “images_type” to rbd from default, is it related to format convert? You patch code should already be hit, as we failed with EB307 with below error msg you added in your patch. if CONF.libvirt.images_type in ('rbd',): if not CONF.force_raw_images: msg = _("'[DEFAULT]/force_raw_images = False' is not " "allowed with '[libvirt]/images_type = rbd'. " "Please check the two configs and if you really " "do want to use rbd as images_type, set " "force_raw_images to True.") raise exception.InvalidConfiguration(msg) Below command is sent in my virtual multi-node environment, not in EB test environment. [root@compute-0 instances]# rbd ls -p vms rbd: error opening pool 'vms': (2) No such file or directory rbd: list: (2) No such file or directory [root@compute-0 instances]# ls compute_nodes locks lost+found Unfortunately, we have no whole test log. @Lemus Contreras, Cristopher J<mailto:cristopher.j.lemus.contreras@intel.com> Need your help to provide complete logs tested by EB308 Thanks! Zhipeng From: 朱博祥 <zhu.boxiang@99cloud.net> Sent: 2019年10月31日 12:06 To: Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng and All, First of all, I found in your nova.conf, images_type = default So I think you just use qcow2 imagebackend and the root disk is on your nova-compute node not in vms pool of ceph storage. You can check that with command 'rbd ls -p vms'. So that the creating vm process will not trigger my code in this patch https://review.opendev.org/#/c/640271 From the nova code, in nova/virt/images.py def fetch_to_raw(): -> convert_image(path_tmp, staged, fmt, 'raw') -> cmd = ('qemu-img', 'convert', '-t', cache_mode, '-O', out_format) So if the format of image is not raw, it will call qemu-img command to convert it to raw. But after converting and check the format of image again, it is still not raw, so it raise exception. So we can check the nova-compute logs whether other error msgs exist or not. BTW, can you check the base image's format under '/var/lib/nova/instances/_base'? You can find the image in nova-compute container. -- Thanks, Boxiang From: "Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Date: 2019-10-31 11:38:36 To: "Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>,"朱博祥" <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>,"Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>,"starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>,"Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>,"Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>,"Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Below is current test config. [libvirt] connection_uri = qemu+tcp://127.0.0.1/system cpu_mode = host-model disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = default live_migration_completion_timeout = 180 live_migration_inbound_addr = 192.168.206.150 live_migration_permit_auto_converge = true mem_stats_period_seconds = 0 rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337 rbd_user = cinder remove_unused_resized_minimum_age_seconds = 86400 virt_type = qemu I also did below convert command manually, it seems OK. controller-0:~$ qemu-img convert -t none -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test1.img controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test2.img controller-0:~$ qemu-img info test1.img image: test1.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M controller-0:~$ qemu-img info test2.img image: test2.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M Need further check why in the code after convert, the image format is still qcow2. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: 2019年10月31日 11:12 To: 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further. Thanks! Zhipeng From: 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Sent: 2019年10月31日 10:41 To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services? -- Thanks, Boxiang 发件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> 发送日期：2019-10-31 10:14:44 收件人："Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>,"Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>,"'朱博祥'" <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> 抄送人："Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>,"Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>,"Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>,"Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> 主题：RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all, In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2' You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch. “When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.” However, it seems not work! Below code will throw error when image format is not raw and block the VM creation. In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s") Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹ Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade + mailing list in case anybody can help on debug the issue. From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED. For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad. Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/ Please, let us know if you need additional details. Thanks & Regards, Cristopher Lemus

Hi all, After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now. It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file. Thanks! Zhipeng From: Liu, ZhipengS Sent: 2019年10月31日 17:31 To: Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all, For remote storage configuration, we have overridden below configurations in helm/nova.py. if remote_storage: default_config.update({'force_raw_images': True}) libvirt_config.update({'images_type': 'rbd', 'images_rbd_pool': rbd_pool, 'images_rbd_ceph_conf': rbd_ceph_conf}) else: libvirt_config.update({'images_type': 'default'}) Next step: The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code. I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally. (I copied yellow code block to else branch) In /var/lib/nova/instances/_base/ folder, I can see one qcow2 image and one raw image which should be converted by nova. [root@compute-1 _base]# ls 3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee [root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4 image: 3cb926303b665709318d2c099722b914025023b4 file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false [root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee file format: raw virtual size: 44M (46137344 bytes) disk size: 13M Any comment? Thanks! zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: 2019年10月31日 10:15 To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all, In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed. Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang, It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2' You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch. “When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.” However, it seems not work! Below code will throw error when image format is not raw and block the VM creation. In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s") Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹ Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade + mailing list in case anybody can help on debug the issue. From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com<mailto:cristopher.j.lemus.contreras@intel.com>> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net<mailto:zhu.boxiang@99cloud.net>> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Zhipeng, The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED. For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad. Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/ Please, let us know if you need additional details. Thanks & Regards, Cristopher Lemus

Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX? In addition, @Zhipeng, we might get this image to dig out the root-cause further. Regards, Yong On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:

Hello,

Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.

*AIO – Simplex*

**

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]

TOTAL: [ 61 TCs PASS ]

*AIO – Duplex*

**

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]

TOTAL: [ 64 TCs PASS ]

*Standard - Local Storage (2+2)*

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]

TOTAL: [ 65 TCs PASS ]

*Standard - Dedicated Storage (2+2+2)*

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   09 TCs [PASS]

TOTAL: [ 66 TCs PASS ]

===========================================

Just the cirros image was replaced. Centos file is still a .qcow2.

Please, let me know if you need further details.

Thanks & Regards,

Cristopher Lemus

*From: *"Liu, ZhipengS" <zhipengs.liu@intel.com> *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" <cindy.xie@intel.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, '朱博祥' <zhu.boxiang@99cloud.net>, "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com> *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi all,

After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.

It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code.  We are not sure how/where we got this image file.

Thanks!

Zhipeng

*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

For remote storage configuration,  we have overridden below configurations in helm/nova.py.

     if remote_storage:

default_config.update({'force_raw_images': True})

            libvirt_config.update({'images_type': 'rbd',

                                   'images_rbd_pool': rbd_pool,

                                   'images_rbd_ceph_conf': rbd_ceph_conf})

        else:

            libvirt_config.update({'images_type': 'default'})

Next step:

The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.

I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.

(I copied yellow code block to else branch)

In /var/lib/nova/instances/_base/ folder,

I can see one qcow2 image and one raw image which should be converted by nova.

[root@compute-1 _base]# ls

3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4

image: 3cb926303b665709318d2c099722b914025023b4

file format: qcow2

virtual size: 44M (46137344 bytes)

disk size: 12M

cluster_size: 65536

Format specific information:

    compat: 1.1

    lazy refcounts: false

    refcount bits: 16

    corrupt: false

[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

file format: raw

virtual size: 44M (46137344 bytes)

disk size: 13M

Any comment?

Thanks!

zhipeng

*From:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

In fetch_to_raw(),

After force convert image from qcow2 to raw,  the format is still not raw,  then throw below error.

It seems qemu-image convert image failed.

Zhipeng

*From:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade

*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

It should be an issue in nova train.

Error log is below:

build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'

You tried to fix this issue before in nova LP:

https://bugs.launchpad.net/nova/+bug/1816686

In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)

You added below commit message in the patch.

“When we import image into rbd, check the format of cache

images. If the format is not raw, remove it first and

fetch it again. It will be raw format now.”

However, it seems not work!  Below code will throw error when image format is not raw and block the VM creation.

In nova/virt/images.py

def fetch_to_raw() è

if data.file_format != "raw":

                      raise exception.ImageUnacceptable(image_id=image_href,

                        reason=_("Converted to raw, but format is now %s")

Not sure if your patch fixed this issue or is there any regression introduced in nova code later.

Any comment from you?

I will check nova code further tomorrow and sync with you.

We may need a patch for nova☹

Thanks!

Zhipeng

*From:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

+ mailing list in case anybody can help on debug the issue.

*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.

For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.

Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/

Please, let us know if you need additional details.

Thanks & Regards,

Cristopher Lemus

Hi Yong and all, The root cause should be below. Download image from https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img It is already qcow2, if we use below command to convert to qcow2 again, it will be added with an additional qcow2 header. This is the problem!!! controller-0:~$ qemu-img convert -f raw -O qcow2 cirros-0.4.0-x86_64-disk.img cirros-0.4.0-x86_64-disk.qcow2 So, if we run test case in nova train release. It will just like below. controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.qcow2 test controller-0:~$ qemu-img info test image: test file format: qcow2 //it is still qcow2, just removed additional qcow header. virtual size: 44M (46137344 bytes) controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 test test22 controller-0:~$ qemu-img info test22 image: test22 file format: raw //after second convert, it becomes raw!!! virtual size: 44M (46137344 bytes) That's why we use the cirros-0.4.0-x86_64-disk.img downloaded from official website can pass sanity test. Thanks! Zhipeng -----Original Message----- From: Hu, Yong <yong.hu@intel.com> Sent: 2019年11月4日 21:56 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX? In addition, @Zhipeng, we might get this image to dig out the root-cause further. Regards, Yong On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:

Hello,

Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.

*AIO – Simplex*

**

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]

TOTAL: [ 61 TCs PASS ]

*AIO – Duplex*

**

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   07 TCs [PASS]

TOTAL: [ 64 TCs PASS ]

*Standard - Local Storage (2+2)*

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   08 TCs [PASS]

TOTAL: [ 65 TCs PASS ]

*Standard - Dedicated Storage (2+2+2)*

Setup             04 TCs [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   09 TCs [PASS]

TOTAL: [ 66 TCs PASS ]

===========================================

Just the cirros image was replaced. Centos file is still a .qcow2.

Please, let me know if you need further details.

Thanks & Regards,

Cristopher Lemus

*From: *"Liu, ZhipengS" <zhipengs.liu@intel.com> *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" <cindy.xie@intel.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, '朱博祥' <zhu.boxiang@99cloud.net>, "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com> *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi all,

After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.

It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code.  We are not sure how/where we got this image file.

Thanks!

Zhipeng

*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy <cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

For remote storage configuration,  we have overridden below configurations in helm/nova.py.

     if remote_storage:

default_config.update({'force_raw_images': True})

            libvirt_config.update({'images_type': 'rbd',

                                   'images_rbd_pool': rbd_pool,

                                   'images_rbd_ceph_conf': rbd_ceph_conf})

        else:

            libvirt_config.update({'images_type': 'default'})

Next step:

The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.

I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.

(I copied yellow code block to else branch)

In /var/lib/nova/instances/_base/ folder,

I can see one qcow2 image and one raw image which should be converted by nova.

[root@compute-1 _base]# ls

3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4

image: 3cb926303b665709318d2c099722b914025023b4

file format: qcow2

virtual size: 44M (46137344 bytes)

disk size: 12M

cluster_size: 65536

Format specific information:

    compat: 1.1

    lazy refcounts: false

    refcount bits: 16

    corrupt: false

[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee

file format: raw

virtual size: 44M (46137344 bytes)

disk size: 13M

Any comment?

Thanks!

zhipeng

*From:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang and all,

In fetch_to_raw(),

After force convert image from qcow2 to raw,  the format is still not raw,  then throw below error.

It seems qemu-image convert image failed.

Zhipeng

*From:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; '朱博祥' <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade

*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Boxiang,

It should be an issue in nova train.

Error log is below:

build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'

You tried to fix this issue before in nova LP:

https://bugs.launchpad.net/nova/+bug/1816686

In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)

You added below commit message in the patch.

“When we import image into rbd, check the format of cache

images. If the format is not raw, remove it first and

fetch it again. It will be raw format now.”

However, it seems not work!  Below code will throw error when image format is not raw and block the VM creation.

In nova/virt/images.py

def fetch_to_raw() è

if data.file_format != "raw":

                      raise exception.ImageUnacceptable(image_id=image_href,

                        reason=_("Converted to raw, but format is now %s")

Not sure if your patch fixed this issue or is there any regression introduced in nova code later.

Any comment from you?

I will check nova code further tomorrow and sync with you.

We may need a patch for nova☹

Thanks!

Zhipeng

*From:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade

+ mailing list in case anybody can help on debug the issue.

*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com <mailto:cristopher.j.lemus.contreras@intel.com>> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com <mailto:zhipengs.liu@intel.com>>; 朱博祥 <zhu.boxiang@99cloud.net <mailto:zhu.boxiang@99cloud.net>> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com <mailto:juan.carlos.alonso@intel.com>>; Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com <mailto:ada.cabrales@intel.com>>; Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com <mailto:elio.martinez.monroy@intel.com>> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade

Hi Zhipeng,

The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.

For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.

Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/

Please, let us know if you need additional details.

Thanks & Regards,

Cristopher Lemus