Re: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Xie, Cindy cindy.xie@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
From: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Date: Tuesday, October 29, 2019 at 7:23 PM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>, 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>, "Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Cristopher Lemus,
Yes, exactly! Do we have a test result with my latest EB now? Thank Boxiang for clarification! I have included related change in latest EB 308.
Thanks! Zhipeng
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: 2019年10月30日 4:16 To: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi All,
I think that we are facing the exact same issue that Boxiang mentions, it’s a failure for Standard external storage (2+2+2).
We are not able to launch instances on Standard with External Storage, here’s a sample output: http://paste.openstack.org/show/785641/, the error matches the bug: https://bugs.launchpad.net/starlingx/+bug/1837241, which was reverted on the review that Boxiang listed.
The rest of the configurations are not affected, this is specific to external storage.
Thanks ,
Cristopher Lemus
From: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Date: Tuesday, October 29, 2019 at 6:15 AM To: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>, "Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
Yeah, now when we use the openstack train, we should set force_raw_images to True when we use rbd as images_types in nova.conf. Otherwise the nova-compute service will fail to start. The patch[0] was meant to do that before but it caused the sanity test failure and we didn't find the root cause.
[0] https://review.opendev.org/#/c/661512/
Thanks, Boxiang
From: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Date: 2019-10-29 10:54:13
To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>,"Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>,"Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>,"朱博祥" <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>
Cc: "Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>,"Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+baoxiang
Hi JC,
Thanks for your quick update!
I have found the root cause from your logs.
{"log":"2019-10-25 15:36:06.371 235059 ERROR oslo_service.service [-] Error starting thread.: InvalidConfiguration: '[DEFAULT]/force_raw_images = False' is not allowed with '[libvirt]/images_type = rbd'. Please check the two configs and if you really do want to use rbd as images_type, set force_raw_images to True.\n","stream":"stdout","time":"2019-10-25T15:36:06.372419534Z"}
This is related to an old LP
Last time, I revert the patch provided by Boxiang due to we were using staging nova version with some of Starlingx patches.
Now, we used upstream Nova, we need set force_raw_images = True again.
@Boxiang, any comment or additional change needed?
@JC, I will provide EB to you later for this issue.
Thanks!
Zhipeng
-----Original Message-----
From: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>
Sent: 2019年10月29日 5:33
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Standard External Storage failed again, same issue.
Logs available here: http://10.219.115.222/ISOS/ALL_NODES_20191028.125727.tar
See the log attached too.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Alonso, Juan Carlos
Sent: Monday, October 28, 2019 9:45 AM
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <Cristopher.J.Lemus.Contreras@intel.commailto:Cristopher.J.Lemus.Contreras@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hello,
Deployment and sanity execution was launched last Friday.
AIO-SX, AIO-DX and Standard Local Storage deployed successfully and sanity passed.
Cristopher found that for Standard External Storage there was a failure with nova-compute pods, this is causing system application-apply stuck at 54%.
Cristopher also put the logs from collect here: http://10.219.115.222/ISOS/ALL_NODES_20191025.153624.tar
Today only Standard External Storage was launched in BareMetal and virtual environments to assure this is a real issue. Will let you know.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Sent: Friday, October 25, 2019 12:22 AM
To: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
Please use below EB, which based on 1024 daily build
Thanks!
Zhipeng
-----Original Message-----
From: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>
Sent: 2019年10月25日 3:24
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hello ZhipengS,
Based on last StarlingX release meeting, we need a new EB with the fix merged since we have to execute sanity in both standard configurations.
Please can you generate a new EB to star testing?
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Sent: Wednesday, October 23, 2019 8:47 PM
To: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
Thanks for your update!
According to your test result, it seems just some known issues caused by other changes seen in your test.
Standards setup failure should be introduced in recent code merge.
Since you passed this case when you used my first EB that based on earlier daily build.
BTW, I also passed simplex test in bare metal environment yesterday with EB03
@all,
Based on current status, I think we can claim sanity test is good enough and can push related patches to be merged now.
For images, as this is just for internal verification, they are private images pushed by me. So, you could not get them in your private registry and only get them from dock.io. That's not the problem!
Thanks!
Zhipeng
-----Original Message-----
From: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>
Sent: 2019年10月24日 3:41
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Last update, on Virtual environment we have the same issue: AIO-SX and AIO-DX success. Both standards failed.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Alonso, Juan Carlos
Sent: Wednesday, October 23, 2019 10:44 AM
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <Cristopher.J.Lemus.Contreras@intel.commailto:Cristopher.J.Lemus.Contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hello ZhipengS,
On Baremetal, deployment and Sanity on Simplex and Duplex success, but failed in both standards.
Your EB is based on master branch, right? Has been an issue last days and the fix has been submitted and merged: https://review.opendev.org/#/c/689438/
So, deployment on standards should work and if you generate a new EB should work too.
On virtual environment, all deployments for all configs success.
During the apply of stx-openstack, it failed due to the lack of some images, we are using a private registry and images could not found there.
Images were downloaded manually and stx-openstack re applied.
Do you have the updated list of the images you download? I CC to Cristopher who helped me with deployment and figure out this issue.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Sent: Wednesday, October 23, 2019 3:33 AM
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
I have tested EB 303 in our simplex bare metal environment Deployment and basic VM test pass!
Zhipeng
-----Original Message-----
From: Liu, ZhipengS
Sent: 2019年10月23日 10:27
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
I saw the fixed provided by Bob already in my EB 303.
Have you tested this one, any update?
Thanks!
Zhipeng
-----Original Message-----
From: Liu, ZhipengS
Sent: 2019年10月23日 9:30
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
I saw that sanity test of daily build also has the same issue as you have.
I will trigger a new EB with the fixed provide by Bob.
https://review.opendev.org/#/c/689643/ Ensure minimal replicas for platform-integ-apps managed apply
Thanks!
Zhipeng
-----Original Message-----
From: Liu, ZhipengS
Sent: 2019年10月22日 22:27
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Yes
But the failure place JC met is not the same as I saw.
I met one failure on applying openstack, but JC saw the failure on applying platform app, which should not be related to train upgrade.
For my failure case, it should disable nova schedule node probe as we did before, I add this fix in my new EB based on latest Daily build 1021.
@JC, I triggered EB-303, it may be available in 2 hours, you can check the status below http://dcp-dev.intel.com/jenkins/blue/organizations/jenkins/StarlingX%2Fstx-...
When it is ready, you can fetch EB from here and retest in BM setup.
http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/303/outputs/
Thanks!
Zhipeng
-----Original Message-----
From: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>
Sent: 2019年10月22日 22:07
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Thanks Zhipeng, I suppose RC means Root Cause. :-)
On 2019/10/22 9:37 PM, Liu, ZhipengS wrote:
Hi JC,
I should find the RC, will send you a new EB in 1 hour.
Thanks!
Zhipeng
*From:*Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
*Sent:* 2019年10月22日9:27
*To:* Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales,
Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
*Cc:* Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>;
Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi JC,
Thanks for your quick update!
Could you help share your logs?
Anyway, I will also try it in BM environment.
Thanks!
Zhipeng
--
*From:*Alonso, Juan Carlos
*Sent:* Tuesday, October 22, 2019 1:13 AM
*To:* Liu, ZhipengS; Cabrales, Ada
*Cc:* Xie, Cindy; Hu, Yong; Jones, Bruce E
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hello ZhipengS,
A Sanity execution was launched last Friday on Virtual and BareMetal
environment.
In virtual environment there was not issues during installation,
provisioning nor sanity.
In BareMetal environment it could be installed but failed during
provisioning in 4 configurations:
======================================================================
========
Provision :: Tests for provisioning and unlocking controllers,
computes and
======================================================================
========
Provisioning Standard Storage System :: Validates provisioning of ...
| FAIL |
Maximum limit of started keywords exceeded.
The issue was during system application-apply:
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+---------+-------------------------------+---------------+---------------+------------------------------------------+
| application | version | manifest name |
manifest file | status | progress
|
+---------------------+---------+-------------------------------+---------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest |
manifest.yaml | remove-failed | operation aborted, check logs for
detail |
+---------------------+---------+-------------------------------+---------------+---------------+------------------------------------------+
Regards.
Juan Carlos Alonso
*From:*Liu, ZhipengS <zhipengs.liu@intel.com
*Sent:* Sunday, October 20, 2019 8:35 PM
*To:* Cabrales, Ada <ada.cabrales@intel.com
*Cc:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.commailto:cindy.xie@intel.com%20%3cmailto:cindy.xie@intel.com>>;
Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.commailto:yong.hu@intel.com%20%3cmailto:yong.hu@intel.com>>; Jones, Bruce
E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>; Alonso,
Juan Carlos <juan.carlos.alonso@intel.com
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi Ada,
Do we have test result now?
Thanks!
Zhipeng
*From:*Cabrales, Ada <ada.cabrales@intel.com
*Sent:* 2019年10月14日22:20
*To:* Liu, ZhipengS <zhipengs.liu@intel.com
*Cc:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.commailto:cindy.xie@intel.com%20%3cmailto:cindy.xie@intel.com>>;
Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.commailto:yong.hu@intel.com%20%3cmailto:yong.hu@intel.com>>; Jones, Bruce
E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>; Alonso,
Juan Carlos <juan.carlos.alonso@intel.com
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Sure, let us try with this one.
Thanks!
Ada
*From:*Liu, ZhipengS <zhipengs.liu@intel.com
*Sent:* Sunday, October 13, 2019 6:44 AM
*To:* Cabrales, Ada <ada.cabrales@intel.com
*Cc:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.commailto:cindy.xie@intel.com%20%3cmailto:cindy.xie@intel.com>>;
Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.commailto:yong.hu@intel.com%20%3cmailto:yong.hu@intel.com>>; Jones, Bruce
E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi Ada,
Could you please use below new EB?
One horizon issue was fixed, found during my duplex test.
http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/294/outputs/
Thanks!
Zhipeng
*From:*Liu, ZhipengS <zhipengs.liu@intel.com
*Sent:* 2019年10月11日20:43
*To:* Cabrales, Ada <ada.cabrales@intel.com
mailto:ada.cabrales@intel.com>; Jones, Bruce E
<bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>
*Cc:* starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io
*Subject:* [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi Ada and Bruce,
So far, I have finished openstack train upgrade and
openstack-helm/helm-infra upgrade.
Basic deployment and VM creation/remove test pass in AIO setup!
@Ada mailto:ada.cabrales@intel.com, Could you help to arrange sanity
test with below EB, thanks!
http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/290/outputs/
Current status summarized as below.
1)Openstack upgrade
Finished 13 services upgrade, including nova, neutron, keystone,
cinder, glance, placement, aodh, ironic, panko, barbican, ceilometer, heat, horizon.
2)Openstack-helm/helm-infra upgrade
Upgrade openstack-helm to below version. 13 patches removed
commit 82c72367c85ca94270f702661c7b984899c1ae38
https://review.opendev.org/#/q/82c72367c85ca94270f702661c7b984899c1ae38 //Date:
Sat Sep 14 06:40:03 2019 +0000
Upgrade openstack-helm-infra to below version. 1 patch removed
commit c9d6676bf9a5aceb311dc31dadd07cba6a3d6392
https://review.opendev.org/#/q/c9d6676bf9a5aceb311dc31dadd07cba6a3d6392 //Date:
Mon Sep 16 17:15:12 2019 +0000
6 patches submitted for review, your comments are appreciated, thanks!
Thanks!
zhipeng
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy cindy.xie@intel.com Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
From: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Date: Tuesday, October 29, 2019 at 7:23 PM To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>, 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>, "Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Cristopher Lemus,
Yes, exactly! Do we have a test result with my latest EB now? Thank Boxiang for clarification! I have included related change in latest EB 308.
Thanks! Zhipeng
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: 2019年10月30日 4:16 To: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi All,
I think that we are facing the exact same issue that Boxiang mentions, it’s a failure for Standard external storage (2+2+2).
We are not able to launch instances on Standard with External Storage, here’s a sample output: http://paste.openstack.org/show/785641/, the error matches the bug: https://bugs.launchpad.net/starlingx/+bug/1837241, which was reverted on the review that Boxiang listed.
The rest of the configurations are not affected, this is specific to external storage.
Thanks ,
Cristopher Lemus
From: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Date: Tuesday, October 29, 2019 at 6:15 AM To: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>, "Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>, "Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>, "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
Yeah, now when we use the openstack train, we should set force_raw_images to True when we use rbd as images_types in nova.conf. Otherwise the nova-compute service will fail to start. The patch[0] was meant to do that before but it caused the sanity test failure and we didn't find the root cause.
[0] https://review.opendev.org/#/c/661512/
Thanks, Boxiang
From: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Date: 2019-10-29 10:54:13
To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>,"Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>,"Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>,"朱博祥" <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>
Cc: "Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>,"Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+baoxiang
Hi JC,
Thanks for your quick update!
I have found the root cause from your logs.
{"log":"2019-10-25 15:36:06.371 235059 ERROR oslo_service.service [-] Error starting thread.: InvalidConfiguration: '[DEFAULT]/force_raw_images = False' is not allowed with '[libvirt]/images_type = rbd'. Please check the two configs and if you really do want to use rbd as images_type, set force_raw_images to True.\n","stream":"stdout","time":"2019-10-25T15:36:06.372419534Z"}
This is related to an old LP
Last time, I revert the patch provided by Boxiang due to we were using staging nova version with some of Starlingx patches.
Now, we used upstream Nova, we need set force_raw_images = True again.
@Boxiang, any comment or additional change needed?
@JC, I will provide EB to you later for this issue.
Thanks!
Zhipeng
-----Original Message-----
From: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>
Sent: 2019年10月29日 5:33
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Standard External Storage failed again, same issue.
Logs available here: http://10.219.115.222/ISOS/ALL_NODES_20191028.125727.tar
See the log attached too.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Alonso, Juan Carlos
Sent: Monday, October 28, 2019 9:45 AM
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <Cristopher.J.Lemus.Contreras@intel.commailto:Cristopher.J.Lemus.Contreras@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hello,
Deployment and sanity execution was launched last Friday.
AIO-SX, AIO-DX and Standard Local Storage deployed successfully and sanity passed.
Cristopher found that for Standard External Storage there was a failure with nova-compute pods, this is causing system application-apply stuck at 54%.
Cristopher also put the logs from collect here: http://10.219.115.222/ISOS/ALL_NODES_20191025.153624.tar
Today only Standard External Storage was launched in BareMetal and virtual environments to assure this is a real issue. Will let you know.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Sent: Friday, October 25, 2019 12:22 AM
To: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
Please use below EB, which based on 1024 daily build
Thanks!
Zhipeng
-----Original Message-----
From: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>
Sent: 2019年10月25日 3:24
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hello ZhipengS,
Based on last StarlingX release meeting, we need a new EB with the fix merged since we have to execute sanity in both standard configurations.
Please can you generate a new EB to star testing?
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Sent: Wednesday, October 23, 2019 8:47 PM
To: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
Thanks for your update!
According to your test result, it seems just some known issues caused by other changes seen in your test.
Standards setup failure should be introduced in recent code merge.
Since you passed this case when you used my first EB that based on earlier daily build.
BTW, I also passed simplex test in bare metal environment yesterday with EB03
@all,
Based on current status, I think we can claim sanity test is good enough and can push related patches to be merged now.
For images, as this is just for internal verification, they are private images pushed by me. So, you could not get them in your private registry and only get them from dock.io. That's not the problem!
Thanks!
Zhipeng
-----Original Message-----
From: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>
Sent: 2019年10月24日 3:41
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Last update, on Virtual environment we have the same issue: AIO-SX and AIO-DX success. Both standards failed.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Alonso, Juan Carlos
Sent: Wednesday, October 23, 2019 10:44 AM
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Lemus Contreras, Cristopher J <Cristopher.J.Lemus.Contreras@intel.commailto:Cristopher.J.Lemus.Contreras@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hello ZhipengS,
On Baremetal, deployment and Sanity on Simplex and Duplex success, but failed in both standards.
Your EB is based on master branch, right? Has been an issue last days and the fix has been submitted and merged: https://review.opendev.org/#/c/689438/
So, deployment on standards should work and if you generate a new EB should work too.
On virtual environment, all deployments for all configs success.
During the apply of stx-openstack, it failed due to the lack of some images, we are using a private registry and images could not found there.
Images were downloaded manually and stx-openstack re applied.
Do you have the updated list of the images you download? I CC to Cristopher who helped me with deployment and figure out this issue.
Regards.
Juan Carlos Alonso
-----Original Message-----
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Sent: Wednesday, October 23, 2019 3:33 AM
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
I have tested EB 303 in our simplex bare metal environment Deployment and basic VM test pass!
Zhipeng
-----Original Message-----
From: Liu, ZhipengS
Sent: 2019年10月23日 10:27
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
I saw the fixed provided by Bob already in my EB 303.
Have you tested this one, any update?
Thanks!
Zhipeng
-----Original Message-----
From: Liu, ZhipengS
Sent: 2019年10月23日 9:30
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi JC,
I saw that sanity test of daily build also has the same issue as you have.
I will trigger a new EB with the fixed provide by Bob.
https://review.opendev.org/#/c/689643/ Ensure minimal replicas for platform-integ-apps managed apply
Thanks!
Zhipeng
-----Original Message-----
From: Liu, ZhipengS
Sent: 2019年10月22日 22:27
To: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Yes
But the failure place JC met is not the same as I saw.
I met one failure on applying openstack, but JC saw the failure on applying platform app, which should not be related to train upgrade.
For my failure case, it should disable nova schedule node probe as we did before, I add this fix in my new EB based on latest Daily build 1021.
@JC, I triggered EB-303, it may be available in 2 hours, you can check the status below http://dcp-dev.intel.com/jenkins/blue/organizations/jenkins/StarlingX%2Fstx-...
When it is ready, you can fetch EB from here and retest in BM setup.
http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/303/outputs/
Thanks!
Zhipeng
-----Original Message-----
From: Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>
Sent: 2019年10月22日 22:07
To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
Cc: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Thanks Zhipeng, I suppose RC means Root Cause. :-)
On 2019/10/22 9:37 PM, Liu, ZhipengS wrote:
Hi JC,
I should find the RC, will send you a new EB in 1 hour.
Thanks!
Zhipeng
*From:*Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
*Sent:* 2019年10月22日9:27
*To:* Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Cabrales,
Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>
*Cc:* Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>;
Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi JC,
Thanks for your quick update!
Could you help share your logs?
Anyway, I will also try it in BM environment.
Thanks!
Zhipeng
--
*From:*Alonso, Juan Carlos
*Sent:* Tuesday, October 22, 2019 1:13 AM
*To:* Liu, ZhipengS; Cabrales, Ada
*Cc:* Xie, Cindy; Hu, Yong; Jones, Bruce E
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hello ZhipengS,
A Sanity execution was launched last Friday on Virtual and BareMetal
environment.
In virtual environment there was not issues during installation,
provisioning nor sanity.
In BareMetal environment it could be installed but failed during
provisioning in 4 configurations:
======================================================================
========
Provision :: Tests for provisioning and unlocking controllers,
computes and
======================================================================
========
Provisioning Standard Storage System :: Validates provisioning of ...
| FAIL |
Maximum limit of started keywords exceeded.
The issue was during system application-apply:
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+---------+-------------------------------+---------------+---------------+------------------------------------------+
| application | version | manifest name |
manifest file | status | progress
|
+---------------------+---------+-------------------------------+---------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-8 | platform-integration-manifest |
manifest.yaml | remove-failed | operation aborted, check logs for
detail |
+---------------------+---------+-------------------------------+---------------+---------------+------------------------------------------+
Regards.
Juan Carlos Alonso
*From:*Liu, ZhipengS <zhipengs.liu@intel.com
*Sent:* Sunday, October 20, 2019 8:35 PM
*To:* Cabrales, Ada <ada.cabrales@intel.com
*Cc:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.commailto:cindy.xie@intel.com%20%3cmailto:cindy.xie@intel.com>>;
Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.commailto:yong.hu@intel.com%20%3cmailto:yong.hu@intel.com>>; Jones, Bruce
E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>; Alonso,
Juan Carlos <juan.carlos.alonso@intel.com
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi Ada,
Do we have test result now?
Thanks!
Zhipeng
*From:*Cabrales, Ada <ada.cabrales@intel.com
*Sent:* 2019年10月14日22:20
*To:* Liu, ZhipengS <zhipengs.liu@intel.com
*Cc:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.commailto:cindy.xie@intel.com%20%3cmailto:cindy.xie@intel.com>>;
Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.commailto:yong.hu@intel.com%20%3cmailto:yong.hu@intel.com>>; Jones, Bruce
E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>; Alonso,
Juan Carlos <juan.carlos.alonso@intel.com
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Sure, let us try with this one.
Thanks!
Ada
*From:*Liu, ZhipengS <zhipengs.liu@intel.com
*Sent:* Sunday, October 13, 2019 6:44 AM
*To:* Cabrales, Ada <ada.cabrales@intel.com
*Cc:* Xie, Cindy <cindy.xie@intel.com <mailto:cindy.xie@intel.commailto:cindy.xie@intel.com%20%3cmailto:cindy.xie@intel.com>>;
Hu, Yong <yong.hu@intel.com <mailto:yong.hu@intel.commailto:yong.hu@intel.com%20%3cmailto:yong.hu@intel.com>>; Jones, Bruce
E <bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>
*Subject:* RE: [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi Ada,
Could you please use below new EB?
One horizon issue was fixed, found during my duplex test.
http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/294/outputs/
Thanks!
Zhipeng
*From:*Liu, ZhipengS <zhipengs.liu@intel.com
*Sent:* 2019年10月11日20:43
*To:* Cabrales, Ada <ada.cabrales@intel.com
mailto:ada.cabrales@intel.com>; Jones, Bruce E
<bruce.e.jones@intel.com <mailto:bruce.e.jones@intel.commailto:bruce.e.jones@intel.com%20%3cmailto:bruce.e.jones@intel.com>>
*Cc:* starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io
*Subject:* [Starlingx-discuss] About EB verification for openstack
train upgrade
Hi Ada and Bruce,
So far, I have finished openstack train upgrade and
openstack-helm/helm-infra upgrade.
Basic deployment and VM creation/remove test pass in AIO setup!
@Ada mailto:ada.cabrales@intel.com, Could you help to arrange sanity
test with below EB, thanks!
http://dcp-dev.intel.com/pub/starlingx/stx-eng-build/290/outputs/
Current status summarized as below.
1)Openstack upgrade
Finished 13 services upgrade, including nova, neutron, keystone,
cinder, glance, placement, aodh, ironic, panko, barbican, ceilometer, heat, horizon.
2)Openstack-helm/helm-infra upgrade
Upgrade openstack-helm to below version. 13 patches removed
commit 82c72367c85ca94270f702661c7b984899c1ae38
https://review.opendev.org/#/q/82c72367c85ca94270f702661c7b984899c1ae38 //Date:
Sat Sep 14 06:40:03 2019 +0000
Upgrade openstack-helm-infra to below version. 1 patch removed
commit c9d6676bf9a5aceb311dc31dadd07cba6a3d6392
https://review.opendev.org/#/q/c9d6676bf9a5aceb311dc31dadd07cba6a3d6392 //Date:
Mon Sep 16 17:15:12 2019 +0000
6 patches submitted for review, your comments are appreciated, thanks!
Thanks!
zhipeng
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 9:35 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Zhipeng,
I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm
The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false
So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services?
-- Thanks, Boxiang
发件人:"Liu, ZhipengS" zhipengs.liu@intel.com 发送日期:2019-10-31 10:14:44 收件人:"Liu, ZhipengS" zhipengs.liu@intel.com,"Xie, Cindy" cindy.xie@intel.com,"Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com,"'朱博祥'" zhu.boxiang@99cloud.net,"'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io 抄送人:"Alonso, Juan Carlos" juan.carlos.alonso@intel.com,"Hu, Yong" yong.hu@intel.com,"Cabrales, Ada" ada.cabrales@intel.com,"Jones, Bruce E" bruce.e.jones@intel.com,"Martinez Monroy, Elio" elio.martinez.monroy@intel.com 主题:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 9:35 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() è if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy cindy.xie@intel.com Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Xie, Cindy cindy.xie@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Boxiang,
Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further.
Thanks! Zhipeng
From: 朱博祥 zhu.boxiang@99cloud.net Sent: 2019年10月31日 10:41 To: Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm
The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false
So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services?
-- Thanks, Boxiang
发件人:"Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> 发送日期:2019-10-31 10:14:44 收件人:"Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>,"Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>,"'朱博祥'" <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> 抄送人:"Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>,"Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>,"Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>,"Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> 主题:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
From the nova code, in nova/virt/images.py def fetch_to_raw(): -> convert_image(path_tmp, staged, fmt, 'raw') -> cmd = ('qemu-img', 'convert', '-t', cache_mode, '-O', out_format) So if the format of image is not raw, it will call qemu-img command to convert it to raw. But now we got
-- Thanks, Boxiang
发件人:"Liu, ZhipengS" zhipengs.liu@intel.com 发送日期:2019-10-31 11:11:37 收件人:"朱博祥" zhu.boxiang@99cloud.net,"Xie, Cindy" cindy.xie@intel.com,"Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com,"'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io 抄送人:"Alonso, Juan Carlos" juan.carlos.alonso@intel.com,"Hu, Yong" yong.hu@intel.com,"Cabrales, Ada" ada.cabrales@intel.com,"Jones, Bruce E" bruce.e.jones@intel.com,"Martinez Monroy, Elio" elio.martinez.monroy@intel.com 主题:RE: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang,
Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further.
Thanks! Zhipeng
From: 朱博祥 zhu.boxiang@99cloud.net Sent: 2019年10月31日 10:41 To: Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage).
In my nova.conf file, the section of libvirt is as followed:
[libvirt]
connection_uri = qemu+tcp://172.16.140.14/system
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
disk_cachemodes = network=writeback
hw_disk_discard = unmap
rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9
virt_type = kvm
The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file.
I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder.
It's my image's info:
(nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img
image: cirros-0.4.0-x86_64-disk.img
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX?
Are there other error logs in nova-compute or other nova-* services?
-- Thanks,
Boxiang
发件人:"Liu, ZhipengS" zhipengs.liu@intel.com 发送日期:2019-10-31 10:14:44 收件人:"Liu, ZhipengS" zhipengs.liu@intel.com,"Xie, Cindy" cindy.xie@intel.com,"Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com,"'朱博祥'" zhu.boxiang@99cloud.net,"'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io 抄送人:"Alonso, Juan Carlos" juan.carlos.alonso@intel.com,"Hu, Yong" yong.hu@intel.com,"Cabrales, Ada" ada.cabrales@intel.com,"Jones, Bruce E" bruce.e.jones@intel.com,"Martinez Monroy, Elio" elio.martinez.monroy@intel.com 主题:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 9:35 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() è if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy cindy.xie@intel.com Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Xie, Cindy cindy.xie@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Below is current test config. [libvirt] connection_uri = qemu+tcp://127.0.0.1/system cpu_mode = host-model disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = default live_migration_completion_timeout = 180 live_migration_inbound_addr = 192.168.206.150 live_migration_permit_auto_converge = true mem_stats_period_seconds = 0 rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337 rbd_user = cinder remove_unused_resized_minimum_age_seconds = 86400 virt_type = qemu
I also did below convert command manually, it seems OK. controller-0:~$ qemu-img convert -t none -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test1.img controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test2.img controller-0:~$ qemu-img info test1.img image: test1.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M controller-0:~$ qemu-img info test2.img image: test2.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M
Need further check why in the code after convert, the image format is still qcow2.
Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 11:12 To: 朱博祥 zhu.boxiang@99cloud.net; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further.
Thanks! Zhipeng
From: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Sent: 2019年10月31日 10:41 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm
The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false
So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services?
-- Thanks, Boxiang
发件人:"Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> 发送日期:2019-10-31 10:14:44 收件人:"Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>,"Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>,"'朱博祥'" <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> 抄送人:"Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>,"Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>,"Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>,"Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> 主题:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Zhipeng and All,
First of all, I found in your nova.conf, images_type = default So I think you just use qcow2 imagebackend and the root disk is on your nova-compute node not in vms pool of ceph storage. You can check that with command 'rbd ls -p vms'.
So that the creating vm process will not trigger my code in this patch https://review.opendev.org/#/c/640271
From the nova code, in nova/virt/images.py def fetch_to_raw(): -> convert_image(path_tmp, staged, fmt, 'raw') -> cmd = ('qemu-img', 'convert', '-t', cache_mode, '-O', out_format) So if the format of image is not raw, it will call qemu-img command to convert it to raw. But after converting and check the format of image again, it is still not raw, so it raise exception.
So we can check the nova-compute logs whether other error msgs exist or not. BTW, can you check the base image's format under '/var/lib/nova/instances/_base'? You can find the image in nova-compute container.
-- Thanks, Boxiang
From: "Liu, ZhipengS" zhipengs.liu@intel.com Date: 2019-10-31 11:38:36 To: "Liu, ZhipengS" zhipengs.liu@intel.com,"朱博祥" zhu.boxiang@99cloud.net,"Xie, Cindy" cindy.xie@intel.com,"Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com,"starlingx-discuss@lists.starlingx.io" starlingx-discuss@lists.starlingx.io Cc: "Alonso, Juan Carlos" juan.carlos.alonso@intel.com,"Hu, Yong" yong.hu@intel.com,"Cabrales, Ada" ada.cabrales@intel.com,"Jones, Bruce E" bruce.e.jones@intel.com,"Martinez Monroy, Elio" elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade Below is current test config. [libvirt] connection_uri = qemu+tcp://127.0.0.1/system cpu_mode = host-model disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = default live_migration_completion_timeout = 180 live_migration_inbound_addr = 192.168.206.150 live_migration_permit_auto_converge = true mem_stats_period_seconds = 0 rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337 rbd_user = cinder remove_unused_resized_minimum_age_seconds = 86400 virt_type = qemu
I also did below convert command manually, it seems OK. controller-0:~$ qemu-img convert -t none -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test1.img controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test2.img controller-0:~$ qemu-img info test1.img image: test1.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M controller-0:~$ qemu-img info test2.img image: test2.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M
Need further check why in the code after convert, the image format is still qcow2.
Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 11:12 To: 朱博祥 zhu.boxiang@99cloud.net; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further.
Thanks! Zhipeng
From: 朱博祥 zhu.boxiang@99cloud.net Sent: 2019年10月31日 10:41 To: Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage).
In my nova.conf file, the section of libvirt is as followed:
[libvirt]
connection_uri = qemu+tcp://172.16.140.14/system
images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
disk_cachemodes = network=writeback
hw_disk_discard = unmap
rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9
virt_type = kvm
The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file.
I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder.
It's my image's info:
(nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img
image: cirros-0.4.0-x86_64-disk.img
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX?
Are there other error logs in nova-compute or other nova-* services?
-- Thanks,
Boxiang
发件人:"Liu, ZhipengS" zhipengs.liu@intel.com 发送日期:2019-10-31 10:14:44 收件人:"Liu, ZhipengS" zhipengs.liu@intel.com,"Xie, Cindy" cindy.xie@intel.com,"Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com,"'朱博祥'" zhu.boxiang@99cloud.net,"'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io 抄送人:"Alonso, Juan Carlos" juan.carlos.alonso@intel.com,"Hu, Yong" yong.hu@intel.com,"Cabrales, Ada" ada.cabrales@intel.com,"Jones, Bruce E" bruce.e.jones@intel.com,"Martinez Monroy, Elio" elio.martinez.monroy@intel.com 主题:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 9:35 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() è if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy cindy.xie@intel.com Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS zhipengs.liu@intel.com; 朱博祥 zhu.boxiang@99cloud.net Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Xie, Cindy cindy.xie@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Boxiang,
Yes, I just want to check with you about images_type. Do we really need to change “images_type” to rbd from default, is it related to format convert? You patch code should already be hit, as we failed with EB307 with below error msg you added in your patch.
if CONF.libvirt.images_type in ('rbd',): if not CONF.force_raw_images: msg = _("'[DEFAULT]/force_raw_images = False' is not " "allowed with '[libvirt]/images_type = rbd'. " "Please check the two configs and if you really " "do want to use rbd as images_type, set " "force_raw_images to True.") raise exception.InvalidConfiguration(msg)
Below command is sent in my virtual multi-node environment, not in EB test environment. [root@compute-0 instances]# rbd ls -p vms rbd: error opening pool 'vms': (2) No such file or directory rbd: list: (2) No such file or directory [root@compute-0 instances]# ls compute_nodes locks lost+found
Unfortunately, we have no whole test log. @Lemus Contreras, Cristopher Jmailto:cristopher.j.lemus.contreras@intel.com Need your help to provide complete logs tested by EB308
Thanks! Zhipeng
From: 朱博祥 zhu.boxiang@99cloud.net Sent: 2019年10月31日 12:06 To: Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng and All,
First of all, I found in your nova.conf, images_type = default So I think you just use qcow2 imagebackend and the root disk is on your nova-compute node not in vms pool of ceph storage. You can check that with command 'rbd ls -p vms'.
So that the creating vm process will not trigger my code in this patch https://review.opendev.org/#/c/640271
From the nova code, in nova/virt/images.py def fetch_to_raw(): -> convert_image(path_tmp, staged, fmt, 'raw') -> cmd = ('qemu-img', 'convert', '-t', cache_mode, '-O', out_format) So if the format of image is not raw, it will call qemu-img command to convert it to raw. But after converting and check the format of image again, it is still not raw, so it raise exception.
So we can check the nova-compute logs whether other error msgs exist or not. BTW, can you check the base image's format under '/var/lib/nova/instances/_base'? You can find the image in nova-compute container.
-- Thanks, Boxiang
From: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Date: 2019-10-31 11:38:36 To: "Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>,"朱博祥" <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>,"Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>,"starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>,"Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>,"Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>,"Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Below is current test config. [libvirt] connection_uri = qemu+tcp://127.0.0.1/system cpu_mode = host-model disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = default live_migration_completion_timeout = 180 live_migration_inbound_addr = 192.168.206.150 live_migration_permit_auto_converge = true mem_stats_period_seconds = 0 rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337 rbd_user = cinder remove_unused_resized_minimum_age_seconds = 86400 virt_type = qemu
I also did below convert command manually, it seems OK. controller-0:~$ qemu-img convert -t none -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test1.img controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.img test2.img controller-0:~$ qemu-img info test1.img image: test1.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M controller-0:~$ qemu-img info test2.img image: test2.img file format: raw virtual size: 44M (46137344 bytes) disk size: 13M
Need further check why in the code after convert, the image format is still qcow2.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 11:12 To: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
Yes, my below patch can be abandoned, just need to remove “force_raw_images: false” in stx/openstack-armada-app/stx-openstack-helm/stx-openstack-helm/manifests/manifest.yamll file since the default value is true currently. https://review.opendev.org/#/c/691802/ I will modify my patch. However, it is not related to current issue since this flag is set to true in the test. The issue is just as error log mentioned below. “ Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2 ” We need check this point further.
Thanks! Zhipeng
From: 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Sent: 2019年10月31日 10:41 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re:RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
I have succeeded to create the vm in openstack upstream(master, code is two weeks ago) now. I deployed the environment with kolla-ansible(ceph as the backend storage). In my nova.conf file, the section of libvirt is as followed: [libvirt] connection_uri = qemu+tcp://172.16.140.14/system images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = nova disk_cachemodes = network=writeback hw_disk_discard = unmap rbd_secret_uuid = 5c79d567-e142-4ab4-87f6-0fdf53060ab9 virt_type = kvm
The default value of 'force_raw_images' in nova is True, so I did not set it as True again in my nova.conf file. I use the min image cirros with qcow2 format and created the vm without volume which is created by cinder. It's my image's info: (nova-compute)[root@zbx-allinone /]# qemu-img info cirros-0.4.0-x86_64-disk.img image: cirros-0.4.0-x86_64-disk.img file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false
So as you said, command qemu-img failed to convert the image from qcow2 to raw. BTW, is it a phenomenon that must recur now in STX? Are there other error logs in nova-compute or other nova-* services?
-- Thanks, Boxiang
发件人:"Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> 发送日期:2019-10-31 10:14:44 收件人:"Liu, ZhipengS" <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>,"Xie, Cindy" <cindy.xie@intel.commailto:cindy.xie@intel.com>,"Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>,"'朱博祥'" <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>,"'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> 抄送人:"Alonso, Juan Carlos" <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>,"Hu, Yong" <yong.hu@intel.commailto:yong.hu@intel.com>,"Cabrales, Ada" <ada.cabrales@intel.commailto:ada.cabrales@intel.com>,"Jones, Bruce E" <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>,"Martinez Monroy, Elio" <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> 主题:RE: [Starlingx-discuss] About EB verification for openstack train upgrade Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py. if remote_storage: default_config.update({'force_raw_images': True}) libvirt_config.update({'images_type': 'rbd', 'images_rbd_pool': rbd_pool, 'images_rbd_ceph_conf': rbd_ceph_conf}) else: libvirt_config.update({'images_type': 'default'})
Next step: The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally. (I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder, I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls 3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee [root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4 image: 3cb926303b665709318d2c099722b914025023b4 file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false [root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee file format: raw virtual size: 44M (46137344 bytes) disk size: 13M
Any comment?
Thanks! zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年10月31日 10:15 To: Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now. It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks! Zhipeng From: Liu, ZhipengS Sent: 2019年10月31日 17:31 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py. if remote_storage: default_config.update({'force_raw_images': True}) libvirt_config.update({'images_type': 'rbd', 'images_rbd_pool': rbd_pool, 'images_rbd_ceph_conf': rbd_ceph_conf}) else: libvirt_config.update({'images_type': 'default'})
Next step: The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally. (I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder, I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls 3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee [root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4 image: 3cb926303b665709318d2c099722b914025023b4 file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false [root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee file format: raw virtual size: 44M (46137344 bytes) disk size: 13M
Any comment?
Thanks! zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 10:15 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hello,
Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.
AIO – Simplex
Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
AIO – Duplex
Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
Standard - Local Storage (2+2)
Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
Standard - Dedicated Storage (2+2+2)
Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Just the cirros image was replaced. Centos file is still a .qcow2.
Please, let me know if you need further details.
Thanks & Regards,
Cristopher Lemus
From: "Liu, ZhipengS" zhipengs.liu@intel.com Date: Friday, November 1, 2019 at 12:02 AM To: "Xie, Cindy" cindy.xie@intel.com, "Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com, '朱博祥' zhu.boxiang@99cloud.net, "'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io Cc: "Alonso, Juan Carlos" juan.carlos.alonso@intel.com, "Hu, Yong" yong.hu@intel.com, "Cabrales, Ada" ada.cabrales@intel.com, "Jones, Bruce E" bruce.e.jones@intel.com, "Martinez Monroy, Elio" elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now. It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks! Zhipeng From: Liu, ZhipengS Sent: 2019年10月31日 17:31 To: Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py. if remote_storage: default_config.update({'force_raw_images': True}) libvirt_config.update({'images_type': 'rbd', 'images_rbd_pool': rbd_pool, 'images_rbd_ceph_conf': rbd_ceph_conf}) else: libvirt_config.update({'images_type': 'default'})
Next step: The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally. (I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder, I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls 3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee [root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4 image: 3cb926303b665709318d2c099722b914025023b4 file format: qcow2 virtual size: 44M (46137344 bytes) disk size: 12M cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false [root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee file format: raw virtual size: 44M (46137344 bytes) disk size: 13M
Any comment?
Thanks! zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 10:15 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(), After force convert image from qcow2 to raw, the format is still not raw, then throw below error. It seems qemu-image convert image failed.
Zhipeng From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: 2019年10月31日 9:35 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
From: Liu, ZhipengS Sent: 2019年10月30日 23:24 To: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train. Error log is below: build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP: https://bugs.launchpad.net/nova/+bug/1816686 In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now) You added below commit message in the patch.
“When we import image into rbd, check the format of cache images. If the format is not raw, remove it first and fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py def fetch_to_raw() ==> if data.file_format != "raw": raise exception.ImageUnacceptable(image_id=image_href, reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later. Any comment from you? I will check nova code further tomorrow and sync with you. We may need a patch for nova☹
Thanks! Zhipeng From: Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com> Sent: 2019年10月30日 22:49 To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: RE: [Starlingx-discuss] About EB verification for openstack train upgrade
+ mailing list in case anybody can help on debug the issue.
From: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.commailto:cristopher.j.lemus.contreras@intel.com> Sent: Wednesday, October 30, 2019 7:45 PM To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.netmailto:zhu.boxiang@99cloud.net> Cc: Alonso, Juan Carlos <juan.carlos.alonso@intel.commailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.commailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.commailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.commailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.commailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.commailto:elio.martinez.monroy@intel.com> Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX?
In addition, @Zhipeng, we might get this image to dig out the root-cause further.
Regards, Yong
On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
Hello,
Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.
*AIO – Simplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
*AIO – Duplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
*Standard - Local Storage (2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
*Standard - Dedicated Storage (2+2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Just the cirros image was replaced. Centos file is still a .qcow2.
Please, let me know if you need further details.
Thanks & Regards,
Cristopher Lemus
*From: *"Liu, ZhipengS" zhipengs.liu@intel.com *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" cindy.xie@intel.com, "Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com, '朱博祥' zhu.boxiang@99cloud.net, "'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io *Cc: *"Alonso, Juan Carlos" juan.carlos.alonso@intel.com, "Hu, Yong" yong.hu@intel.com, "Cabrales, Ada" ada.cabrales@intel.com, "Jones, Bruce E" bruce.e.jones@intel.com, "Martinez Monroy, Elio" elio.martinez.monroy@intel.com *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.
It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks!
Zhipeng
*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py.
if remote_storage:
default_config.update({'force_raw_images': True})
libvirt_config.update({'images_type': 'rbd',
'images_rbd_pool': rbd_pool,
'images_rbd_ceph_conf': rbd_ceph_conf})
else:
libvirt_config.update({'images_type': 'default'})
Next step:
The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.
(I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder,
I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls
3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4
image: 3cb926303b665709318d2c099722b914025023b4
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M
Any comment?
Thanks!
zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(),
After force convert image from qcow2 to raw, the format is still not raw, then throw below error.
It seems qemu-image convert image failed.
Zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.
“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py
def fetch_to_raw() è
if data.file_format != "raw":
raise exception.ImageUnacceptable(image_id=image_href,
reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹
Thanks!
Zhipeng
*From:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
- mailing list in case anybody can help on debug the issue.
*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Christopher is on vacations today. I Will provide the image ASAP.
-----Original Message----- From: Hu, Yong yong.hu@intel.com Sent: Monday, November 4, 2019 7:56 AM To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX?
In addition, @Zhipeng, we might get this image to dig out the root-cause further.
Regards, Yong
On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
Hello,
Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.
*AIO – Simplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
*AIO – Duplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
*Standard - Local Storage (2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
*Standard - Dedicated Storage (2+2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Just the cirros image was replaced. Centos file is still a .qcow2.
Please, let me know if you need further details.
Thanks & Regards,
Cristopher Lemus
*From: *"Liu, ZhipengS" zhipengs.liu@intel.com *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" cindy.xie@intel.com, "Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com, '朱博祥' zhu.boxiang@99cloud.net, "'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io *Cc: *"Alonso, Juan Carlos" juan.carlos.alonso@intel.com, "Hu, Yong" yong.hu@intel.com, "Cabrales, Ada" ada.cabrales@intel.com, "Jones, Bruce E" bruce.e.jones@intel.com, "Martinez Monroy, Elio" elio.martinez.monroy@intel.com *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.
It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks!
Zhipeng
*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py.
if remote_storage:
default_config.update({'force_raw_images': True})
libvirt_config.update({'images_type': 'rbd',
'images_rbd_pool': rbd_pool,
'images_rbd_ceph_conf': rbd_ceph_conf})
else:
libvirt_config.update({'images_type': 'default'})
Next step:
The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.
(I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder,
I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls
3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4
image: 3cb926303b665709318d2c099722b914025023b4
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M
Any comment?
Thanks!
zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(),
After force convert image from qcow2 to raw, the format is still not raw, then throw below error.
It seems qemu-image convert image failed.
Zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.
“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py
def fetch_to_raw() è
if data.file_format != "raw":
raise exception.ImageUnacceptable(image_id=image_href,
reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹
Thanks!
Zhipeng
*From:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
- mailing list in case anybody can help on debug the issue.
*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Yong and all,
The root cause should be below. Download image from https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img It is already qcow2, if we use below command to convert to qcow2 again, it will be added with an additional qcow2 header. This is the problem!!! controller-0:~$ qemu-img convert -f raw -O qcow2 cirros-0.4.0-x86_64-disk.img cirros-0.4.0-x86_64-disk.qcow2
So, if we run test case in nova train release. It will just like below.
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.qcow2 test controller-0:~$ qemu-img info test image: test file format: qcow2 //it is still qcow2, just removed additional qcow header. virtual size: 44M (46137344 bytes)
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 test test22 controller-0:~$ qemu-img info test22 image: test22 file format: raw //after second convert, it becomes raw!!! virtual size: 44M (46137344 bytes)
That's why we use the cirros-0.4.0-x86_64-disk.img downloaded from official website can pass sanity test.
Thanks! Zhipeng
-----Original Message----- From: Hu, Yong yong.hu@intel.com Sent: 2019年11月4日 21:56 To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX?
In addition, @Zhipeng, we might get this image to dig out the root-cause further.
Regards, Yong
On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
Hello,
Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.
*AIO – Simplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
*AIO – Duplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
*Standard - Local Storage (2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
*Standard - Dedicated Storage (2+2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Just the cirros image was replaced. Centos file is still a .qcow2.
Please, let me know if you need further details.
Thanks & Regards,
Cristopher Lemus
*From: *"Liu, ZhipengS" zhipengs.liu@intel.com *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" cindy.xie@intel.com, "Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com, '朱博祥' zhu.boxiang@99cloud.net, "'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io *Cc: *"Alonso, Juan Carlos" juan.carlos.alonso@intel.com, "Hu, Yong" yong.hu@intel.com, "Cabrales, Ada" ada.cabrales@intel.com, "Jones, Bruce E" bruce.e.jones@intel.com, "Martinez Monroy, Elio" elio.martinez.monroy@intel.com *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.
It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks!
Zhipeng
*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py.
if remote_storage:
default_config.update({'force_raw_images': True})
libvirt_config.update({'images_type': 'rbd',
'images_rbd_pool': rbd_pool,
'images_rbd_ceph_conf': rbd_ceph_conf})
else:
libvirt_config.update({'images_type': 'default'})
Next step:
The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.
(I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder,
I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls
3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4
image: 3cb926303b665709318d2c099722b914025023b4
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M
Any comment?
Thanks!
zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(),
After force convert image from qcow2 to raw, the format is still not raw, then throw below error.
It seems qemu-image convert image failed.
Zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.
“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py
def fetch_to_raw() è
if data.file_format != "raw":
raise exception.ImageUnacceptable(image_id=image_href,
reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹
Thanks!
Zhipeng
*From:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
- mailing list in case anybody can help on debug the issue.
*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
Hi Elio and JC,
I found that in your test machine, there are both good and bad images. tester tester 13107200 Nov 4 03:58 ./Downloads/cirros-0.4.0-x86_64-disk.qcow2 is bad tester tester 12716032 May 20 08:24 ./elio/cirros-0.4.0-x86_64-disk.qcow2 is good
So, I propose you to use image downloaded from official website directly, https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img This is already qcow2 format.
Thanks! Zhipeng -----Original Message----- From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2019年11月5日 10:39 To: Hu, Yong yong.hu@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Xie, Cindy cindy.xie@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Yong and all,
The root cause should be below. Download image from https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img It is already qcow2, if we use below command to convert to qcow2 again, it will be added with an additional qcow2 header. This is the problem!!! controller-0:~$ qemu-img convert -f raw -O qcow2 cirros-0.4.0-x86_64-disk.img cirros-0.4.0-x86_64-disk.qcow2
So, if we run test case in nova train release. It will just like below.
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 cirros-0.4.0-x86_64-disk.qcow2 test controller-0:~$ qemu-img info test image: test file format: qcow2 //it is still qcow2, just removed additional qcow header. virtual size: 44M (46137344 bytes)
controller-0:~$ qemu-img convert -t writeback -O raw -f qcow2 test test22 controller-0:~$ qemu-img info test22 image: test22 file format: raw //after second convert, it becomes raw!!! virtual size: 44M (46137344 bytes)
That's why we use the cirros-0.4.0-x86_64-disk.img downloaded from official website can pass sanity test.
Thanks! Zhipeng
-----Original Message----- From: Hu, Yong yong.hu@intel.com Sent: 2019年11月4日 21:56 To: Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; Liu, ZhipengS zhipengs.liu@intel.com; Xie, Cindy cindy.xie@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Cc: Alonso, Juan Carlos juan.carlos.alonso@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com Subject: Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Christopher, A quick question, were we using this image "cirros-0.4.0-x86_64-disk.qcow2" for the previous versions of StarlingX?
In addition, @Zhipeng, we might get this image to dig out the root-cause further.
Regards, Yong
On 2019/11/2 4:53 AM, Lemus Contreras, Cristopher J wrote:
Hello,
Using the EB that Zhipeng provided, and replacing “cirros-0.4.0-x86_64-disk.qcow2” with “cirros-0.4.0-x86_64-disk.img”, Sanity status is Green.
*AIO – Simplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
*AIO – Duplex*
**
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 64 TCs PASS ]
*Standard - Local Storage (2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 08 TCs [PASS]
TOTAL: [ 65 TCs PASS ]
*Standard - Dedicated Storage (2+2+2)*
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 09 TCs [PASS]
TOTAL: [ 66 TCs PASS ]
===========================================
Just the cirros image was replaced. Centos file is still a .qcow2.
Please, let me know if you need further details.
Thanks & Regards,
Cristopher Lemus
*From: *"Liu, ZhipengS" zhipengs.liu@intel.com *Date: *Friday, November 1, 2019 at 12:02 AM *To: *"Xie, Cindy" cindy.xie@intel.com, "Lemus Contreras, Cristopher J" cristopher.j.lemus.contreras@intel.com, '朱博祥' zhu.boxiang@99cloud.net, "'starlingx-discuss@lists.starlingx.io'" starlingx-discuss@lists.starlingx.io *Cc: *"Alonso, Juan Carlos" juan.carlos.alonso@intel.com, "Hu, Yong" yong.hu@intel.com, "Cabrales, Ada" ada.cabrales@intel.com, "Jones, Bruce E" bruce.e.jones@intel.com, "Martinez Monroy, Elio" elio.martinez.monroy@intel.com *Subject: *RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi all,
After using new image file (cirros-0.4.0-x86_64-disk.img), we can launch instance on standard external Storage (2+2+2) now.
It seems the old image file (cirros-0.4.0-x86_64-disk.qcow2) used by Cristopher Lemus causes qemu-img detecting wrong format after format convert from qcow2to raw in Nova code. We are not sure how/where we got this image file.
Thanks!
Zhipeng
*From:* Liu, ZhipengS *Sent:* 2019年10月31日 17:31 *To:* Xie, Cindy cindy.xie@intel.com; Lemus Contreras, Cristopher J cristopher.j.lemus.contreras@intel.com; '朱博祥' zhu.boxiang@99cloud.net; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos juan.carlos.alonso@intel.com; Hu, Yong yong.hu@intel.com; Cabrales, Ada ada.cabrales@intel.com; Jones, Bruce E bruce.e.jones@intel.com; Martinez Monroy, Elio elio.martinez.monroy@intel.com *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
For remote storage configuration, we have overridden below configurations in helm/nova.py.
if remote_storage:
default_config.update({'force_raw_images': True})
libvirt_config.update({'images_type': 'rbd',
'images_rbd_pool': rbd_pool,
'images_rbd_ceph_conf': rbd_ceph_conf})
else:
libvirt_config.update({'images_type': 'default'})
Next step:
The only issue we need to figure out is why image format is still qcow2 after image format convert by qemu-img in nova code.
I tried to reproduce it in my virtualized multi-node environment, but no issue found, VM created normally.
(I copied yellow code block to else branch)
In /var/lib/nova/instances/_base/ folder,
I can see one qcow2 image and one raw image which should be converted by nova.
[root@compute-1 _base]# ls
3cb926303b665709318d2c099722b914025023b4 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
[root@compute-1 _base]# qemu-img info 3cb926303b665709318d2c099722b914025023b4
image: 3cb926303b665709318d2c099722b914025023b4
file format: qcow2
virtual size: 44M (46137344 bytes)
disk size: 12M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
[root@compute-1 _base]# qemu-img info 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
image: 6edc9bb94ba7363ae9d40d43c9d0eeaffdc0f2ee
file format: raw
virtual size: 44M (46137344 bytes)
disk size: 13M
Any comment?
Thanks!
zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 10:15 *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang and all,
In fetch_to_raw(),
After force convert image from qcow2 to raw, the format is still not raw, then throw below error.
It seems qemu-image convert image failed.
Zhipeng
*From:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com> *Sent:* 2019年10月31日 9:35 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; '朱博祥' <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
*From:* Liu, ZhipengS *Sent:* 2019年10月30日 23:24 *To:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Boxiang,
It should be an issue in nova train.
Error log is below:
build of instance 363dddce-7af6-4701-8cea-52630b91ebcf aborted: Image 70f74018-24f6-4d7d-8f99-2194a4449cfd is unacceptable: Converted to raw, but format is now qcow2'
You tried to fix this issue before in nova LP:
https://bugs.launchpad.net/nova/+bug/1816686
In your patch: https://review.opendev.org/#/c/640271 (This patch is already in train release now)
You added below commit message in the patch.
“When we import image into rbd, check the format of cache
images. If the format is not raw, remove it first and
fetch it again. It will be raw format now.”
However, it seems not work! Below code will throw error when image format is not raw and block the VM creation.
In nova/virt/images.py
def fetch_to_raw() è
if data.file_format != "raw":
raise exception.ImageUnacceptable(image_id=image_href,
reason=_("Converted to raw, but format is now %s")
Not sure if your patch fixed this issue or is there any regression introduced in nova code later.
Any comment from you?
I will check nova code further tomorrow and sync with you.
We may need a patch for nova☹
Thanks!
Zhipeng
*From:* Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com> *Sent:* 2019年10月30日 22:49 *To:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net>; starlingx-discuss@lists.starlingx.io mailto:starlingx-discuss@lists.starlingx.io *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* RE: [Starlingx-discuss] About EB verification for openstack train upgrade
- mailing list in case anybody can help on debug the issue.
*From:* Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com mailto:cristopher.j.lemus.contreras@intel.com> *Sent:* Wednesday, October 30, 2019 7:45 PM *To:* Liu, ZhipengS <zhipengs.liu@intel.com mailto:zhipengs.liu@intel.com>; 朱博祥 <zhu.boxiang@99cloud.net mailto:zhu.boxiang@99cloud.net> *Cc:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com mailto:juan.carlos.alonso@intel.com>; Hu, Yong <yong.hu@intel.com mailto:yong.hu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com mailto:ada.cabrales@intel.com>; Xie, Cindy <cindy.xie@intel.com mailto:cindy.xie@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com mailto:bruce.e.jones@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com mailto:elio.martinez.monroy@intel.com> *Subject:* Re: [Starlingx-discuss] About EB verification for openstack train upgrade
Hi Zhipeng,
The results are OK for Simplex, Duplex and Standard Local Storage, all Test Cases PASSED.
For Standard External Storage (2+2+2) there is a high quantity of Test Cases failing because we cannot launch instances. Here’s a sample output: http://paste.openstack.org/show/785641/ with the error that matches what is described on the launchpad.
Sanity summary for this specific configuration: http://paste.openstack.org/show/785655/
Please, let us know if you need additional details.
Thanks & Regards,
Cristopher Lemus
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (6)
-
Lemus Contreras, Cristopher J
-
Liu, ZhipengS
-
Martinez Monroy, Elio
-
Xie, Cindy
-
Yong Hu
-
朱博祥