[Starlingx-discuss] [OpenStack Ussuri Upgrade Task] Call for patch review!!
Scott Little
scott.little at windriver.com
Wed Jun 10 20:28:07 UTC 2020
Six of the nine updates are in a state of merge conflict.
Please resolve the conflicts so that I can make progress wit a CENGN build.
Scott
On 2020-06-10 9:20 a.m., Scott Little wrote:
> CENGN cycles aren't a problem. People resources is a challenge.
>
> So the ask is for a manual build, on CENGN, adding in the nine patches
> listed by https://review.opendev.org/#/q/topic:for_ussuri+(status:open).
>
> .. and the addition of two repos to the build-stx-base.sh step
>
> build-stx-base.sh
> --repo local-stx-build,... \
> --repo stx-distro,... \
> --repo ussuri-ceph,http://download.ceph.com/rpm-mimic/el7/x86_64/ \
> --repo ussuri-wsgi,http://mirror.centos.org/centos/7/sclo/x86_64/rh/
>
>
> Is that correct?
>
> Scott
>
>
> On 2020-06-09 9:04 a.m., Saul Wold wrote:
>>
>> Frank, Scott, Davelet:
>>
>> Are there cycles available on Cengn (and people resources) to do a
>> Cengn build with the Ussuri patch set applied? I know this is
>> different than a branch build. I think we have done this kind of
>> thing in the past.
>>
>> This might help to make sure we don't have any more Cengn build
>> issues and could give the Test team a sanity spin with a Ussuri/Cengn
>> build.
>>
>> Note there is a comment for Scott/Davelet at the bottom of Zhipeng's
>> email.
>>
>> Thanks
>> Sau!
>>
>>
>> On 6/9/20 1:39 AM, Liu, ZhipengS wrote:
>>> Hi all,
>>>
>>> So far, all block issues and concerns have been addressed.
>>> Since we have passed all sanity test, and Ussuri OpenStack has been
>>> officially released last month,
>>> there should be no more reason to block these patches merge.
>>>
>>> Next step:
>>> Let's push to get ussuri upgrade/openstack-helm rebasing patches
>>> merged. We need great help from core guys!
>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open)
>>>
>>> # Below 6 patches are for OpenStack-helm/infra rebase. (we set first
>>> patch with workflow-1 and add depends-on for other patches as we
>>> need to merge them together.)
>>> Upgrade openstack-helm-infra zhipeng liu
>>> starlingx/openstack-armada-app workflow-1
>>> Add mariadb database config override to support ipv6 zhipeng liu
>>> starlingx/openstack-armada-app
>>> Fix render error in cinder during openstack-helm rebase zhipeng
>>> liu starlingx/openstack-armada-app
>>> Update download list for openstack-helm upgrade zhipeng liu
>>> starlingx/openstack-armada-app
>>> Update manifest.yaml file for openstack-helm upgrade.
>>> zhipeng liu starlingx/openstack-armada-app
>>> Upgrade openstack-helm zhipeng liu starlingx/openstack-armada-app
>>>
>>> # Below 3 patches is for OpenStack upgrade.
>>> Update manifest.yaml file for ussuri openstack
>>> YU CHENGDE starlingx/openstack-armada-app
>>> Modify build-tools and stable-wheels for Ussuri upgrading YU
>>> CHENGDE starlingx/root
>>> Upgrade openstack docker images for stable/ussuri YU
>>> CHENGDE starlingx/upstream
>>>
>>>
>>> After removing required python3 dependent packages from local, we
>>> can build out base image and OpenStack service images successfully
>>> with below command.
>>> ===============================================================================
>>>
>>> @Scott, please help to update cengn build script with below 2
>>> additional repos and help to trigger image build
>>> build-stx-base.sh
>>> --repo local-stx-build,... \
>>> --repo stx-distro,... \
>>> --repo ussuri-ceph,http://download.ceph.com/rpm-mimic/el7/x86_64/ \
>>> --repo ussuri-wsgi,http://mirror.centos.org/centos/7/sclo/x86_64/rh/
>>>
>>> Thanks a lot!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS
>>> Sent: 2020年6月8日 16:54
>>> To: 'Miller, Frank' <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Friesen, Chris
>>> <Chris.Friesen at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> It is not easy to figure out whether/how/when OpenStack-helm-info
>>> upstream introduce this issue and then fix it.
>>> I also could not find any fix in LP[1], which just mentioned that
>>> this intermittent issue not hit us after some changes in related field.
>>>
>>> Anyhow, below 2 patches should fix potential bug and I could not see
>>> the same error log again in our ussuri upgrade EB.
>>> https://review.opendev.org/#/c/704034/ Prevent splitbrain during
>>> full Galera restart https://review.opendev.org/#/c/708071/ mariadb:
>>> avoid state management thread death
>>>
>>> Since we have passed fully test, we'd better push to merge ussuri
>>> upgrade/openstack-helm rebasing patches soon.
>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open)
>>>
>>> [1] https://bugs.launchpad.net/starlingx/+bug/1816842/
>>>
>>> Thanks!
>>> Zhipeng
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年6月5日 22:32
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io; Friesen, Chris
>>> <Chris.Friesen at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Zhipeng:
>>>
>>> This looks promising. Your theory is that the 2
>>> openstack-helm-infra patches will fix the mariadb recovery issues.
>>> These 2 patches were merged in the openstack-helm-infra project in
>>> January and February of 2020. What would be good to know is what
>>> broke mariadb recovery between April of 2019 when Chris Friesen
>>> finished up his story [1] and our current loads today. The most
>>> likely explanation is the upversion of Train or the upversion to
>>> openstack-helm-infra done in November 2019 introduced the mariadb
>>> recovery issues. And then the openstack-helm folks found and fixed
>>> the issue earlier in 2020.
>>>
>>> If we had more time the preferred approach would be to merge just
>>> the openstack-helm-infra changes first to prove they address mariadb
>>> recovery and then in a separate commit merge Ussuri. But since you
>>> have validated that mariadb recovers with your Ussuri branch and
>>> this branch has these openstack-helm commits, I support letting
>>> Ussuri merge into stx.4.0.
>>>
>>> Frank
>>> [1] https://storyboard.openstack.org/#!/story/2004712
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Friday, June 05, 2020 2:36 AM
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Friesen, Chris
>>> <Chris.Friesen at windriver.com>
>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> As for OpenStack not recovering after both controllers are reset [1]
>>> I could not reproduce this issue with my Ussuri upgrade EB.
>>> My test step is:
>>> 1) ssh to standby controller and sudo reboot -f for it.
>>> 2) sudo reboot -f for activated controller All pods can resume after
>>> a while.
>>>
>>> However, I could reproduce this issue with DB 20200516T080009Z.
>>> From error logs, it is an old issue analyzed by Chris Friesen in
>>> [2] early last year.
>>>
>>> In ussuri upgrade EB, we rebased openstack-helm-infra/mariadb.
>>> It includes below 2 patches which fixed this stability issue.
>>> https://review.opendev.org/#/c/704034/ Prevent splitbrain during
>>> full Galera restart https://review.opendev.org/#/c/708071/ mariadb:
>>> avoid state management thread death
>>>
>>> [1] https://bugs.launchpad.net/starlingx/+bug/1881899
>>> [2] https://bugs.launchpad.net/starlingx/+bug/1816842/comments/3
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年6月3日 22:35
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Zhipeng:
>>>
>>> This is not a new requirement. Users expect the software to recover
>>> when resets occur.
>>>
>>> As I had mentioned at the PTG yesterday I know personally that this
>>> test passed in stx3.0 before the upversion to train. Someone else
>>> who performs testing can look to determine when this test was done
>>> as part of feature testing after train was delivered as it should
>>> have been tested as part of stx.3.0 as well. I do not know when
>>> this started to break. One topic we will discuss at the PTG
>>> tomorrow will be how to improve our test coverage and automation so
>>> this type of issue can be found immediately as new code is being
>>> delivered.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Wednesday, June 03, 2020 10:28 AM
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Frank,
>>>
>>> Have we pass this case before? Is it a new requirement?
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年6月3日 22:12
>>> To: Miller, Frank <Frank.Miller at windriver.com>; Liu, ZhipengS
>>> <zhipengs.liu at intel.com>; starlingx-discuss at lists.starlingx.io;
>>> Church, Robert <Robert.Church at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Yong/Zhipeng - the LP for openstack not recovering after both
>>> controllers are reset is
>>> https://bugs.launchpad.net/starlingx/+bug/1881899
>>>
>>> Ovidiu is investigating and will provide any updates from his
>>> investigation. Please continue to keep us informed of your
>>> investigation.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: Tuesday, June 02, 2020 10:38 PM
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> We used a build from May 28.
>>>
>>> As for the decoupling issue these are actively being worked. If you
>>> run the system helm-override-show command when the stx-openstack app
>>> is applied you won’t see the CLI command fail. It only fails when
>>> you try a helm-override-show when the app is in uploaded state. In
>>> any case this will be fixed shortly.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Tuesday, June 02, 2020 10:04 PM
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> Thanks for your quick update!
>>> Which build are you using to test this case?
>>> Since decoupling commits introduced several regressions (at least
>>> 2), not propose to do this kind of stability test with latest build.
>>> BTW, do we have plan to revert them considering this stability
>>> risk? Our Ussuri upgrade patches is waiting for it☹
>>>
>>> Furthermore, we have not seen this test case that force reboot both
>>> controllers at the same time. Is it a new requirement? If not , have
>>> we pass this case before, which build?
>>> I'd like to help on it with the pass build for comparative analysis.
>>> From my point , mariadb might not work if we reboot both controllers.
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年6月3日 8:55
>>> To: Miller, Frank <Frank.Miller at windriver.com>; Liu, ZhipengS
>>> <zhipengs.liu at intel.com>; starlingx-discuss at lists.starlingx.io;
>>> Church, Robert <Robert.Church at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Zhipeng:
>>>
>>> An update on our testing and analysis today. We are able to
>>> reproduce the issue with OpenStack not recovering when we trigger a
>>> reboot of both AIO controllers at the same time. This results in
>>> MariaDB and multiple other OpenStack pods in CrashLoopBackoff and
>>> openstack commands not working indefinitely after the controllers
>>> recover. We'll create a launchpad tomorrow to track this issue.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: Tuesday, June 02, 2020 12:25 PM
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Thanks Zhipeng for the analysis. What is challenging here is the
>>> multitude of issues.
>>>
>>> In our debug of openstack the past few days we are seeing the app
>>> fail completely. After investigation this issue is a Day 1
>>> containerd issue. This is tracked in LP:
>>> https://bugs.launchpad.net/starlingx/+bug/1881353
>>>
>>> The issue you are seeing on a swact is a new and very recent issue
>>> tied to the decoupling commits that were merged late last week. Bob
>>> is investigating and I expect he'll have a fix soon for that.
>>>
>>> But the issues we are most concerned with are when we see mariadb
>>> crashing and not able to recover or with openstack services not
>>> working for longer periods of time. We're attempting to isolate the
>>> sequence of events that trigger this.
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Tuesday, June 02, 2020 11:47 AM
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> For LP https://bugs.launchpad.net/starlingx/+bug/1881454
>>> Unable to unlock controller after swact and lock w/
>>> openstack applied I also tested with daily build 20200516T080009Z.
>>> However, it could not be reproduced.
>>> We should fix this regression ASAP!
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: 2020年6月2日 16:48
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Church, Robert
>>> <Robert.Church at windriver.com>
>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank and all,
>>>
>>> Update for issue 2.
>>> I raised a new LP to track it.
>>> https://bugs.launchpad.net/starlingx/+bug/1881722
>>> Below is the time statistics. It seems reasonable. No obvious issue
>>> found.
>>> 1) 3~4min for host restart and get ready.
>>> 2) 2~3min for mariadb terminating, initialization, get ready. (then
>>> configmap sync is ready)
>>> 3) 2min for ovs-db ready (reduce probe live/ready timer can improve
>>> a little, as it can retry quickly to connect ovs-vsctl:
>>> unix:/var/run/openvswitch/db.sock)
>>> 4) 1min for other pods ready, like neutron-ovs-agent which depends
>>> on ovs-db. ) Any comment?
>>>
>>> For LP https://bugs.launchpad.net/starlingx/+bug/1881454
>>> Unable to unlock controller after swact and lock w/
>>> openstack applied
>>> And https://bugs.launchpad.net/starlingx/+bug/1881711
>>> system helm-override-show stx-openstack mariadb
>>> openstack crash It seems related to openstack plugin decouple
>>> related patches. Should be a regression.
>>> Please see our update in this 2 LPs for detail info. @Bob, could
>>> you pls help further check it and your patches, thanks!
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS
>>> Sent: 2020年6月1日 16:20
>>> To: 'Miller, Frank' <Frank.Miller at windriver.com>;
>>> 'starlingx-discuss at lists.starlingx.io'
>>> <starlingx-discuss at lists.starlingx.io>; Jascanu, Nicolae
>>> <nicolae.jascanu at intel.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> I also tested the issue 2 with latest daily build on duplex setup.
>>> The conclusion is that the issue is there all the time.
>>> This issue might not be fixed soon, but should not block OpenStack
>>> upgrade, right?
>>>
>>> For 9 OpenStack patches below, I have removed all workflow-1, except
>>> the first patch and add depends-on all them.
>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open)
>>> Your review and comments are welcome!
>>>
>>> As for issue 2, some detail info FYI.
>>> It also needs to wait for around 10 min before all pods are ready
>>> again after reboot for master build.
>>> It stuck on below 2 pods for 10 min. The same as the one I saw with
>>> my OpenStack upgrade engineering build.
>>> neutron-ovs-agent-controller-0-937646f6-xxznw(depends
>>> openvswitch-db)
>>> openvswitch-db-8fxkw
>>> Related key logs below.
>>> Warning FailedMount 2m19s kubelet, controller-1
>>> MountVolume.SetUp failed for volume "openvswitch-db-token-7g4qk" :
>>> failed to sync secret cache: timed out waiting for the condition
>>> Warning FailedMount 2m19s kubelet, controller-1
>>> MountVolume.SetUp failed for volume "openvswitch-bin" : failed to
>>> sync configmap cache: timed out waiting for the condition
>>> Warning FailedMount 105s kubelet, controller-1
>>> MountVolume.SetUp failed for volume "openvswitch-db-token-7g4qk" :
>>> failed to sync secret cache: timed out waiting for the condition
>>> Warning FailedMount 105s kubelet, controller-1
>>> MountVolume.SetUp failed for volume "openvswitch-bin" : failed to
>>> sync configmap cache: timed out waiting for the condition
>>> Warning Unhealthy 30s kubelet, controller-1
>>> Liveness probe failed: ovs-vsctl: unix:/var/run/openvswitch/db.sock:
>>> database connection failed (Permission denied)
>>> Warning Unhealthy 7s kubelet, controller-1
>>> Readiness probe failed: ovs-vsctl:
>>> unix:/var/run/openvswitch/db.sock: database connection failed
>>> (Permission denied)
>>>
>>> Is it the same stability issue as the one reported from your test
>>> team? I can only see this issue after force rebooting. What is our
>>> expected recovery time?
>>> Your comment is appreciated!
>>>
>>> Thanks!
>>> Zhipeng
>>> -----Original Message-----
>>> From: Liu, ZhipengS
>>> Sent: 2020年5月29日 9:42
>>> To: 'Miller, Frank' <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae
>>> <nicolae.jascanu at intel.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> Glad to see your quick reply!!
>>> For OpenStack upgrade task, we have finished all test and get
>>> patches ready for more than 2 weeks, but no any review comments and
>>> feedback from your side. What's the next step?
>>>
>>> For issue # 2, in community meeting notes, I saw that you had some
>>> stability issue from WR local test team. But so far, I do not see
>>> any LP for the detail info. You should ask them to do that! Right?
>>>
>>> According to your concern, I tried to reproduce it with my build
>>> (cherry pick OpenStack upgrade patches)yesterday, and the original
>>> issue [1] was not seen any more, mariadb got ready quickly, no
>>> regression.
>>>
>>> [1] https://bugs.launchpad.net/starlingx/+bug/1855474
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年5月29日 1:07
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae
>>> <nicolae.jascanu at intel.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Thanks Zhipeng.
>>>
>>> Good to see progress on IPv6.
>>> Waiting for 10 minutes for pods to recover isn't a good result. Is
>>> there a LP open on this issue? Which pods are not ready? What can
>>> you tell us about this 10 minute outage?
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Thursday, May 28, 2020 5:06 AM
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae
>>> <nicolae.jascanu at intel.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> Nicolae already added test case description. Thanks Nicolae!
>>>
>>> I also did below test on AIO-DX virtual setup, exactly according to
>>> your mentioned steps.
>>> No issue found, but just need to wait for around 10 min before all
>>> pods are ready again after reboot.
>>>
>>> For ipv6 issue, I have submitted new patch for it since dynamic
>>> override for database config did not work.
>>> https://review.opendev.org/#/c/731461/
>>> https://review.opendev.org/#/c/731470/
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年5月27日 22:43
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae
>>> <nicolae.jascanu at intel.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Zhipeng:
>>>
>>> Thanks for the info. You have provided the # of testcases but not
>>> what those testcase do. Where can I find a description of what the
>>> OpenStack testcases do?
>>>
>>> For the controller reset testcases I'd like to see the test result
>>> for the following:
>>> Is openstack usable during the following scenarios on AIO-DX and on
>>> Standard configurations:
>>> - Lock/unlock of standby controller
>>> - reset (ie: reboot -f) of the standby controller
>>> - reset (ie: reboot -f) of the active controller
>>> - reapply of stx-openstack after the above scenarios
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Wednesday, May 27, 2020 9:15 AM
>>> To: Miller, Frank <Frank.Miller at windriver.com>;
>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae
>>> <nicolae.jascanu at intel.com>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi Frank,
>>>
>>> We have done below tests.
>>> 1) Sanity tests by Nicolae.
>>> AIO - Simplex
>>> Setup 04 TCs [PASS]
>>> Provisioning 01 TCs [PASS]
>>> Sanity OpenStack 49 TCs [PASS]
>>> Sanity Platform 07 TCs [PASS]
>>>
>>> TOTAL: [ 61 TCs ]
>>>
>>> AIO - Duplex
>>> Setup 04 TCs [PASS]
>>> Provisioning 01 TCs [PASS]
>>> Sanity OpenStack 52 TCs [PASS]
>>> Sanity Platform 07 TCs [PASS]
>>>
>>> TOTAL: [ 64 TCs ]
>>>
>>> Standard - Local Storage (2+2)
>>> Setup 04 TCs [PASS]
>>> Provisioning 01 TCs [PASS]
>>> Sanity OpenStack 52 TCs [PASS]
>>> Sanity Platform 08 TCs [PASS]
>>>
>>> TOTAL: [ 65 TCs ]
>>>
>>> Standard External - Dedicated Storage (2+2+2)
>>> Setup 04 TCs [PASS]
>>> Provisioning 01 TCs [PASS]
>>> Sanity OpenStack 52 TCs [PASS]
>>> Sanity Platform 09 TCs [PASS]
>>>
>>> TOTAL: [ 66 TCs ]
>>>
>>> 2) NFV scenario test by me
>>> on duplex/multi standard virtual setup
>>> duplex bare metal setup
>>> ===== Setup
>>> =================================================================================================================================
>>> 2020-05-14 02:30:05.524 Create flavor small
>>> ........................................ [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor small_ephemeral
>>> .............................. [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor small_swap
>>> ................................... [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor small_ephemeral_swap
>>> ......................... [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor medium
>>> ....................................... [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor medium_ephemeral
>>> ............................. [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor medium_swap
>>> .................................. [OKAY]
>>> 2020-05-14 02:30:05.524 Create flavor medium_ephemeral_swap
>>> ........................ [OKAY]
>>> 2020-05-14 02:30:05.653 Create image cirros
>>> ........................................ [OKAY]
>>> 2020-05-14 02:30:05.695 Create volume cirros
>>> ....................................... [OKAY]
>>> 2020-05-14 02:30:05.695 Create volume cirros-ephemeral
>>> ............................. [OKAY]
>>> 2020-05-14 02:30:05.695 Create volume cirros-swap
>>> .................................. [OKAY]
>>> 2020-05-14 02:30:05.695 Create volume cirros-ephemeral-swap
>>> ........................ [OKAY]
>>> 2020-05-14 02:30:05.695 Create volume empty_volume
>>> ................................. [OKAY]
>>> 2020-05-14 02:30:05.786 Create network internal
>>> .................................... [OKAY]
>>> 2020-05-14 02:30:06.158 Create network external
>>> .................................... [OKAY]
>>> 2020-05-14 02:30:06.772 Create subnet internal
>>> ..................................... [OKAY]
>>> 2020-05-14 02:30:07.661 Create subnet external
>>> ..................................... [OKAY]
>>> 2020-05-14 02:30:08.553 Create instance cirros-1
>>> ................................... [OKAY]
>>> 2020-05-14 02:30:29.918 Create instance cirros-ephemeral-1
>>> ......................... [OKAY]
>>> 2020-05-14 02:30:43.160 Create instance cirros-swap-1
>>> .............................. [OKAY]
>>> 2020-05-14 02:30:56.101 Create instance cirros-ephemeral-swap-1
>>> .................... [OKAY]
>>> 2020-05-14 02:31:09.077 Create instance cirros-image-1
>>> ............................. [OKAY]
>>> 2020-05-14 02:31:21.241 Create instance
>>> cirros-image-with-volumes-1 ................ [OKAY]
>>> =============================================================================================================================================
>>> ===== Test Iteration 0 (single-execution)
>>> ===================================================================================================
>>> 2020-05-14 02:33:04.172 Test Instance-Pause
>>> ........................................ [OKAY] (2020-05-14
>>> 02:33:18.078 Δ=0:00:12.870)
>>> 2020-05-14 02:33:35.073 Test Instance-Unpause
>>> ...................................... [OKAY] (2020-05-14
>>> 02:33:41.608 Δ=0:00:05.866)
>>> 2020-05-14 02:33:53.049 Test Instance-Suspend
>>> ...................................... [OKAY] (2020-05-14
>>> 02:33:59.546 Δ=0:00:05.792)
>>> 2020-05-14 02:34:11.103 Test Instance-Resume
>>> ....................................... [OKAY] (2020-05-14
>>> 02:34:17.756 Δ=0:00:05.937)
>>> 2020-05-14 02:34:29.269 Test Instance-Reboot (soft)
>>> ................................ [OKAY] (2020-05-14 02:36:45.923
>>> Δ=0:02:15.748)
>>> 2020-05-14 02:37:02.160 Test Instance-Reboot (hard)
>>> ................................ [OKAY] (2020-05-14 02:37:14.504
>>> Δ=0:00:11.704)
>>> 2020-05-14 02:37:30.673 Test Instance-Stop
>>> ......................................... [OKAY] (2020-05-14
>>> 02:38:44.543 Δ=0:01:13.220)
>>> 2020-05-14 02:39:00.481 Test Instance-Start
>>> ........................................ [OKAY] (2020-05-14
>>> 02:39:07.198 Δ=0:00:06.068)
>>> 2020-05-14 02:39:18.578 Test Instance-Live-Migrate
>>> ................................. [OKAY] (2020-05-14 02:39:41.692
>>> Δ=0:00:22.306)
>>> 2020-05-14 02:39:57.927 Test Instance-Cold-Migrate
>>> ................................. [OKAY] (2020-05-14 02:41:22.720
>>> Δ=0:01:24.179)
>>> 2020-05-14 02:41:38.995 Test Instance-Cold-Migrate-Confirm
>>> ......................... [OKAY] (2020-05-14 02:41:45.441
>>> Δ=0:00:05.884)
>>> 2020-05-14 02:41:57.108 Test Instance-Cold-Migrate-Revert
>>> .......................... [OKAY] (2020-05-14 02:43:36.381
>>> Δ=0:00:21.637)
>>> 2020-05-14 02:43:52.320 Test Instance-Resize
>>> ....................................... [OKAY] (2020-05-14
>>> 02:45:16.409 Δ=0:01:22.812)
>>> 2020-05-14 02:45:32.723 Test Instance-Resize-Confirm
>>> ............................... [OKAY] (2020-05-14 02:45:39.119
>>> Δ=0:00:05.777)
>>> 2020-05-14 02:45:50.437 Test Instance-Resize-Revert
>>> ................................ [OKAY] (2020-05-14 02:47:30.175
>>> Δ=0:00:21.748)
>>> 2020-05-14 02:47:46.230 Test Instance-Rebuild
>>> ...................................... [OKAY] (2020-05-14
>>> 02:48:59.762 Δ=0:01:12.980)
>>> Total-Tests: 16 Execution-Time: 0:16:11.676
>>>
>>> 3) Another 2 test
>>> a) Using IPv6
>>> It can pass with workaround now. I need one more fix for it.
>>> In my previous patch https://review.opendev.org/#/c/716524
>>> (merged), I dynamically override below
>>> config_override: |
>>> [mysqld]
>>> bind_address=::
>>> However, it did not work now. From log, it shows error
>>> "OpenStack-Helm Mariadb - INFO - b'error: Found option without
>>> preceding group in config file: /etc/mysql/conf.d/20-override.cnf at
>>> line: 1'"
>>> I tried many methods, but could not remove the first line
>>> in 20-override.cnf
>>> mysql at mariadb-server-0:/etc/mysql/conf.d$ cat
>>> 20-override.cnf
>>> |-
>>> [mysqld]
>>> bind_address=::
>>> I can only add it in manifest.yaml as a static override
>>> like below.
>>> values:
>>> conf:
>>> database:
>>> config_override: |
>>> [mysqld]
>>> bind_address=::
>>> b) Reset of controllers and check status of
>>> OpenStack while a controller is rebooting.
>>> I have tested it and pass on simplex.
>>> For duplex, I have a setup issue in my side.
>>> @Jascanu, Nicolae Could you help me on it for duplex
>>> test, if you have time today. Thanks!
>>>
>>> Zhipeng
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>> Sent: 2020年5月26日 21:13
>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>;
>>> starlingx-discuss at lists.starlingx.io
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Zhipeng:
>>>
>>> Can you publish the list of tests that have been run for openstack?
>>>
>>> Also has openstack been tested for the following scenarios:
>>> 1) Using IPv6
>>> 2) Reset of controllers and check status of openstack while a
>>> controller is rebooting?
>>>
>>> Frank
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>> Sent: Monday, May 25, 2020 3:14 AM
>>> To: starlingx-discuss at lists.starlingx.io
>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi all,
>>>
>>> We have passed all sanity test on all setup. Thanks Nicolae!!
>>> We also built out OpenStack service images from layered build
>>> environment.
>>>
>>> Please help to review and push below patches to be merged, thanks!
>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open+OR+status)
>>>
>>> BRs
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS
>>> Sent: 2020年5月14日 16:49
>>> To: 'Saul Wold' <sgw at linux.intel.com>;
>>> 'starlingx-discuss at lists.starlingx.io'
>>> <starlingx-discuss at lists.starlingx.io>
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Hi all,
>>>
>>> Call for patch review again!
>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open+OR+status)
>>>
>>> Thanks!
>>> Zhipeng
>>>
>>> -----Original Message-----
>>> From: Liu, ZhipengS
>>> Sent: 2020年5月9日 8:38
>>> To: Saul Wold <sgw at linux.intel.com>;
>>> starlingx-discuss at lists.starlingx.io
>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> Agree!
>>>
>>> -----Original Message-----
>>> From: Saul Wold <sgw at linux.intel.com>
>>> Sent: 2020年5月9日 0:29
>>> To: starlingx-discuss at lists.starlingx.io
>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task]
>>> Call for patch review!!
>>>
>>> I would strengthen that to no changes until we get Green Sanity
>>> other than what's required to make them Green.
>>>
>>> Full Stop!
>>>
>>> Sau!
>>>
>>>
>>> On 5/8/20 9:05 AM, Miller, Frank wrote:
>>>> Until we can get sanity passing for several days in a row I strongly
>>>> suggest we do not allow any further changes into the load related to
>>>> OpenStack. Folks can continue with reviews but let’s hold off
>>>> allowing merges related to a new OpenStack version.
>>>>
>>>> Frank
>>>>
>>>> *From:*Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> *Sent:* Friday, May 08, 2020 11:59 AM
>>>> *To:* starlingx-discuss <starlingx-discuss at lists.starlingx.io>
>>>> *Cc:* YU CHENGDE <yu.chengde at 99cloud.net>; Penney, Don
>>>> <Don.Penney at windriver.com>
>>>> *Subject:* [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] Call
>>>> for patch review!!
>>>>
>>>> Hi all,
>>>>
>>>> Please help to review OpenStack Ussuri upgrade patches.
>>>>
>>>> Our target is to get all below patches merged by end of next week.
>>>>
>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open+OR+status
>>>> :merged)
>>>>
>>>> During OpenStack upgrade for StarlingX, we have to move python2.7 to
>>>> python3.6 for OpenStack services as ussuri release only support
>>>> python3.
>>>>
>>>> We also rebased openstack-helm/helm-infra to latest version.
>>>>
>>>> Engineering build test status.
>>>>
>>>> 1. nfv_scenario_tests PASS on simplex/duplex/multi virtual setup.
>>>> 2. nfv_scenario_tests PASS on simplex bare metal setup.
>>>> 3. Sanity test is ongoing. Duplex/standard virtual setup test
>>>> PASS.
>>>>
>>>> Thanks!
>>>>
>>>> Zhipeng
>>>>
>>>>
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>>
>>>
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
More information about the Starlingx-discuss
mailing list