[Starlingx-discuss] [OpenStack Ussuri Upgrade Task] Call for patch review!!

Saul Wold sgw at linux.intel.com
Wed Jun 10 21:51:15 UTC 2020


Scott,

Can you provide details of what you need and what we need to do on 
Jenkins for Davlet and I to work on it tomorrow when your off-line?

I am guessing you need a set of merged branches someplace that we can 
point jenkins at, but if that needs to be on Cengn or someplace else?

Thanks
Sau!


On 6/10/20 1:28 PM, Scott Little wrote:
> Six of the nine updates are in a state of merge conflict.
> 
> Please resolve the conflicts so that I can make progress wit a CENGN build.
> 
> Scott
> 
> 
> 
> On 2020-06-10 9:20 a.m., Scott Little wrote:
>> CENGN cycles aren't a problem.  People resources is a challenge.
>>
>> So the ask is for a manual build, on CENGN, adding in the nine patches 
>> listed by https://review.opendev.org/#/q/topic:for_ussuri+(status:open).
>>
>> .. and the addition of two repos to the build-stx-base.sh step
>>
>> build-stx-base.sh
>>    --repo local-stx-build,... \
>>    --repo stx-distro,... \
>>    --repo ussuri-ceph,http://download.ceph.com/rpm-mimic/el7/x86_64/ \
>>    --repo ussuri-wsgi,http://mirror.centos.org/centos/7/sclo/x86_64/rh/
>>
>>
>> Is that correct?
>>
>> Scott
>>
>>
>> On 2020-06-09 9:04 a.m., Saul Wold wrote:
>>>
>>> Frank, Scott, Davelet:
>>>
>>> Are there cycles available on Cengn (and people resources) to do a 
>>> Cengn build with the Ussuri patch set applied?  I know this is 
>>> different than a branch build.  I think we have done this kind of 
>>> thing in the past.
>>>
>>> This might help to make sure we don't have any more Cengn build 
>>> issues and could give the Test team a sanity spin with a Ussuri/Cengn 
>>> build.
>>>
>>> Note there is a comment for Scott/Davelet at the bottom of Zhipeng's 
>>> email.
>>>
>>> Thanks
>>>   Sau!
>>>
>>>
>>> On 6/9/20 1:39 AM, Liu, ZhipengS wrote:
>>>> Hi all,
>>>>
>>>> So far, all block issues and concerns have been addressed.
>>>> Since we have passed all sanity test, and Ussuri OpenStack has been 
>>>> officially released last month,
>>>> there should be no more reason to block these patches merge.
>>>>
>>>> Next step:
>>>> Let's push to get ussuri upgrade/openstack-helm rebasing patches 
>>>> merged. We need great help from core guys!
>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open)
>>>>
>>>> # Below 6 patches are for OpenStack-helm/infra rebase. (we set first 
>>>> patch with workflow-1 and add depends-on for other patches as we 
>>>> need to merge them together.)
>>>> Upgrade openstack-helm-infra zhipeng liu 
>>>> starlingx/openstack-armada-app       workflow-1
>>>> Add mariadb database config override to support ipv6 zhipeng liu 
>>>> starlingx/openstack-armada-app
>>>> Fix render error in cinder during openstack-helm rebase zhipeng 
>>>> liu    starlingx/openstack-armada-app
>>>> Update download list for openstack-helm upgrade zhipeng liu 
>>>> starlingx/openstack-armada-app
>>>> Update manifest.yaml file for openstack-helm upgrade. zhipeng liu 
>>>> starlingx/openstack-armada-app
>>>> Upgrade openstack-helm zhipeng liu starlingx/openstack-armada-app
>>>>
>>>> # Below 3 patches is for OpenStack upgrade.
>>>> Update manifest.yaml file for ussuri openstack YU CHENGDE 
>>>> starlingx/openstack-armada-app
>>>> Modify build-tools and stable-wheels for Ussuri upgrading YU 
>>>> CHENGDE    starlingx/root
>>>> Upgrade openstack docker images for stable/ussuri        YU 
>>>> CHENGDE    starlingx/upstream
>>>>
>>>>
>>>> After removing required python3 dependent packages from local, we 
>>>> can build out base image and OpenStack service images successfully 
>>>> with below command.
>>>> =============================================================================== 
>>>>
>>>> @Scott, please help to update cengn build script with below 2 
>>>> additional repos and help to trigger image build
>>>> build-stx-base.sh
>>>>    --repo local-stx-build,... \
>>>>    --repo stx-distro,... \
>>>>    --repo ussuri-ceph,http://download.ceph.com/rpm-mimic/el7/x86_64/ \
>>>>    --repo ussuri-wsgi,http://mirror.centos.org/centos/7/sclo/x86_64/rh/
>>>>
>>>> Thanks a lot!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS
>>>> Sent: 2020年6月8日 16:54
>>>> To: 'Miller, Frank' <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Friesen, Chris 
>>>> <Chris.Friesen at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> It is not easy to figure out whether/how/when OpenStack-helm-info 
>>>> upstream introduce this issue and then fix it.
>>>> I also could not find any fix in LP[1], which just mentioned that 
>>>> this intermittent issue not hit us after some changes in related field.
>>>>
>>>> Anyhow, below 2 patches should fix potential bug and I could not see 
>>>> the same error log again in our ussuri upgrade EB.
>>>> https://review.opendev.org/#/c/704034/ Prevent splitbrain during 
>>>> full Galera restart https://review.opendev.org/#/c/708071/ mariadb: 
>>>> avoid state management thread death
>>>>
>>>> Since we have passed fully test, we'd better push to merge ussuri 
>>>> upgrade/openstack-helm rebasing patches soon.
>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open)
>>>>
>>>> [1] https://bugs.launchpad.net/starlingx/+bug/1816842/
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年6月5日 22:32
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Friesen, Chris 
>>>> <Chris.Friesen at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Zhipeng:
>>>>
>>>> This looks promising.  Your theory is that the 2 
>>>> openstack-helm-infra patches will fix the mariadb recovery issues. 
>>>> These 2 patches were merged in the openstack-helm-infra project in 
>>>> January and February of 2020.   What would be good to know is what 
>>>> broke mariadb recovery between April of 2019 when Chris Friesen 
>>>> finished up his story [1] and our current loads today.  The most 
>>>> likely explanation is the upversion of Train or the upversion to 
>>>> openstack-helm-infra done in November 2019 introduced the mariadb 
>>>> recovery issues.  And then the openstack-helm folks found and fixed 
>>>> the issue earlier in 2020.
>>>>
>>>> If we had more time the preferred approach would be to merge just 
>>>> the openstack-helm-infra changes first to prove they address mariadb 
>>>> recovery and then in a separate commit merge Ussuri.  But since you 
>>>> have validated that mariadb recovers with your Ussuri branch and 
>>>> this branch has these openstack-helm commits, I support letting 
>>>> Ussuri merge into stx.4.0.
>>>>
>>>> Frank
>>>> [1] https://storyboard.openstack.org/#!/story/2004712
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Friday, June 05, 2020 2:36 AM
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Friesen, Chris 
>>>> <Chris.Friesen at windriver.com>
>>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> As for OpenStack not recovering after both controllers are reset [1] 
>>>> I could not reproduce this issue with my Ussuri upgrade EB.
>>>> My test step is:
>>>> 1) ssh to standby controller and sudo reboot -f for it.
>>>> 2) sudo reboot -f for activated controller All pods can resume after 
>>>> a while.
>>>>
>>>> However, I could reproduce this issue with DB 20200516T080009Z.
>>>>  From error logs,  it is an old issue analyzed by Chris Friesen in 
>>>> [2] early last year.
>>>>
>>>> In ussuri upgrade EB, we rebased openstack-helm-infra/mariadb.
>>>> It includes below 2 patches which fixed this stability issue.
>>>> https://review.opendev.org/#/c/704034/ Prevent splitbrain during 
>>>> full Galera restart https://review.opendev.org/#/c/708071/ mariadb: 
>>>> avoid state management thread death
>>>>
>>>> [1] https://bugs.launchpad.net/starlingx/+bug/1881899
>>>> [2] https://bugs.launchpad.net/starlingx/+bug/1816842/comments/3
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年6月3日 22:35
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Zhipeng:
>>>>
>>>> This is not a new requirement.  Users expect the software to recover 
>>>> when resets occur.
>>>>
>>>> As I had mentioned at the PTG yesterday I know personally that this 
>>>> test passed in stx3.0 before the upversion to train. Someone else 
>>>> who performs testing can look to determine when this test was done 
>>>> as part of feature testing after train was delivered as it should 
>>>> have been tested as part of stx.3.0 as well.  I do not know when 
>>>> this started to break.  One topic we will discuss at the PTG 
>>>> tomorrow will be how to improve our test coverage and automation so 
>>>> this type of issue can be found immediately as new code is being 
>>>> delivered.
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Wednesday, June 03, 2020 10:28 AM
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Frank,
>>>>
>>>> Have we pass this case before?  Is it a new requirement?
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年6月3日 22:12
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; Liu, ZhipengS 
>>>> <zhipengs.liu at intel.com>; starlingx-discuss at lists.starlingx.io; 
>>>> Church, Robert <Robert.Church at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Yong/Zhipeng - the LP for openstack not recovering after both 
>>>> controllers are reset is 
>>>> https://bugs.launchpad.net/starlingx/+bug/1881899
>>>>
>>>> Ovidiu is investigating and will provide any updates from his 
>>>> investigation.  Please continue to keep us informed of your 
>>>> investigation.
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: Tuesday, June 02, 2020 10:38 PM
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> We used a build from May 28.
>>>>
>>>> As for the decoupling issue these are actively being worked. If you 
>>>> run the system helm-override-show command when the stx-openstack app 
>>>> is applied you won’t see the CLI command fail.  It only fails when 
>>>> you try a helm-override-show when the app is in uploaded state.  In 
>>>> any case this will be fixed shortly.
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Tuesday, June 02, 2020 10:04 PM
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> Thanks for your quick update!
>>>> Which build are you using to test this case?
>>>> Since decoupling commits introduced several regressions (at least 
>>>> 2),  not propose to do this kind of stability test with latest build.
>>>> BTW, do we have plan to revert them considering this stability 
>>>> risk?  Our Ussuri upgrade patches is waiting for it☹
>>>>
>>>> Furthermore, we have not seen this test case that force reboot both 
>>>> controllers at the same time. Is it a new requirement? If not , have 
>>>> we pass this case before, which build?
>>>> I'd like to help on it with the pass build for comparative analysis. 
>>>> From my point , mariadb might not work if we reboot both controllers.
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年6月3日 8:55
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; Liu, ZhipengS 
>>>> <zhipengs.liu at intel.com>; starlingx-discuss at lists.starlingx.io; 
>>>> Church, Robert <Robert.Church at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Zhipeng:
>>>>
>>>> An update on our testing and analysis today.  We are able to 
>>>> reproduce the issue with OpenStack not recovering when we trigger a 
>>>> reboot of both AIO controllers at the same time. This results in 
>>>> MariaDB and multiple other OpenStack pods in CrashLoopBackoff and 
>>>> openstack commands not working indefinitely after the controllers 
>>>> recover.  We'll create a launchpad tomorrow to track this issue.
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: Tuesday, June 02, 2020 12:25 PM
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Thanks Zhipeng for the analysis.  What is challenging here is the 
>>>> multitude of issues.
>>>>
>>>> In our debug of openstack the past few days we are seeing the app 
>>>> fail completely.  After investigation this issue is a Day 1 
>>>> containerd issue.  This is tracked in LP: 
>>>> https://bugs.launchpad.net/starlingx/+bug/1881353
>>>>
>>>> The issue you are seeing on a swact is a new and very recent issue 
>>>> tied to the decoupling commits that were merged late last week.  Bob 
>>>> is investigating and I expect he'll have a fix soon for that.
>>>>
>>>> But the issues we are most concerned with are when we see mariadb 
>>>> crashing and not able to recover or with openstack services not 
>>>> working for longer periods of time.  We're attempting to isolate the 
>>>> sequence of events that trigger this.
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Tuesday, June 02, 2020 11:47 AM
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> For LP https://bugs.launchpad.net/starlingx/+bug/1881454
>>>>              Unable to unlock controller after swact and lock w/ 
>>>> openstack applied I also tested with daily build 20200516T080009Z. 
>>>> However, it could not be reproduced.
>>>> We should  fix this regression ASAP!
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: 2020年6月2日 16:48
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Church, Robert 
>>>> <Robert.Church at windriver.com>
>>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank and all,
>>>>
>>>> Update for issue 2.
>>>> I raised a new LP to track it.
>>>> https://bugs.launchpad.net/starlingx/+bug/1881722
>>>> Below is the time statistics. It seems reasonable. No obvious issue 
>>>> found.
>>>> 1) 3~4min for host restart and get ready.
>>>> 2) 2~3min for mariadb terminating, initialization, get ready. (then 
>>>> configmap sync is ready)
>>>> 3) 2min for ovs-db ready (reduce probe live/ready timer can improve 
>>>> a little, as it can retry quickly to connect ovs-vsctl: 
>>>> unix:/var/run/openvswitch/db.sock)
>>>> 4) 1min for other pods ready, like neutron-ovs-agent which depends 
>>>> on ovs-db. ) Any comment?
>>>>
>>>> For LP https://bugs.launchpad.net/starlingx/+bug/1881454
>>>>              Unable to unlock controller after swact and lock w/ 
>>>> openstack applied
>>>>     And  https://bugs.launchpad.net/starlingx/+bug/1881711
>>>>              system helm-override-show stx-openstack mariadb 
>>>> openstack crash  It seems related to openstack plugin decouple 
>>>> related patches. Should be a regression.
>>>>   Please see our update in this 2 LPs for detail info.  @Bob, could 
>>>> you pls help further check it and your patches, thanks!
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS
>>>> Sent: 2020年6月1日 16:20
>>>> To: 'Miller, Frank' <Frank.Miller at windriver.com>; 
>>>> 'starlingx-discuss at lists.starlingx.io' 
>>>> <starlingx-discuss at lists.starlingx.io>; Jascanu, Nicolae 
>>>> <nicolae.jascanu at intel.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> I also tested the issue 2 with latest daily build on duplex setup.
>>>> The conclusion is that the issue is there all the time.
>>>> This issue might not be fixed soon, but should not block OpenStack 
>>>> upgrade, right?
>>>>
>>>> For 9 OpenStack patches below, I have removed all workflow-1, except 
>>>> the first patch and add depends-on all them.
>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open)
>>>> Your review and comments are welcome!
>>>>
>>>> As for issue 2, some detail info FYI.
>>>> It also needs to wait for around 10 min before all pods are ready 
>>>> again after reboot for master build.
>>>> It stuck on below 2 pods for 10 min. The same as the one I saw with 
>>>> my OpenStack upgrade engineering build.
>>>>       neutron-ovs-agent-controller-0-937646f6-xxznw(depends 
>>>> openvswitch-db)
>>>>       openvswitch-db-8fxkw
>>>> Related key logs below.
>>>>    Warning  FailedMount  2m19s              kubelet, controller-1 
>>>> MountVolume.SetUp failed for volume "openvswitch-db-token-7g4qk" : 
>>>> failed to sync secret cache: timed out waiting for the condition
>>>>    Warning  FailedMount  2m19s              kubelet, controller-1 
>>>> MountVolume.SetUp failed for volume "openvswitch-bin" : failed to 
>>>> sync configmap cache: timed out waiting for the condition
>>>>    Warning  FailedMount  105s               kubelet, controller-1 
>>>> MountVolume.SetUp failed for volume "openvswitch-db-token-7g4qk" : 
>>>> failed to sync secret cache: timed out waiting for the condition
>>>>    Warning  FailedMount  105s               kubelet, controller-1 
>>>> MountVolume.SetUp failed for volume "openvswitch-bin" : failed to 
>>>> sync configmap cache: timed out waiting for the condition
>>>>    Warning  Unhealthy    30s                kubelet, controller-1 
>>>> Liveness probe failed: ovs-vsctl: unix:/var/run/openvswitch/db.sock: 
>>>> database connection failed (Permission denied)
>>>>    Warning  Unhealthy    7s                 kubelet, controller-1 
>>>> Readiness probe failed: ovs-vsctl: 
>>>> unix:/var/run/openvswitch/db.sock: database connection failed 
>>>> (Permission denied)
>>>>
>>>> Is it the same stability issue as the one reported from your test 
>>>> team?  I can only see this issue after force rebooting. What is our 
>>>> expected recovery time?
>>>> Your comment is appreciated!
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS
>>>> Sent: 2020年5月29日 9:42
>>>> To: 'Miller, Frank' <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae 
>>>> <nicolae.jascanu at intel.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> Glad to see your quick reply!!
>>>> For OpenStack upgrade task, we have finished all test and get 
>>>> patches ready for more than 2 weeks, but no any review comments and 
>>>> feedback from your side.  What's the next step?
>>>>
>>>> For issue # 2,  in community meeting notes,  I saw that you had some 
>>>> stability issue from WR local test team. But so far, I do not see 
>>>> any LP for the detail info. You should ask them to do that!  Right?
>>>>
>>>> According to your concern, I tried to reproduce it with my build 
>>>> (cherry pick OpenStack upgrade patches)yesterday, and the original 
>>>> issue [1] was not seen any more, mariadb got ready quickly, no 
>>>> regression.
>>>>
>>>> [1] https://bugs.launchpad.net/starlingx/+bug/1855474
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年5月29日 1:07
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae 
>>>> <nicolae.jascanu at intel.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Thanks Zhipeng.
>>>>
>>>> Good to see progress on IPv6.
>>>> Waiting for 10 minutes for pods to recover isn't a good result. Is 
>>>> there a LP open on this issue?  Which pods are not ready? What can 
>>>> you tell us about this 10 minute outage?
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Thursday, May 28, 2020 5:06 AM
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae 
>>>> <nicolae.jascanu at intel.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> Nicolae already added test case description. Thanks Nicolae!
>>>>
>>>> I also did below test on AIO-DX virtual setup, exactly according to 
>>>> your mentioned steps.
>>>> No issue found, but just need to wait for around 10 min before all 
>>>> pods are ready again after reboot.
>>>>
>>>> For ipv6 issue, I have submitted new patch for it since dynamic 
>>>> override for database config did not work.
>>>>   https://review.opendev.org/#/c/731461/
>>>>   https://review.opendev.org/#/c/731470/
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年5月27日 22:43
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae 
>>>> <nicolae.jascanu at intel.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Zhipeng:
>>>>
>>>> Thanks for the info.  You have provided the # of testcases but not 
>>>> what those testcase do.  Where can I find a description of what the 
>>>> OpenStack testcases do?
>>>>
>>>> For the controller reset testcases I'd like to see the test result 
>>>> for the following:
>>>> Is openstack usable during the following scenarios on AIO-DX and on 
>>>> Standard configurations:
>>>> - Lock/unlock of standby controller
>>>> - reset (ie: reboot -f) of the standby controller
>>>> - reset (ie: reboot -f) of the active controller
>>>> - reapply of stx-openstack after the above scenarios
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Wednesday, May 27, 2020 9:15 AM
>>>> To: Miller, Frank <Frank.Miller at windriver.com>; 
>>>> starlingx-discuss at lists.starlingx.io; Jascanu, Nicolae 
>>>> <nicolae.jascanu at intel.com>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi Frank,
>>>>
>>>> We have done below tests.
>>>> 1) Sanity tests by Nicolae.
>>>> AIO - Simplex
>>>> Setup                                    04 TCs [PASS]
>>>> Provisioning                       01 TCs [PASS]
>>>> Sanity OpenStack             49 TCs [PASS]
>>>> Sanity Platform                 07 TCs [PASS]
>>>>
>>>> TOTAL: [ 61 TCs ]
>>>>
>>>> AIO - Duplex
>>>> Setup                                    04 TCs [PASS]
>>>> Provisioning                       01 TCs [PASS]
>>>> Sanity OpenStack             52 TCs [PASS]
>>>> Sanity Platform                 07 TCs [PASS]
>>>>
>>>> TOTAL: [ 64 TCs ]
>>>>
>>>> Standard - Local Storage (2+2)
>>>> Setup                                    04 TCs [PASS]
>>>> Provisioning                       01 TCs [PASS]
>>>> Sanity OpenStack             52 TCs [PASS]
>>>> Sanity Platform                 08 TCs [PASS]
>>>>
>>>> TOTAL: [ 65 TCs ]
>>>>
>>>> Standard External - Dedicated Storage (2+2+2)
>>>> Setup                                    04 TCs [PASS]
>>>> Provisioning                       01 TCs [PASS]
>>>> Sanity OpenStack             52 TCs [PASS]
>>>> Sanity Platform                 09 TCs [PASS]
>>>>
>>>> TOTAL: [ 66 TCs ]
>>>>
>>>> 2) NFV scenario test by me
>>>>      on duplex/multi standard virtual setup
>>>>            duplex bare metal setup
>>>> ===== Setup 
>>>> ================================================================================================================================= 
>>>>
>>>> 2020-05-14 02:30:05.524  Create flavor small 
>>>> ........................................ [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor small_ephemeral 
>>>> .............................. [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor small_swap 
>>>> ................................... [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor small_ephemeral_swap 
>>>> ......................... [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor medium 
>>>> ....................................... [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor medium_ephemeral 
>>>> ............................. [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor medium_swap 
>>>> .................................. [OKAY]
>>>> 2020-05-14 02:30:05.524  Create flavor medium_ephemeral_swap 
>>>> ........................ [OKAY]
>>>> 2020-05-14 02:30:05.653  Create image cirros 
>>>> ........................................ [OKAY]
>>>> 2020-05-14 02:30:05.695  Create volume cirros 
>>>> ....................................... [OKAY]
>>>> 2020-05-14 02:30:05.695  Create volume cirros-ephemeral 
>>>> ............................. [OKAY]
>>>> 2020-05-14 02:30:05.695  Create volume cirros-swap 
>>>> .................................. [OKAY]
>>>> 2020-05-14 02:30:05.695  Create volume cirros-ephemeral-swap 
>>>> ........................ [OKAY]
>>>> 2020-05-14 02:30:05.695  Create volume empty_volume 
>>>> ................................. [OKAY]
>>>> 2020-05-14 02:30:05.786  Create network internal 
>>>> .................................... [OKAY]
>>>> 2020-05-14 02:30:06.158  Create network external 
>>>> .................................... [OKAY]
>>>> 2020-05-14 02:30:06.772  Create subnet internal 
>>>> ..................................... [OKAY]
>>>> 2020-05-14 02:30:07.661  Create subnet external 
>>>> ..................................... [OKAY]
>>>> 2020-05-14 02:30:08.553  Create instance cirros-1 
>>>> ................................... [OKAY]
>>>> 2020-05-14 02:30:29.918  Create instance cirros-ephemeral-1 
>>>> ......................... [OKAY]
>>>> 2020-05-14 02:30:43.160  Create instance cirros-swap-1 
>>>> .............................. [OKAY]
>>>> 2020-05-14 02:30:56.101  Create instance cirros-ephemeral-swap-1 
>>>> .................... [OKAY]
>>>> 2020-05-14 02:31:09.077  Create instance cirros-image-1 
>>>> ............................. [OKAY]
>>>> 2020-05-14 02:31:21.241  Create instance 
>>>> cirros-image-with-volumes-1  ................ [OKAY] 
>>>> ============================================================================================================================================= 
>>>>
>>>> ===== Test Iteration 0 (single-execution) 
>>>> =================================================================================================== 
>>>>
>>>> 2020-05-14 02:33:04.172  Test Instance-Pause 
>>>> ........................................ [OKAY]  (2020-05-14 
>>>> 02:33:18.078 Δ=0:00:12.870)
>>>> 2020-05-14 02:33:35.073  Test Instance-Unpause 
>>>> ...................................... [OKAY]  (2020-05-14 
>>>> 02:33:41.608 Δ=0:00:05.866)
>>>> 2020-05-14 02:33:53.049  Test Instance-Suspend 
>>>> ...................................... [OKAY]  (2020-05-14 
>>>> 02:33:59.546 Δ=0:00:05.792)
>>>> 2020-05-14 02:34:11.103  Test Instance-Resume 
>>>> ....................................... [OKAY]  (2020-05-14 
>>>> 02:34:17.756 Δ=0:00:05.937)
>>>> 2020-05-14 02:34:29.269  Test Instance-Reboot (soft) 
>>>> ................................ [OKAY]  (2020-05-14 02:36:45.923 
>>>> Δ=0:02:15.748)
>>>> 2020-05-14 02:37:02.160  Test Instance-Reboot (hard) 
>>>> ................................ [OKAY]  (2020-05-14 02:37:14.504 
>>>> Δ=0:00:11.704)
>>>> 2020-05-14 02:37:30.673  Test Instance-Stop 
>>>> ......................................... [OKAY]  (2020-05-14 
>>>> 02:38:44.543 Δ=0:01:13.220)
>>>> 2020-05-14 02:39:00.481  Test Instance-Start 
>>>> ........................................ [OKAY]  (2020-05-14 
>>>> 02:39:07.198 Δ=0:00:06.068)
>>>> 2020-05-14 02:39:18.578  Test Instance-Live-Migrate 
>>>> ................................. [OKAY]  (2020-05-14 02:39:41.692 
>>>> Δ=0:00:22.306)
>>>> 2020-05-14 02:39:57.927  Test Instance-Cold-Migrate 
>>>> ................................. [OKAY]  (2020-05-14 02:41:22.720 
>>>> Δ=0:01:24.179)
>>>> 2020-05-14 02:41:38.995  Test Instance-Cold-Migrate-Confirm 
>>>> ......................... [OKAY]  (2020-05-14 02:41:45.441 
>>>> Δ=0:00:05.884)
>>>> 2020-05-14 02:41:57.108  Test Instance-Cold-Migrate-Revert 
>>>> .......................... [OKAY]  (2020-05-14 02:43:36.381 
>>>> Δ=0:00:21.637)
>>>> 2020-05-14 02:43:52.320  Test Instance-Resize 
>>>> ....................................... [OKAY]  (2020-05-14 
>>>> 02:45:16.409 Δ=0:01:22.812)
>>>> 2020-05-14 02:45:32.723  Test Instance-Resize-Confirm 
>>>> ............................... [OKAY]  (2020-05-14 02:45:39.119 
>>>> Δ=0:00:05.777)
>>>> 2020-05-14 02:45:50.437  Test Instance-Resize-Revert 
>>>> ................................ [OKAY]  (2020-05-14 02:47:30.175 
>>>> Δ=0:00:21.748)
>>>> 2020-05-14 02:47:46.230  Test Instance-Rebuild 
>>>> ...................................... [OKAY]  (2020-05-14 
>>>> 02:48:59.762 Δ=0:01:12.980)
>>>> Total-Tests: 16     Execution-Time: 0:16:11.676
>>>>
>>>> 3) Another 2 test
>>>>      a) Using IPv6
>>>>           It can pass with workaround now.  I need one more fix for it.
>>>>           In my previous patch https://review.opendev.org/#/c/716524 
>>>> (merged), I dynamically override below
>>>>              config_override: |
>>>>                  [mysqld]
>>>>                  bind_address=::
>>>>           However, it did not work now. From log,  it shows error 
>>>> "OpenStack-Helm Mariadb - INFO - b'error: Found option without 
>>>> preceding group in config file: /etc/mysql/conf.d/20-override.cnf at 
>>>> line: 1'"
>>>>           I tried many methods, but could not remove the first line 
>>>> in 20-override.cnf
>>>>                  mysql at mariadb-server-0:/etc/mysql/conf.d$ cat 
>>>> 20-override.cnf
>>>>                  |-
>>>>                  [mysqld]
>>>>                  bind_address=::
>>>>          I can only add it in manifest.yaml as a static override 
>>>> like below.
>>>>                 values:
>>>>                    conf:
>>>>                        database:
>>>>                            config_override: |
>>>>                                [mysqld]
>>>>                                bind_address=::
>>>>                   b) Reset of controllers and check status of 
>>>> OpenStack while a controller is rebooting.
>>>>           I have tested it and pass on simplex.
>>>>           For duplex, I have a setup issue in my side.
>>>>           @Jascanu, Nicolae  Could you help me on it for duplex 
>>>> test, if you have time today. Thanks!
>>>>
>>>> Zhipeng
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Miller, Frank <Frank.Miller at windriver.com>
>>>> Sent: 2020年5月26日 21:13
>>>> To: Liu, ZhipengS <zhipengs.liu at intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Zhipeng:
>>>>
>>>> Can you publish the list of tests that have been run for openstack?
>>>>
>>>> Also has openstack been tested for the following scenarios:
>>>> 1) Using IPv6
>>>> 2) Reset of controllers and check status of openstack while a 
>>>> controller is rebooting?
>>>>
>>>> Frank
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS <zhipengs.liu at intel.com>
>>>> Sent: Monday, May 25, 2020 3:14 AM
>>>> To: starlingx-discuss at lists.starlingx.io
>>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi all,
>>>>
>>>> We have passed all sanity test on all setup. Thanks Nicolae!!
>>>> We also built out OpenStack service images from layered build 
>>>> environment.
>>>>
>>>> Please help to review and push below patches to be merged, thanks!
>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open+OR+status)
>>>>
>>>> BRs
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS
>>>> Sent: 2020年5月14日 16:49
>>>> To: 'Saul Wold' <sgw at linux.intel.com>; 
>>>> 'starlingx-discuss at lists.starlingx.io' 
>>>> <starlingx-discuss at lists.starlingx.io>
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Hi all,
>>>>
>>>> Call for patch review again!
>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open+OR+status)
>>>>
>>>> Thanks!
>>>> Zhipeng
>>>>
>>>> -----Original Message-----
>>>> From: Liu, ZhipengS
>>>> Sent: 2020年5月9日 8:38
>>>> To: Saul Wold <sgw at linux.intel.com>; 
>>>> starlingx-discuss at lists.starlingx.io
>>>> Subject: RE: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> Agree!
>>>>
>>>> -----Original Message-----
>>>> From: Saul Wold <sgw at linux.intel.com>
>>>> Sent: 2020年5月9日 0:29
>>>> To: starlingx-discuss at lists.starlingx.io
>>>> Subject: Re: [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] 
>>>> Call for patch review!!
>>>>
>>>> I would strengthen that to no changes until we get Green Sanity 
>>>> other than what's required to make them Green.
>>>>
>>>> Full Stop!
>>>>
>>>> Sau!
>>>>
>>>>
>>>> On 5/8/20 9:05 AM, Miller, Frank wrote:
>>>>> Until we can get sanity passing for several days in a row I strongly
>>>>> suggest we do not allow any further changes into the load related to
>>>>> OpenStack.  Folks can continue with reviews but let’s hold off
>>>>> allowing merges related to a new OpenStack version.
>>>>>
>>>>> Frank
>>>>>
>>>>> *From:*Liu, ZhipengS <zhipengs.liu at intel.com>
>>>>> *Sent:* Friday, May 08, 2020 11:59 AM
>>>>> *To:* starlingx-discuss <starlingx-discuss at lists.starlingx.io>
>>>>> *Cc:* YU CHENGDE <yu.chengde at 99cloud.net>; Penney, Don
>>>>> <Don.Penney at windriver.com>
>>>>> *Subject:* [Starlingx-discuss] [OpenStack Ussuri Upgrade Task] Call
>>>>> for patch review!!
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Please help to review OpenStack Ussuri upgrade patches.
>>>>>
>>>>> Our target is to get all below patches merged by end of next week.
>>>>>
>>>>> https://review.opendev.org/#/q/topic:for_ussuri+(status:open+OR+status
>>>>> :merged)
>>>>>
>>>>> During OpenStack upgrade for StarlingX, we have to move python2.7 to
>>>>> python3.6 for OpenStack services as ussuri release only support 
>>>>> python3.
>>>>>
>>>>> We also rebased openstack-helm/helm-infra to latest version.
>>>>>
>>>>> Engineering build test status.
>>>>>
>>>>>   1. nfv_scenario_tests PASS on simplex/duplex/multi virtual setup.
>>>>>   2. nfv_scenario_tests PASS on simplex bare metal setup.
>>>>>   3. Sanity test is ongoing.   Duplex/standard virtual setup test 
>>>>> PASS.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Zhipeng
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Starlingx-discuss mailing list
>>>>> Starlingx-discuss at lists.starlingx.io
>>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>>>
>>>>
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>>>
>>>
>>> _______________________________________________
>>> Starlingx-discuss mailing list
>>> Starlingx-discuss at lists.starlingx.io
>>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>>
>> _______________________________________________
>> Starlingx-discuss mailing list
>> Starlingx-discuss at lists.starlingx.io
>> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> 
> 
> 
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss



More information about the Starlingx-discuss mailing list