[Starlingx-discuss] OVS-DPDK Upgrade Testing

Wed Mar 27 09:45:44 UTC 2019

Hi Matt,
I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174230Z/outputs/iso/

Best Regards,
Xu, Chenjie

From: Xu, Chenjie
Sent: Wednesday, March 27, 2019 10:38 AM
To: 'Peters, Matt' <Matt.Peters at windriver.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Lin, Shuicheng <shuicheng.lin at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Perez, Ricardo O <ricardo.o.perez at intel.com>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>; Zhao, Forrest <forrest.zhao at intel.com>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
Sorry for misleading! I mean “NUMA sections are still missing”.

I have multiple systems experiencing this issue:
1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals
2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals
3. 0305 ISO image on 4 bare metals
4. 0315 ISO image one 1 bare metals
The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal.

I will try 0322 ISO image on 4 bare metals.

Best Regards,
Xu, Chenjie

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Tuesday, March 26, 2019 11:10 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hello Chenjie,
Can you clarify this statement? “NUMA sections missing bug still exits”.
Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing?

Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology?  I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue.

As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load.

-Matt

From: Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Tuesday, March 26, 2019 at 9:28 AM
To: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>, "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare.

Ada,
As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated?

Thanks,
Ghada

From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Tuesday, March 26, 2019 7:09 AM
To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits:
https://bugs.launchpad.net/starlingx/+bug/1820378

This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions?

Best Regards,
Xu, Chenjie

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Friday, March 22, 2019 8:31 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hello Chenjie,
Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images.  Since some of them align, I think you might be running a different set of images.  Can you confirm what the build baseline is that you pulled the images from?

I have attached a sample list from our labs for comparison.

-Matt

From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Friday, March 22, 2019 at 7:25 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image.

Best Regards,
Xu, Chenjie

From: Martinez Monroy, Elio [mailto:elio.martinez.monroy at intel.com]
Sent: Thursday, March 21, 2019 11:20 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Perfect, I will be cheking my mail  for that new ISO.

Adding Richo , he is the one that is going to execute the tests.

Thanks

BR

Elio

From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Thursday, March 21, 2019 9:15 AM
To: Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Shuicheng,
We discussed this in the networking bi-weekly team meeting today.

Please re-enable the mlx pmd – re-apply the following patch:
https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/centos/meta_patches/0005-enable-mlx-pmds.patch

Then please rebase to the latest master and build a new load for Ada’s team to start testing

Hi Elio,
One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates.

Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade).  Please note that none of these issues are reproducible in WR labs:
https://bugs.launchpad.net/starlingx/+bug/1821150
https://bugs.launchpad.net/starlingx/+bug/1821135
https://bugs.launchpad.net/starlingx/+bug/1820378

Thanks,
Ghada

From: Khalil, Ghada
Sent: Thursday, March 21, 2019 9:12 AM
To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Shuicheng,
There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well.

Can you confirm that the load being used has the mellanox pmd enabled in it?

Thanks,
Ghada

From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
Sent: Thursday, March 21, 2019 9:03 AM
To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Thanks Chenjie for the testing.

Hi all,
Does it mean greenlight for code merge, or any other test is needed?
Here is the patch list:
https://review.openstack.org/642672
https://review.openstack.org/642673

Best Regards
Shuicheng

From: Xu, Chenjie
Sent: Thursday, March 21, 2019 5:46 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Peng, Peng <Peng.Peng at windriver.com<mailto:Peng.Peng at windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi team,
I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries:

Bug Summary:
No bug related to OVSDPDK Upgrade.
3 bugs related to cutting over to container and configuration.
    https://bugs.launchpad.net/starlingx/+bug/1821150
                   https://bugs.launchpad.net/starlingx/+bug/1821135
                   https://bugs.launchpad.net/starlingx/+bug/1820378

The following features have been tested:
N/S traffic for FLAT/VLAN network
E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts)
QOS
Security Group
MTU

Best Regards,
Xu, Chenjie

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/26108bf4/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0322-standard-build-info.txt
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/26108bf4/attachment-0003.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0322-standard-compute-0.txt
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/26108bf4/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 0322-standard-controller-0.txt
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/26108bf4/attachment-0005.txt>