Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie