Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Peters, Matt <Matt.Peters@windriver.com>; Chris.Winnicki@windriver.com Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Chris.Winnicki@windriver.com; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Chris.Winnicki@windriver.com; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com>; Xu, Chenjie <chenjie.xu@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Chris.Winnicki@windriver.com; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com>, "Xie, Cindy" <cindy.xie@intel.com>, "Zhao, Forrest" <forrest.zhao@intel.com>, "Peters, Matt" <Matt.Peters@windriver.com>, "Richard, Joseph" <Joseph.Richard@windriver.com>, "Winnicki, Chris" <Chris.Winnicki@windriver.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Qin, Kailun" <kailun.qin@intel.com>, "Guo, Ruijing" <ruijing.guo@intel.com>, "Le, Huifeng" <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Xu, Chenjie <chenjie.xu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Xie, Cindy" <cindy.xie@intel.com>, "Zhao, Forrest" <forrest.zhao@intel.com>, "Richard, Joseph" <Joseph.Richard@windriver.com>, "Winnicki, Chris" <Chris.Winnicki@windriver.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Peng, Peng" <Peng.Peng@windriver.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Qin, Kailun" <kailun.qin@intel.com>, "Guo, Ruijing" <ruijing.guo@intel.com>, "Le, Huifeng" <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Xie, Cindy" <cindy.xie@intel.com>, "Zhao, Forrest" <forrest.zhao@intel.com>, "Richard, Joseph" <Joseph.Richard@windriver.com>, "Winnicki, Chris" <Chris.Winnicki@windriver.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Peng, Peng" <Peng.Peng@windriver.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Qin, Kailun" <kailun.qin@intel.com>, "Guo, Ruijing" <ruijing.guo@intel.com>, "Le, Huifeng" <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Chenjie, I added some comments to the bug report and attached a domain XML from one of the Wind River labs. The numa sections you identity are present, so I included some software load information as well in case you are running a different software lineup (other than the openvswitch upgrade) that may be contributing to what you have observed. Regards, Matt From: "Xie, Cindy" <cindy.xie@intel.com> Date: Monday, March 18, 2019 at 11:01 AM To: "Xu, Chenjie" <chenjie.xu@intel.com>, "Peters, Matt" <Matt.Peters@windriver.com>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Zhao, Forrest" <forrest.zhao@intel.com>, "Richard, Joseph" <Joseph.Richard@windriver.com>, "Winnicki, Chris" <Chris.Winnicki@windriver.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Peng, Peng" <Peng.Peng@windriver.com>, "Liu, ZhipengS" <zhipengs.liu@intel.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Jones, Bruce E" <bruce.e.jones@intel.com>, "Qin, Kailun" <kailun.qin@intel.com>, "Guo, Ruijing" <ruijing.guo@intel.com>, "Le, Huifeng" <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Cindy, The feature mentioned below is not included in the patch-set that I’m working on. That is just for migration related Numa data. BTW, using below command openstack server show <VM> I could not see numa topology related info that I can see in non-container build Thanks! Zhipeng From: Xie, Cindy Sent: 2019年3月18日 23:01 To: Xu, Chenjie <chenjie.xu@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Zhao, Forrest <forrest.zhao@intel.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Just to clarify, StarlingX is currently using master nova. As far as I know, there should be no dependency on starlingx specific nova numa patches to have VM connectivity. As noted by Matt Peters and confirmed by Numan and Peng, we do not see this VM connectivity issue in the WR labs using cengn builds. Ada, Can you please clarify if your team is running VM connectivity tests as part of their regular sanity as requested previously? Are you seeing the same issue reported by Chenjie? Thanks, Ghada From: Liu, ZhipengS [mailto:zhipengs.liu@intel.com] Sent: Monday, March 18, 2019 11:07 PM To: Xie, Cindy; Xu, Chenjie; Peters, Matt; Khalil, Ghada; Zhao, Forrest; Richard, Joseph; Winnicki, Chris; Lin, Shuicheng; Peng, Peng Cc: 'starlingx-discuss@lists.starlingx.io'; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, The feature mentioned below is not included in the patch-set that I’m working on. That is just for migration related Numa data. BTW, using below command openstack server show <VM> I could not see numa topology related info that I can see in non-container build Thanks! Zhipeng From: Xie, Cindy Sent: 2019年3月18日 23:01 To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 20, 2019 10:55 PM To: Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Xu, Chenjie <chenjie.xu@intel.com>; Cabrales, Ada <ada.cabrales@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Just to clarify, StarlingX is currently using master nova. As far as I know, there should be no dependency on starlingx specific nova numa patches to have VM connectivity. As noted by Matt Peters and confirmed by Numan and Peng, we do not see this VM connectivity issue in the WR labs using cengn builds. Ada, Can you please clarify if your team is running VM connectivity tests as part of their regular sanity as requested previously? Are you seeing the same issue reported by Chenjie? Thanks, Ghada From: Liu, ZhipengS [mailto:zhipengs.liu@intel.com] Sent: Monday, March 18, 2019 11:07 PM To: Xie, Cindy; Xu, Chenjie; Peters, Matt; Khalil, Ghada; Zhao, Forrest; Richard, Joseph; Winnicki, Chris; Lin, Shuicheng; Peng, Peng Cc: 'starlingx-discuss@lists.starlingx.io'; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, The feature mentioned below is not included in the patch-set that I’m working on. That is just for migration related Numa data. BTW, using below command openstack server show <VM> I could not see numa topology related info that I can see in non-container build Thanks! Zhipeng From: Xie, Cindy Sent: 2019年3月18日 23:01 To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Cabrales, Ada <ada.cabrales@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 20, 2019 10:55 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Just to clarify, StarlingX is currently using master nova. As far as I know, there should be no dependency on starlingx specific nova numa patches to have VM connectivity. As noted by Matt Peters and confirmed by Numan and Peng, we do not see this VM connectivity issue in the WR labs using cengn builds. Ada, Can you please clarify if your team is running VM connectivity tests as part of their regular sanity as requested previously? Are you seeing the same issue reported by Chenjie? Thanks, Ghada From: Liu, ZhipengS [mailto:zhipengs.liu@intel.com] Sent: Monday, March 18, 2019 11:07 PM To: Xie, Cindy; Xu, Chenjie; Peters, Matt; Khalil, Ghada; Zhao, Forrest; Richard, Joseph; Winnicki, Chris; Lin, Shuicheng; Peng, Peng Cc: 'starlingx-discuss@lists.starlingx.io'; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, The feature mentioned below is not included in the patch-set that I’m working on. That is just for migration related Numa data. BTW, using below command openstack server show <VM> I could not see numa topology related info that I can see in non-container build Thanks! Zhipeng From: Xie, Cindy Sent: 2019年3月18日 23:01 To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 20, 2019 10:55 PM To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Just to clarify, StarlingX is currently using master nova. As far as I know, there should be no dependency on starlingx specific nova numa patches to have VM connectivity. As noted by Matt Peters and confirmed by Numan and Peng, we do not see this VM connectivity issue in the WR labs using cengn builds. Ada, Can you please clarify if your team is running VM connectivity tests as part of their regular sanity as requested previously? Are you seeing the same issue reported by Chenjie? Thanks, Ghada From: Liu, ZhipengS [mailto:zhipengs.liu@intel.com] Sent: Monday, March 18, 2019 11:07 PM To: Xie, Cindy; Xu, Chenjie; Peters, Matt; Khalil, Ghada; Zhao, Forrest; Richard, Joseph; Winnicki, Chris; Lin, Shuicheng; Peng, Peng Cc: 'starlingx-discuss@lists.starlingx.io'; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, The feature mentioned below is not included in the patch-set that I’m working on. That is just for migration related Numa data. BTW, using below command openstack server show <VM> I could not see numa topology related info that I can see in non-container build Thanks! Zhipeng From: Xie, Cindy Sent: 2019年3月18日 23:01 To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie to root-cause the issue. I am adding Zhipeng who is working on Nova-NUMA related patches and see if the feature has been included in the patch-set he is working on. Th.x - cindy From: Xu, Chenjie Sent: Monday, March 18, 2019 10:05 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada/Matt, I have located the cause of this bug: the domain XML file created by StarlingX lacks numa related sections. Need nova expert to fix this bug because it’s stx-nova to create domain XML file. By manually adding numa related sections, I can continue the OVSDPDK Upgrade Testing. But some unexpected behavior may exist. And I have tested the E/W traffic for VLAN network ( within the same host and across hosts). I have reported the bug in launchpad and you can find my debugging steps in the following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Monday, March 18, 2019 8:40 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Sorry for the delayed reply. For the port that is created / plugged by OpenStack, can you confirm the flows that are visible to the OVS datapath? Are the packets seen by OVS br-int and then not switched properly or are the packets not being processed by the vhost port at all? If you are seeing packets reach the bridge, can you confirm if all packet types are not getting through, i.e. is ARP working, but not ICMP. You can dump the datapath ports and flows with the following command: ovs-appctl dpif/show ovs-appctl dpif/dump-flows br-int Also, when you did your manual test, did you have the other VM present as well, or was it a fully manual setup with no OpenStack agents present? Regards, Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 15, 2019 at 2:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, We create vhostuser port and VM manually. And the connectivity between manually created VM and br-int works correctly. But the vhostuser port created by StarlingX doesn’t work. Could you please help review our following debugging steps and leave a comment? 1. We manually assigned correct IP to the StarlingX VM’s interface eth0 and ping DHCP server in VM. No packet can be captured by tcpdump and thus we think vhostuser port doesn’t work. Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Check the IP assigned to the VM: openstack server list Manually assign IP to VM: sudo ifconfig 192.168.101.216/24 up Tcpdump in dhcp namespace: sudo ip netns exec $dhcp_namespace tcpdump –i $tap_device Ping DHCP server in VM: ping 192.168.101.1 No packets can be captured. 2. We added our own vhost user port to br-int and use virsh command and cirros-dpdk-vhostuserclient.xml (you can find in attached files) to create VM: Create vhostuser port: ovs-vsctl add-port br-int vhost-user-1 -- set Interface vhost-user-1 type=dpdkvhostuserclient options:vhost-server-path=/var/run/openvswitch/vhost-user-1 Create VM: virsh create cirros-dpdk-vhostuserclient.xml Bring up br-int to verify the connectivity: sudo ifconfig br-int 192.168.101.1/24 up Manually assign an IP to VM: sudo ifconfig eth0 192.168.101.18/24 up In created VM, ping br-int: ping 192.168.101.1 And the ping succeeds. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 13, 2019 9:13 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, That message is a normal message from DPDK if you have multiple huge page sizes configured in your system. Openvswitch is configured to use 1G huge pages and will ignore the 2M huge pages. You can confirm this by looking at the Puppet logs to see what was configured and to ensure the 1G huge pages were mounted prior to openvswitch starting. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 13, 2019 at 8:55 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Peng, Peng" <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I installed StarlingX using following the March 5th ISO image and the issue still exists:http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190305T060... Are there any steps not been written on the guide? I just followed the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard I checked the ovs-vswitchd.log and the following line maybe the cause: 23127 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size It seems that something went wrong with hugepage. Some logs have been attached. If you have any idea about hugepage, please let me know. Peng has been added and hope Peng can provide some information about his environment. https://bugs.launchpad.net/starlingx/+bug/1819031 Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Tuesday, March 12, 2019 12:39 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, For Matt, I just tried to lock compute-1 to ensure the VMs is on the same compute host. But it still doesn’t work. The DHCP, router and VMs all located on compute-0. For Ghada, I checked the test cases listed in sanity test. I see that there is no test case to test the connectivity between VMs. Need confirmation from Ada’s team. https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 12, 2019 12:28 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing If you co-locate the VMs on the same compute host (lock the other computes) do things work as expected? I.e. is it only when you are traversing the physical network that things are not pingable? -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Monday, March 11, 2019 at 10:51 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Xie, Cindy" <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Richard, Joseph" <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>, "Winnicki, Chris" <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Jones, Bruce E" <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>, "Qin, Kailun" <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>, "Guo, Ruijing" <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest’s question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada’s team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we’ve requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We’re waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can’t get IP from DHCP and external network can’t reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can’t get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can’t ping each other. The DHCP exists and VMs can’t ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris’s guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don’t believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don’t know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I’m trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I’m responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don’t know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing “system host-unlock compute-0”, it will report “Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address”. But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Perez, Ricardo O" <ricardo.o.perez@intel.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Zhao, Forrest" <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt, The ISO image for OVSDPDK Upgrade Testing is built by Shuicheng. According to him, the image is built based on latest master code on March 6. The build.info is below: (cat /etc/build.info) SW_VERSION="19.01" BUILD_TARGET="Unknown" BUILD_TYPE="Informal" BUILD_ID="n/a" JOB="n/a" BUILD_BY="slin14" BUILD_NUMBER="n/a" BUILD_HOST="" BUILD_DATE="2019-03-06 16:34:21 +0000" BUILD_DIR="/" WRS_SRC_DIR="/localdisk/designer/slin14/starlingx/cgcs-root" WRS_GIT_BRANCH="centos76" CGCS_SRC_DIR="/localdisk/designer/slin14/starlingx/cgcs-root/stx" CGCS_GIT_BRANCH="centos76" Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com>, "Peters, Matt" <Matt.Peters@windriver.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Perez, Ricardo O" <ricardo.o.perez@intel.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Zhao, Forrest" <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Perez, Ricardo O" <ricardo.o.perez@intel.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Zhao, Forrest" <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com>, "Xu, Chenjie" <chenjie.xu@intel.com>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Perez, Ricardo O" <ricardo.o.perez@intel.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, "Zhao, Forrest" <forrest.zhao@intel.com>, Brent Rowsell <Brent.Rowsell@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
So, in order to develop our NUMA test cases, we should configure that topology in our VM’s? How can we do it? Do you have the steps? BR Elio From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 1:19 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Martinez Monroy, Elio Sent: Thursday, March 28, 2019 3:36 AM To: Peters, Matt <Matt.Peters@windriver.com>; Xu, Chenjie <chenjie.xu@intel.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing So, in order to develop our NUMA test cases, we should configure that topology in our VM’s? How can we do it? Do you have the steps? BR Elio From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 1:19 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Ghada, Ricardo and I were working in different systems ( Duplex and Standard Controller 2+2 ) and We were able to reproduced the issues, bugs have been updated Also We have run the Sanity Test on both configurations with no issues Matt, Also We have attached the logs from both configurations Best Regards, JP From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 7:59 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Martinez Monroy, Elio Sent: Thursday, March 28, 2019 3:36 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing So, in order to develop our NUMA test cases, we should configure that topology in our VM’s? How can we do it? Do you have the steps? BR Elio From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 1:19 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi JP, Could you please run following commands on your compute node to check if the problem in our environment is caused by the same cause? sudo bash virsh list virsh dumpxml $vm_num > vm.xml In my environment, following sections can’t be found in vm.xml and this is the root cause of the problem: <numatune> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune> <numa> <cell id='0' cpus='0' memory='524288' unit='KiB' memAccess='shared'/> </numa> I have a workaround for this problem and it may be useful to you. You can find my workaround in the comment #7 in following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Gomez, Juan P Sent: Friday, March 29, 2019 7:01 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Ricardo and I were working in different systems ( Duplex and Standard Controller 2+2 ) and We were able to reproduced the issues, bugs have been updated Also We have run the Sanity Test on both configurations with no issues Matt, Also We have attached the logs from both configurations Best Regards, JP From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 7:59 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Martinez Monroy, Elio Sent: Thursday, March 28, 2019 3:36 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing So, in order to develop our NUMA test cases, we should configure that topology in our VM’s? How can we do it? Do you have the steps? BR Elio From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 1:19 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Chenjie, We have dumped the VM xml file, and as you mentioned we haven’t found the NUMA sections either. So we are going to follow the workaround described in your launchpad and share the results as soon as we have it. Regards -Ricardo From: Xu, Chenjie Sent: Thursday, March 28, 2019 11:32 PM To: Gomez, Juan P <juan.p.gomez@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi JP, Could you please run following commands on your compute node to check if the problem in our environment is caused by the same cause? sudo bash virsh list virsh dumpxml $vm_num > vm.xml In my environment, following sections can’t be found in vm.xml and this is the root cause of the problem: <numatune> <memnode cellid='0' mode='strict' nodeset='0'/> </numatune> <numa> <cell id='0' cpus='0' memory='524288' unit='KiB' memAccess='shared'/> </numa> I have a workaround for this problem and it may be useful to you. You can find my workaround in the comment #7 in following bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 Best Regards, Xu, Chenjie From: Gomez, Juan P Sent: Friday, March 29, 2019 7:01 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Ricardo and I were working in different systems ( Duplex and Standard Controller 2+2 ) and We were able to reproduced the issues, bugs have been updated Also We have run the Sanity Test on both configurations with no issues Matt, Also We have attached the logs from both configurations Best Regards, JP From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 7:59 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Martinez Monroy, Elio Sent: Thursday, March 28, 2019 3:36 AM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing So, in order to develop our NUMA test cases, we should configure that topology in our VM’s? How can we do it? Do you have the steps? BR Elio From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 1:19 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Tuesday, March 26, 2019 11:10 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you clarify this statement? “NUMA sections missing bug still exits”. Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing? Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue. As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load. -Matt From: Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Date: Tuesday, March 26, 2019 at 9:28 AM To: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Hi Ghada, Compute-0 and compute-1 don’t have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1’s server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn’t exist in my BIOS. Only NUMA Optimized exists. An image for “Memory RAS and Performance Configuration” has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don’t have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1’s server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn’t exist in my BIOS. Only NUMA Optimized exists. An image for “Memory RAS and Performance Configuration” has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Hi Matt, Zhipeng is working on "PCI Affinity dependency on Nova NUMA topology". This may relate to BUG "VM can't send packet through vhostuser port due to missing numa settings in domain xml": https://bugs.launchpad.net/starlingx/+bug/1820378 According to him, after cutting over to nova master, the NUMA topology and PCI device info has been removed from nova. Before nova master, StarlingX uses nova stage which has NUMA topology. Some detailed information are listed below: * PCI Affinity dependency on Nova NUMA topology - Zhipeng o This affinity agent could not get both pci_device info and numa info of server from nova. o In nova stage version, we added below two for server. o server["wrs-res:topology"] o server["wrs-res:pci_devices"] o In nova master, no these attributions for server. o For topology, I can see that there is a patch of adding numa topology pending for merge o https://review.openstack.org/#/c/621476 Add server sub-resource topology API o Next step - Zhipeng to investigate alternative implementations that don't have dependencies on Nova. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 11:50 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Gauld, James <James.Gauld@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don't have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1's server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn't exist in my BIOS. Only NUMA Optimized exists. An image for "Memory RAS and Performance Configuration" has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Hello Folks, Thanks to Chris Friesen for point this out, but we believe the issues you are experiencing is due to the requirement for guests to be backed by huge pages to operate with OVS-DPDK vhost-user based ports/interfaces. The master (default) behavior for the latest nova will default to 4K pages for the guest, but this is not compatible with OVS-DPDK. The guests must be configured to use a flavor that has the property hw:mem_page_size=large set. You can follow this link to read more about the requirements on the guests for OVS-DPDK: https://docs.openstack.org/neutron/rocky/admin/config-ovs-dpdk.html “vhost-user requires file descriptor-backed shared memory. Currently, the only way to request this is by requesting large pages. This is why instances spawned on hosts with OVS-DPDK must request large pages”. Hope this helps. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com> Date: Tuesday, April 2, 2019 at 11:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com>, "Liu, ZhipengS" <zhipengs.liu@intel.com>, "He, Yongli" <yongli.he@intel.com> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io>, Ghada Khalil <Ghada.Khalil@windriver.com>, "Zhao, Forrest" <forrest.zhao@intel.com>, Brent Rowsell <Brent.Rowsell@windriver.com>, "Gauld, James" <James.Gauld@windriver.com>, "Le, Huifeng" <huifeng.le@intel.com>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com>, "Perez, Ricardo O" <ricardo.o.perez@intel.com>, "Cabrales, Ada" <ada.cabrales@intel.com>, "Lin, Shuicheng" <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Zhipeng is working on “PCI Affinity dependency on Nova NUMA topology”. This may relate to BUG “VM can't send packet through vhostuser port due to missing numa settings in domain xml”: https://bugs.launchpad.net/starlingx/+bug/1820378 According to him, after cutting over to nova master, the NUMA topology and PCI device info has been removed from nova. Before nova master, StarlingX uses nova stage which has NUMA topology. Some detailed information are listed below: • PCI Affinity dependency on Nova NUMA topology - Zhipeng o This affinity agent could not get both pci_device info and numa info of server from nova. o In nova stage version, we added below two for server. o server["wrs-res:topology"] o server["wrs-res:pci_devices"] o In nova master, no these attributions for server. o For topology, I can see that there is a patch of adding numa topology pending for merge o https://review.openstack.org/#/c/621476 Add server sub-resource topology API o Next step - Zhipeng to investigate alternative implementations that don't have dependencies on Nova. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 11:50 PM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Gauld, James <James.Gauld@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don’t have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1’s server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn’t exist in my BIOS. Only NUMA Optimized exists. An image for “Memory RAS and Performance Configuration” has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Hi team, I have reinstalled StarlingX on my 4 bare metals but meet some problems. As soon as I deploy StarlingX successfully, I will try to add property “hw:mem_page_size=large” to flavor and then create VMs. Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Thursday, April 4, 2019 3:21 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; He, Yongli <yongli.he@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Gauld, James <James.Gauld@windriver.com>; Le, Huifeng <huifeng.le@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Folks, Thanks to Chris Friesen for point this out, but we believe the issues you are experiencing is due to the requirement for guests to be backed by huge pages to operate with OVS-DPDK vhost-user based ports/interfaces. The master (default) behavior for the latest nova will default to 4K pages for the guest, but this is not compatible with OVS-DPDK. The guests must be configured to use a flavor that has the property hw:mem_page_size=large set. You can follow this link to read more about the requirements on the guests for OVS-DPDK: https://docs.openstack.org/neutron/rocky/admin/config-ovs-dpdk.html “vhost-user requires file descriptor-backed shared memory. Currently, the only way to request this is by requesting large pages. This is why instances spawned on hosts with OVS-DPDK must request large pages”. Hope this helps. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Tuesday, April 2, 2019 at 11:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>, "He, Yongli" <yongli.he@intel.com<mailto:yongli.he@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>>, "Gauld, James" <James.Gauld@windriver.com<mailto:James.Gauld@windriver.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Zhipeng is working on “PCI Affinity dependency on Nova NUMA topology”. This may relate to BUG “VM can't send packet through vhostuser port due to missing numa settings in domain xml”: https://bugs.launchpad.net/starlingx/+bug/1820378 According to him, after cutting over to nova master, the NUMA topology and PCI device info has been removed from nova. Before nova master, StarlingX uses nova stage which has NUMA topology. Some detailed information are listed below: • PCI Affinity dependency on Nova NUMA topology - Zhipeng o This affinity agent could not get both pci_device info and numa info of server from nova. o In nova stage version, we added below two for server. o server["wrs-res:topology"] o server["wrs-res:pci_devices"] o In nova master, no these attributions for server. o For topology, I can see that there is a patch of adding numa topology pending for merge o https://review.openstack.org/#/c/621476 Add server sub-resource topology API o Next step - Zhipeng to investigate alternative implementations that don't have dependencies on Nova. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 11:50 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>>; Gauld, James <James.Gauld@windriver.com<mailto:James.Gauld@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don’t have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1’s server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn’t exist in my BIOS. Only NUMA Optimized exists. An image for “Memory RAS and Performance Configuration” has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Hi all, After setting property "hw:mem_page_size=large" to flavor, the newly created VM can get IP from DHCP and ping other VM successfully. And NUMA related sections exist in the domain XML file (new domain XML mem_page_size.xml is attached). My steps are listed in the bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 I think it’s better to modify the installation guide to include how to create VM on different environment (OVS/OVSDPDK). Please let me know your idea. Best Regards Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Thursday, April 4, 2019 3:21 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; He, Yongli <yongli.he@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Gauld, James <James.Gauld@windriver.com>; Le, Huifeng <huifeng.le@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Folks, Thanks to Chris Friesen for point this out, but we believe the issues you are experiencing is due to the requirement for guests to be backed by huge pages to operate with OVS-DPDK vhost-user based ports/interfaces. The master (default) behavior for the latest nova will default to 4K pages for the guest, but this is not compatible with OVS-DPDK. The guests must be configured to use a flavor that has the property hw:mem_page_size=large set. You can follow this link to read more about the requirements on the guests for OVS-DPDK: https://docs.openstack.org/neutron/rocky/admin/config-ovs-dpdk.html “vhost-user requires file descriptor-backed shared memory. Currently, the only way to request this is by requesting large pages. This is why instances spawned on hosts with OVS-DPDK must request large pages”. Hope this helps. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Tuesday, April 2, 2019 at 11:48 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Liu, ZhipengS" <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>, "He, Yongli" <yongli.he@intel.com<mailto:yongli.he@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>>, "Gauld, James" <James.Gauld@windriver.com<mailto:James.Gauld@windriver.com>>, "Le, Huifeng" <huifeng.le@intel.com<mailto:huifeng.le@intel.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Zhipeng is working on “PCI Affinity dependency on Nova NUMA topology”. This may relate to BUG “VM can't send packet through vhostuser port due to missing numa settings in domain xml”: https://bugs.launchpad.net/starlingx/+bug/1820378 According to him, after cutting over to nova master, the NUMA topology and PCI device info has been removed from nova. Before nova master, StarlingX uses nova stage which has NUMA topology. Some detailed information are listed below: • PCI Affinity dependency on Nova NUMA topology - Zhipeng o This affinity agent could not get both pci_device info and numa info of server from nova. o In nova stage version, we added below two for server. o server["wrs-res:topology"] o server["wrs-res:pci_devices"] o In nova master, no these attributions for server. o For topology, I can see that there is a patch of adding numa topology pending for merge o https://review.openstack.org/#/c/621476 Add server sub-resource topology API o Next step - Zhipeng to investigate alternative implementations that don't have dependencies on Nova. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 11:50 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>>; Gauld, James <James.Gauld@windriver.com<mailto:James.Gauld@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don’t have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1’s server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn’t exist in my BIOS. Only NUMA Optimized exists. An image for “Memory RAS and Performance Configuration” has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Thanks Chenjie for confirming. I added the following note in the container installation guide[0]: IMPORTANT: When deploying OVS-DPDK, VMs must be configured to use a flavor with property: hw:mem_page_size=large [0] https://wiki.openstack.org/wiki/StarlingX/Containers/Installation#Configure_... Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, April 08, 2019 3:39 AM To: Peters, Matt; Liu, ZhipengS; He, Yongli; Friesen, Chris Cc: 'starlingx-discuss@lists.starlingx.io'; Khalil, Ghada; Zhao, Forrest; Rowsell, Brent; Gauld, James; Le, Huifeng; Martinez Monroy, Elio; Perez, Ricardo O; Cabrales, Ada; Lin, Shuicheng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, After setting property "hw:mem_page_size=large" to flavor, the newly created VM can get IP from DHCP and ping other VM successfully. And NUMA related sections exist in the domain XML file (new domain XML mem_page_size.xml is attached). My steps are listed in the bug report: https://bugs.launchpad.net/starlingx/+bug/1820378 I think it’s better to modify the installation guide to include how to create VM on different environment (OVS/OVSDPDK). Please let me know your idea. Best Regards Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Thursday, April 4, 2019 3:21 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; He, Yongli <yongli.he@intel.com<mailto:yongli.he@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>>; Gauld, James <James.Gauld@windriver.com<mailto:James.Gauld@windriver.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Folks, Thanks to Chris Friesen for point this out, but we believe the issues you are experiencing is due to the requirement for guests to be backed by huge pages to operate with OVS-DPDK vhost-user based ports/interfaces. The master (default) behavior for the latest nova will default to 4K pages for the guest, but this is not compatible with OVS-DPDK. The guests must be configured to use a flavor that has the property hw:mem_page_size=large set. You can follow this link to read more about the requirements on the guests for OVS-DPDK: https://docs.openstack.org/neutron/rocky/admin/config-ovs-dpdk.html “vhost-user requires file descriptor-backed shared memory. Currently, the only way to request this is by requesting large pages. This is why instances spawned on hosts with OVS-DPDK must request large pages”. Hope this helps. -Matt
Hi Ghada, Attached you can find the output of my compute node. Regards _Richo From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 9:50 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Gauld, James <James.Gauld@windriver.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don’t have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1’s server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn’t exist in my BIOS. Only NUMA Optimized exists. An image for “Memory RAS and Performance Configuration” has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
Hi Ghada, The output of following commands on compute-0 have been attached: sudo lscpu sudo virsh capabilities My controllers are wolf pass servers and have skylake processors. I'm reinstalling the standard 0322 ISO image and this time I will use wolf pass servers as compute node. Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, April 3, 2019 11:48 AM To: Peters, Matt <Matt.Peters@windriver.com>; Liu, ZhipengS <zhipengs.liu@intel.com>; He, Yongli <yongli.he@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; 'Khalil, Ghada' <Ghada.Khalil@windriver.com>; Zhao, Forrest <forrest.zhao@intel.com>; Rowsell, Brent <Brent.Rowsell@windriver.com>; Gauld, James <James.Gauld@windriver.com>; Le, Huifeng <huifeng.le@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Zhipeng is working on "PCI Affinity dependency on Nova NUMA topology". This may relate to BUG "VM can't send packet through vhostuser port due to missing numa settings in domain xml": https://bugs.launchpad.net/starlingx/+bug/1820378 According to him, after cutting over to nova master, the NUMA topology and PCI device info has been removed from nova. Before nova master, StarlingX uses nova stage which has NUMA topology. Some detailed information are listed below: * PCI Affinity dependency on Nova NUMA topology - Zhipeng o This affinity agent could not get both pci_device info and numa info of server from nova. o In nova stage version, we added below two for server. o server["wrs-res:topology"] o server["wrs-res:pci_devices"] o In nova master, no these attributions for server. o For topology, I can see that there is a patch of adding numa topology pending for merge o https://review.openstack.org/#/c/621476 Add server sub-resource topology API o Next step - Zhipeng to investigate alternative implementations that don't have dependencies on Nova. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 11:50 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>>; Gauld, James <James.Gauld@windriver.com<mailto:James.Gauld@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, April 02, 2019 2:55 AM To: Khalil, Ghada; Martinez Monroy, Elio; Peters, Matt; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, Compute-0 and compute-1 don't have skylake processors. The processors both are: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz Compute-0 and compute-1's server model both are: Manufacturer: Intel Corporation Product Name: S2600JF Sub-NUMA Clustering doesn't exist in my BIOS. Only NUMA Optimized exists. An image for "Memory RAS and Performance Configuration" has been attached. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, April 2, 2019 6:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, Can you let us know the model of your server? Does it have skylake processors? Can you also double-check that you have Sub-NUMA Clustering disabled in the BIOS? On a wolfpass, the BIOS setting is at: Enter Setup
Advanced Memory Configuration Memory RAS and Performance Configuration Sub-NUMA Cluster <Disabled>
Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Wednesday, March 27, 2019 9:59 PM To: Martinez Monroy, Elio; Peters, Matt; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I run the following commands on compute-0. The outputs of commands have been attached. sudo /usr/sbin/dmidecode > dmidecode.txt virsh nodeinfo > nodeinfo.txt /usr/bin/topology > topology.txt grep -i numa /var/log/dmesg > dmesg.txt And the StarlingX is installed with standard 0322 ISO image: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie
On 4/2/2019 9:50 AM, Khalil, Ghada wrote:
Thanks Chenjie.
Can you please run the following on your compute nodes and attach the output?
sudo lscpu sudo virsh capabilities
Richardo / Juan P,
Please provide the cpu models and the above output from your two hardware systems as well.
In addition to the above, can you please provide the logs from one of the nova-compute pods on an affected system? You can get the logs by running : POD=`kubectl -n openstack get pod -l application=nova,component=compute \ -o=jsonpath='{.items[0].metadata.name'}` kubectl -n openstack logs -c nova-compute $POD Thanks, Chris
Hi Chris, Lastnight I have sent out the lspci and virsh capabilities, but seems like the moderator it’s still moderating my last e-mail. So please find attached my output again, including your latest command. Please let me know if anything else is required. Regards -Richo From: Chris Friesen [mailto:chris.friesen@windriver.com] Sent: Wednesday, April 3, 2019 9:54 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing On 4/2/2019 9:50 AM, Khalil, Ghada wrote: Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. In addition to the above, can you please provide the logs from one of the nova-compute pods on an affected system? You can get the logs by running : POD=`kubectl -n openstack get pod -l application=nova,component=compute \ -o=jsonpath='{.items[0].metadata.name'}` kubectl -n openstack logs -c nova-compute $POD Thanks, Chris
Hi Chris, I have reinstalled StarlingX on the 4 bare metals but meet some problems. So I can’t execute your commands now. As soon as I deploy StarlingX successfully, I will execute those command and let you know the result. Best Regards, Xu, Chenjie From: Chris Friesen [mailto:chris.friesen@windriver.com] Sent: Wednesday, April 3, 2019 11:54 PM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing On 4/2/2019 9:50 AM, Khalil, Ghada wrote: Thanks Chenjie. Can you please run the following on your compute nodes and attach the output? sudo lscpu sudo virsh capabilities Richardo / Juan P, Please provide the cpu models and the above output from your two hardware systems as well. In addition to the above, can you please provide the logs from one of the nova-compute pods on an affected system? You can get the logs by running : POD=`kubectl -n openstack get pod -l application=nova,component=compute \ -o=jsonpath='{.items[0].metadata.name'}` kubectl -n openstack logs -c nova-compute $POD Thanks, Chris
Given that we think the networking issue is likely due to using 4KB pages instead of hugepages, it would probably make sense to try specifying hugepages first. If that doesn't work, then we can start looking at the nova logs. Chris On 4/4/2019 8:16 AM, Xu, Chenjie wrote:
Hi Chris,
I have reinstalled StarlingX on the 4 bare metals but meet some problems. So I can’t execute your commands now. As soon as I deploy StarlingX successfully, I will execute those command and let you know the result.
Best Regards,
Xu, Chenjie
*From:*Chris Friesen [mailto:chris.friesen@windriver.com] *Sent:* Wednesday, April 3, 2019 11:54 PM *To:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing
On 4/2/2019 9:50 AM, Khalil, Ghada wrote:
Thanks Chenjie.
Can you please run the following on your compute nodes and attach the output?
sudo lscpu sudo virsh capabilities
Richardo / Juan P,
Please provide the cpu models and the above output from your two hardware systems as well.
In addition to the above, can you please provide the logs from one of the nova-compute pods on an affected system?
You can get the logs by running :
POD=`kubectl -n openstack get pod -l application=nova,component=compute \
-o=jsonpath='{.items[0].metadata.name'}`
kubectl -n openstack logs -c nova-compute $POD
Thanks,
Chris
Hi Elio, ovs-dpdk testing needs to be done on hardware. Virtual environments will be using ovs only moving forward. The code required to allow this was merged earlier today. More details will be sent shortly. https://review.openstack.org/#/c/633924/ Using OVS in a virtual env is required because master neutron doesn’t support ovs-dpdk in virtual environments. We have been carrying an stx specific patch to allow users to continue working on virtual env. But with the above code now merging, this patch will be discarded once the above feature is sufficiently tested. Hope this clarifies things. Thanks, Ghada From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Wednesday, March 27, 2019 3:36 PM To: Peters, Matt; Xu, Chenjie; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Rowsell, Brent Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing So, in order to develop our NUMA test cases, we should configure that topology in our VM’s? How can we do it? Do you have the steps? BR Elio From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 1:19 PM To: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello, From the output below, this data is taken from a virtual system. In a QEMU/KVM environment, it is not expected to have a NUMA topology unless it has been explicitly configured in the domain configuration of the emulated host. I’m assuming the tests being performed by Chenjie are in a HW lab. This can be confirmed by providing the same set of data for those test systems. Thanks, Matt From: "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Date: Wednesday, March 27, 2019 at 2:47 PM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>, Brent Rowsell <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed sudo /usr/sbin/dmdecode Password: sudo: /usr/sbin/dmdecode: command not found compute-0:~$ sudo /usr/sbin/dmidecode # dmidecode 3.1 Getting SMBIOS data from sysfs. SMBIOS 2.8 present. 10 structures occupying 419 bytes. Table at 0x000F6460. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: SeaBIOS Version: Ubuntu-1.8.2-1ubuntu1 Release Date: 04/01/2014 Address: 0xE8000 Runtime Size: 96 kB ROM Size: 64 kB Characteristics: BIOS characteristics not supported Targeted content distribution is supported BIOS Revision: 0.0 Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: QEMU Product Name: Standard PC (i440FX + PIIX, 1996) Version: pc-i440fx-2.5 Serial Number: Not Specified UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824 Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified Handle 0x0300, DMI type 3, 21 bytes Chassis Information Manufacturer: QEMU Type: Other Lock: Not Present Version: pc-i440fx-2.5 Serial Number: Not Specified Asset Tag: Not Specified Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: Unknown OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: Unspecified Contained Elements: 0 Handle 0x0400, DMI type 4, 42 bytes Processor Information Socket Designation: CPU 0 Type: Central Processor Family: Other Manufacturer: QEMU ID: A3 06 01 00 FF FB 8B 07 Version: pc-i440fx-2.5 Voltage: Unknown External Clock: Unknown Max Speed: 2000 MHz Current Speed: 2000 MHz Status: Populated, Enabled Upgrade: Other L1 Cache Handle: Not Provided L2 Cache Handle: Not Provided L3 Cache Handle: Not Provided Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Core Count: 6 Core Enabled: 6 Thread Count: 1 Characteristics: None Handle 0x1000, DMI type 16, 23 bytes Physical Memory Array Location: Other Use: System Memory Error Correction Type: Multi-bit ECC Maximum Capacity: 10 GB Error Information Handle: Not Provided Number Of Devices: 1 Handle 0x1100, DMI type 17, 40 bytes Memory Device Array Handle: 0x1000 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: 10240 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: Not Specified Type: RAM Type Detail: Other Speed: Unknown Manufacturer: QEMU Serial Number: Not Specified Asset Tag: Not Specified Part Number: Not Specified Rank: Unknown Configured Clock Speed: Unknown Minimum Voltage: Unknown Maximum Voltage: Unknown Configured Voltage: Unknown Handle 0x1300, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x000BFFFFFFF Range Size: 3 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x1301, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00100000000 Ending Address: 0x002BFFFFFFF Range Size: 7 GB Physical Array Handle: 0x1000 Partition Width: 1 Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected Handle 0x7F00, DMI type 127, 4 bytes End Of Table compute-0:~$ virsh nodeinfo CPU model: x86_64 CPU(s): 6 CPU frequency: 3792 MHz CPU socket(s): 1 Core(s) per socket: 6 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10485236 KiB compute-0:~$ /usr/bin/topology TOPOLOGY: logical cpus : 6 sockets : 1 cores_per_pkg : 6 threads_per_core : 1 numa_nodes : 1 total_memory : 9.61 GiB memory_per_node : 10.00 GiB LOGICAL CPU TOPOLOGY: cpu_id : 0 1 2 3 4 5 socket_id : 0 0 0 0 0 0 core_id : 0 1 2 3 4 5 thread_id : 0 0 0 0 0 0 CORE TOPOLOGY: cpu_id socket_id core_id thread_id affinity 0 0 0 0 0x1 1 0 1 0 0x2 2 0 2 0 0x4 3 0 3 0 0x8 4 0 4 0 0x10 5 0 5 0 0x20 compute-0:~$ grep -i numa /var/log/dmesg [ 0.000000] No NUMA configuration found From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Wednesday, March 27, 2019 6:10 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Rowsell, Brent <Brent.Rowsell@windriver.com<mailto:Brent.Rowsell@windriver.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Can you run the following set of commands on your test system compute hosts and provide the output for each? sudo /usr/sbin/dmidecode virsh nodeinfo /usr/bin/topology grep -i numa /var/log/dmesg From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Wednesday, March 27, 2019 at 5:45 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174... Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Wednesday, March 27, 2019 10:38 AM To: 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, Sorry for misleading! I mean “NUMA sections are still missing”. I have multiple systems experiencing this issue: 1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals 3. 0305 ISO image on 4 bare metals 4. 0315 ISO image one 1 bare metals The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal. I will try 0322 ISO image on 4 bare metals. Best Regards, Xu, Chenjie
Elio confirmed last week that he was able to deploy and configure the ISO, but let me check if he created VMs. Ricardo (Richo) is setting the bare metal config today. VM connectivity on sanity: we started to run the test manually last week. We will continue running it in this way in the meantime the automatic test is done. Ada From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Tuesday, March 26, 2019 7:28 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Perez, Ricardo O <ricardo.o.perez@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare. Ada, As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated? Thanks, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 26, 2019 7:09 AM To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits: https://bugs.launchpad.net/starlingx/+bug/1820378 This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions? Best Regards, Xu, Chenjie From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, March 22, 2019 8:31 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hello Chenjie, Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from? I have attached a sample list from our labs for comparison. -Matt From: "Xu, Chenjie" <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>> Date: Friday, March 22, 2019 at 7:25 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>, Ghada Khalil <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>, "Lin, Shuicheng" <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>, "Cabrales, Ada" <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>, "Perez, Ricardo O" <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: "'starlingx-discuss@lists.starlingx.io'" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Matt, The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image. Best Regards, Xu, Chenjie From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Thursday, March 21, 2019 11:20 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Perez, Ricardo O <ricardo.o.perez@intel.com<mailto:ricardo.o.perez@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing Perfect, I will be cheking my mail for that new ISO. Adding Richo , he is the one that is going to execute the tests. Thanks BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 9:15 AM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi Ghada, The mlx pmd patch is already included in below file: https://review.openstack.org/#/c/642672/2/networking/openvswitch/files/x86_6... # Compile burst-oriented Mellanox ConnectX-3 (MLX4) PMD CONFIG_RTE_LIBRTE_MLX4_PMD=y … # Compile burst-oriented Mellanox ConnectX-4 & ConnectX-5 (MLX5) PMD CONFIG_RTE_LIBRTE_MLX5_PMD=y I will share the test ISO with latest master code with Elio in another mail thread today. Thanks. Best Regards Shuicheng From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Thursday, March 21, 2019 11:15 PM To: Lin, Shuicheng <shuicheng.lin@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>; Zhao, Forrest <forrest.zhao@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, We discussed this in the networking bi-weekly team meeting today. Please re-enable the mlx pmd – re-apply the following patch: https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/ce... Then please rebase to the latest master and build a new load for Ada’s team to start testing Hi Elio, One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates. Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs: https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 Thanks, Ghada From: Khalil, Ghada Sent: Thursday, March 21, 2019 9:12 AM To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Shuicheng, There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well. Can you confirm that the load being used has the mellanox pmd enabled in it? Thanks, Ghada From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Thursday, March 21, 2019 9:03 AM To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada Cc: 'starlingx-discuss@lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Thanks Chenjie for the testing. Hi all, Does it mean greenlight for code merge, or any other test is needed? Here is the patch list: https://review.openstack.org/642672 https://review.openstack.org/642673 Best Regards Shuicheng From: Xu, Chenjie Sent: Thursday, March 21, 2019 5:46 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries: Bug Summary: No bug related to OVSDPDK Upgrade. 3 bugs related to cutting over to container and configuration. https://bugs.launchpad.net/starlingx/+bug/1821150 https://bugs.launchpad.net/starlingx/+bug/1821135 https://bugs.launchpad.net/starlingx/+bug/1820378 The following features have been tested: N/S traffic for FLAT/VLAN network E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts) QOS Security Group MTU Best Regards, Xu, Chenjie
Hi guys, Those sanity results doesn't contain any OVS-DPDK feature testing. We are just starting with our tests this week. Please refer to the following document https://docs.google.com/spreadsheets/d/1zf3U1QSxxfsokWQpvNuO7sVvA5zTP0-OARoR.... Beside that. What I'm thinking is that for our feature testing (manual execution), we can run the sanity set as well. And if you decide we can add some test for ovs in a short future. Please let me know what do you think. Adding Juan Carlos (Sanity testing) and Richo (ovs/dpdk testing) BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, March 11, 2019 8:52 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Elio / JC / Ada, Sanity should include a basic VM connectivity test. Please update the sanity suite to cover this. Thanks, Ghada From: Martinez Monroy, Elio [mailto:elio.martinez.monroy@intel.com] Sent: Monday, March 11, 2019 12:29 PM To: Khalil, Ghada; Xu, Chenjie; Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng; Perez, Ricardo O; Alonso, Juan Carlos Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi guys, Those sanity results doesn't contain any OVS-DPDK feature testing. We are just starting with our tests this week. Please refer to the following document https://docs.google.com/spreadsheets/d/1zf3U1QSxxfsokWQpvNuO7sVvA5zTP0-OARoR.... Beside that. What I'm thinking is that for our feature testing (manual execution), we can run the sanity set as well. And if you decide we can add some test for ovs in a short future. Please let me know what do you think. Adding Juan Carlos (Sanity testing) and Richo (ovs/dpdk testing) BR Elio From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Monday, March 11, 2019 8:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
We have not seen https://bugs.launchpad.net/starlingx/+bug/1819031 issue in other labs (AIO-SX, AIO-DX and Multi-node with storage). We have successful VM connectivity in 20190305T060000Z build. Thanks, Numan. From: Khalil, Ghada <Ghada.Khalil@windriver.com> Sent: March-11-19 10:52 AM To: Xu, Chenjie <chenjie.xu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Hi Bruce/Ghada, We suspect that the some steps are missing in the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard According to Numan, they have verified the VM connectivity in 20190305T060000Z build. Thus the ISO image should be fine. Cindy team provided 2 versions of ISO images and I have tested the first version last week. The first version works correctly and thus hardware of my environment should be fine. Based on above analysis, we suspect that the some steps are missing in the guide. We have been blocked by this issue for several days. If somebody can review the guide and provide some comments, it will be pretty good. Best Regards, Xu, Chenjie From: Waheed, Numan [mailto:Numan.Waheed@windriver.com] Sent: Wednesday, March 13, 2019 11:45 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Xu, Chenjie <chenjie.xu@intel.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Jones, Bruce E <bruce.e.jones@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing We have not seen https://bugs.launchpad.net/starlingx/+bug/1819031 issue in other labs (AIO-SX, AIO-DX and Multi-node with storage). We have successful VM connectivity in 20190305T060000Z build. Thanks, Numan. From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: March-11-19 10:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
Adding Richo, We will try to investigate a little bit deeper inside this problem, he is going to be deploying in Bare metal , while I will try on virtual. In which part is failing for you? BR Elio From: Xu, Chenjie Sent: Thursday, March 14, 2019 10:23 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; Waheed, Numan <Numan.Waheed@windriver.com>; Khalil, Ghada <Ghada.Khalil@windriver.com>; Xie, Cindy <cindy.xie@intel.com>; Zhao, Forrest <forrest.zhao@intel.com>; Peters, Matt <Matt.Peters@windriver.com>; Richard, Joseph <Joseph.Richard@windriver.com>; Winnicki, Chris <Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com>; Peng, Peng <Peng.Peng@windriver.com> Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio <elio.martinez.monroy@intel.com>; Cabrales, Ada <ada.cabrales@intel.com>; Qin, Kailun <kailun.qin@intel.com>; Guo, Ruijing <ruijing.guo@intel.com>; Le, Huifeng <huifeng.le@intel.com> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Bruce/Ghada, We suspect that the some steps are missing in the following guide: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard According to Numan, they have verified the VM connectivity in 20190305T060000Z build. Thus the ISO image should be fine. Cindy team provided 2 versions of ISO images and I have tested the first version last week. The first version works correctly and thus hardware of my environment should be fine. Based on above analysis, we suspect that the some steps are missing in the guide. We have been blocked by this issue for several days. If somebody can review the guide and provide some comments, it will be pretty good. Best Regards, Xu, Chenjie From: Waheed, Numan [mailto:Numan.Waheed@windriver.com] Sent: Wednesday, March 13, 2019 11:45 PM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Peng, Peng <Peng.Peng@windriver.com<mailto:Peng.Peng@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing We have not seen https://bugs.launchpad.net/starlingx/+bug/1819031 issue in other labs (AIO-SX, AIO-DX and Multi-node with storage). We have successful VM connectivity in 20190305T060000Z build. Thanks, Numan. From: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>> Sent: March-11-19 10:52 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
To answer Forrest's question: Do you know if OVS/DPDK has been tested after cut-over to container? I have seen Green sanity reports from Ada's team since the cut-over. I expect this covers basic ovs-dpdk functional testing. Ada, please correct me if I am wrong. Regarding https://bugs.launchpad.net/starlingx/+bug/1819031, we've requested information from the reporter on how reproducible this issue is. As far as I know, it was reported on one system only. We're waiting for more information from Peng. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Monday, March 11, 2019 10:40 AM To: Xie, Cindy; Zhao, Forrest; Peters, Matt; Richard, Joseph; Khalil, Ghada; Winnicki, Chris; Lin, Shuicheng Cc: starlingx-discuss@lists.starlingx.io; Martinez Monroy, Elio; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Cindy, Thank you for your information! The following issue is not exactly the same as mine: https://bugs.launchpad.net/starlingx/+bug/1819031 In this issue, the DHCP works and external network can reach the VMs. However in my case, VMs can't get IP from DHCP and external network can't reach VMs even with an IP allocated by myself. Best Regards, Xu, Chenjie From: Xie, Cindy Sent: Monday, March 11, 2019 10:05 PM To: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Is the below issue similar? https://bugs.launchpad.net/starlingx/+bug/1819031 From: Zhao, Forrest Sent: Monday, March 11, 2019 9:54 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Ghada, Matt, Joseph and validation team, Do you know if OVS/DPDK has been tested after cut-over to container? I think the below issue reported by Chenjie might already exist after cut-over to container. Thanks, Forrest From: Xu, Chenjie Sent: Monday, March 11, 2019 9:17 PM To: Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi team, I have installed StarlingX multimode with new ISO image on 4 bare metals (2 controllers and 2 computes). However the VMs can't get an IP address. I allocate the IP address which is same as neutron allocated to the VM, but the VMs still can't ping each other. The DHCP exists and VMs can't ping DHCP or router. Some logs have been attached. This issue blocks basic functional testing for OVS-DPDK Upgrade and please let me know if you have any ideas. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 8, 2019 9:49 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi all, I just shared the new test ISO with Chenjie/Elio in another mail thread. It is built from centos76 feature branch with latest master code rebased. Best Regards Shuicheng From: Xu, Chenjie Sent: Wednesday, March 6, 2019 11:07 AM To: Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Ghada, That will be very helpful and many thanks to you and Joseph! Looking forward to Chris's guide. Best Regards, Xu, Chenjie From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com] Sent: Wednesday, March 6, 2019 12:07 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>>; Winnicki, Chris <Chris.Winnicki@windriver.com<mailto:Chris.Winnicki@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Chenjie, I spoke with Joseph. We don't believe you need to make any puppet changes to test qos. Storyboard https://storyboard.openstack.org/#!/story/2003948 (which you already noted below) removed the STX/WR custom QoS extensions, so StarlingX is now aligned with the upstream neutron behavior. I have added Chris Winnicki to this email thread. He recently verified QoS functionality in StarlingX after the above Story was merged. He can provide you with the steps. Chris is away today, but should be able to respond tomorrow. Regards, Ghada From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Tuesday, March 05, 2019 8:54 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, About the configuration of QOS, I don't know which puppet file should be modified to add the following configuration into the file /etc/neutron/plugins/ml2/openvswitch.ini: extensions = qos Could you please tell which puppet file I should modify? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 11:39 PM To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'Ghada.Khalil@windriver.com' <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; 'Peters, Matt' <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: FW: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, For the following bug sent on last Friday: system host-port-list compute-1 'pdevice' I attached the /var/log/sysinv.log of controller-0 and compute-1. According to the logs, the compute-1 is not correctly installed. Could you please take a look at the logs and provide some comments? Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Monday, March 4, 2019 10:08 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, Thank you for your reply! For now I'm trying to configure QOS in StarlingX by changing puppet to change neutron configuration files. But I think this is not a good way to configure QOS in StarlingX. Could you please provide some guide on how to configure QOS in StarlingX? I noticed that the StarlingX custom QOS extension has been removed. (https://storyboard.openstack.org/#!/story/2003948) Best Regards, Xu, Chenjie From: Richard, Joseph [mailto:Joseph.Richard@windriver.com] Sent: Saturday, March 2, 2019 12:41 AM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Khalil, Ghada <Ghada.Khalil@windriver.com<mailto:Ghada.Khalil@windriver.com>>; Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing 1: You need to use sysinv to add the address, not add it using ip commands. usage: system host-addr-add <hostname or id> <interface name or id> <ipv4/ipv6 address> <prefix length> 2: Check in /var/log/sysinv.log for the traceback at that time. From: Xu, Chenjie [mailto:chenjie.xu@intel.com] Sent: Friday, March 1, 2019 9:55 AM To: Richard, Joseph; Khalil, Ghada; Peters, Matt Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Lin, Shuicheng; Martinez Monroy, Elio; Zhao, Forrest; Xie, Cindy; Cabrales, Ada; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng Subject: [Starlingx-discuss] OVS-DPDK Upgrade Testing Hi Joseph, I'm responsible for the Basic Functional Testing for OVS-DPDK Upgrade. I have set up a StarlingX multimode environment (1 controller and 2 computes) using the ISO image for OVS/DPDK upgrade test. I have 2 questions for you: 1. I don't know how associate a vxlan providernet with compute node. Could you please provide the steps how to associate a vxlan providernet with compute node? The commands I use are following: neutron providernet-create tenant_vxlan --type=vxlan --mtu=1400 neutron providernet-range-create --name tenant_vxlan-a-range1 --range 100-400 --group 224.0.0.1 --ttl 255 tenant_vxlan system datanetwork-add tenant_vxlan vxlan --multicast_group 224.0.0.1 --ttl 255 --port_num 4789 system host-if-modify -p tenant_vxlan -c data compute-0 enp4s0f1 system host-unlock compute-0 After executing "system host-unlock compute-0", it will report "Interface enp4s0f1 is associated to VXLAN data network tenant_vxlan which requires an IPv4 address". But I do allocate a IP address for enp4s0f1. You can check the attached images. 2. I have found one bug and will investigate it. If you have any ideas please let me know. After installing the compute-1, I try to list the interface on compute-1 by following command: system host-port-list compute-1 'pdevice' The normal output should be: system host-port-list compute-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ | e072d576-90a6-4040-87df-66a913076df0 | eno1 | ethernet | 0000:07:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | 485b6e35-5b75-4b73-93dc-7deae0fb5c5c | eno2 | ethernet | 0000:07:00.1 | 0 | 0 | True | I350 Gigabit Network Connection | | 8b448ab6-e9df-4ce7-84a4-256d97f6fd5b | enp4s0f0 | ethernet | 0000:04:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | | 6d5b1ce7-ae76-4781-9e51-93bfe8417dc0 | enp4s0f1 | ethernet | 0000:04:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+------------------------------------------------+ Best Regards, Xu, Chenjie From: Xu, Chenjie Sent: Friday, March 1, 2019 2:32 PM To: Lin, Shuicheng <shuicheng.lin@intel.com<mailto:shuicheng.lin@intel.com>> Cc: Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>>; Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>>; Le, Huifeng <huifeng.le@intel.com<mailto:huifeng.le@intel.com>> Subject: RE: ISO image for OVS/DPDK upgrade Hi Shuicheng, Thank you for your information! I will start to do the Basic Functional Test. Best Regards, Xu, Chenjie From: Lin, Shuicheng Sent: Friday, March 1, 2019 2:03 PM To: Xu, Chenjie <chenjie.xu@intel.com<mailto:chenjie.xu@intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy@intel.com<mailto:elio.martinez.monroy@intel.com>> Cc: Zhao, Forrest <forrest.zhao@intel.com<mailto:forrest.zhao@intel.com>>; Xie, Cindy <cindy.xie@intel.com<mailto:cindy.xie@intel.com>>; Cabrales, Ada <ada.cabrales@intel.com<mailto:ada.cabrales@intel.com>>; Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; Qin, Kailun <kailun.qin@intel.com<mailto:kailun.qin@intel.com>>; Guo, Ruijing <ruijing.guo@intel.com<mailto:ruijing.guo@intel.com>> Subject: ISO image for OVS/DPDK upgrade Hi Chenjie/Elio, Here is the ISO image for OVS/DPDK upgrade test. I have done basic deploy test, and could ping VM from each other successfully. Please help have a try with it, and notify me if there is any issue. It is built with recent code of centos76 feature branch. ec@10.239.154.150:/home/ec/bootimage_ovs.iso<mailto:ec@10.239.154.150:/home/ec/bootimage_ovs.iso> scp passwd: 1 Here is the patch list for the upgrade in case you want to have a check: https://review.openstack.org/640244 https://review.openstack.org/640245 Best Regards Shuicheng
participants (13)
-
Cabrales, Ada
-
Chris Friesen
-
Gomez, Juan P
-
Khalil, Ghada
-
Lin, Shuicheng
-
Liu, ZhipengS
-
Martinez Monroy, Elio
-
Perez, Ricardo O
-
Peters, Matt
-
Waheed, Numan
-
Xie, Cindy
-
Xu, Chenjie
-
Zhao, Forrest