[Starlingx-discuss] OVS-DPDK Upgrade Testing
Martinez Monroy, Elio
elio.martinez.monroy at intel.com
Wed Mar 27 18:47:13 UTC 2019
Hi Matt
Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed
sudo /usr/sbin/dmdecode
Password:
sudo: /usr/sbin/dmdecode: command not found
compute-0:~$ sudo /usr/sbin/dmidecode
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 419 bytes.
Table at 0x000F6460.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: Ubuntu-1.8.2-1ubuntu1
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
Characteristics:
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: QEMU
Product Name: Standard PC (i440FX + PIIX, 1996)
Version: pc-i440fx-2.5
Serial Number: Not Specified
UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824
Wake-up Type: Power Switch
SKU Number: Not Specified
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.5
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A3 06 01 00 FF FB 8B 07
Version: pc-i440fx-2.5
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 6
Core Enabled: 6
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 10 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 10240 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x002BFFFFFFF
Range Size: 7 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table
compute-0:~$ virsh nodeinfo
CPU model: x86_64
CPU(s): 6
CPU frequency: 3792 MHz
CPU socket(s): 1
Core(s) per socket: 6
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 10485236 KiB
compute-0:~$ /usr/bin/topology
TOPOLOGY:
logical cpus : 6
sockets : 1
cores_per_pkg : 6
threads_per_core : 1
numa_nodes : 1
total_memory : 9.61 GiB
memory_per_node : 10.00 GiB
LOGICAL CPU TOPOLOGY:
cpu_id : 0 1 2 3 4 5
socket_id : 0 0 0 0 0 0
core_id : 0 1 2 3 4 5
thread_id : 0 0 0 0 0 0
CORE TOPOLOGY:
cpu_id socket_id core_id thread_id affinity
0 0 0 0 0x1
1 0 1 0 0x2
2 0 2 0 0x4
3 0 3 0 0x8
4 0 4 0 0x10
5 0 5 0 0x20
compute-0:~$ grep -i numa /var/log/dmesg
[ 0.000000] No NUMA configuration found
From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Wednesday, March 27, 2019 6:10 AM
To: Xu, Chenjie <chenjie.xu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Lin, Shuicheng <shuicheng.lin at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Perez, Ricardo O <ricardo.o.perez at intel.com>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>; Zhao, Forrest <forrest.zhao at intel.com>; Rowsell, Brent <Brent.Rowsell at windriver.com>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hello Chenjie,
Can you run the following set of commands on your test system compute hosts and provide the output for each?
sudo /usr/sbin/dmidecode
virsh nodeinfo
/usr/bin/topology
grep -i numa /var/log/dmesg
From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Wednesday, March 27, 2019 at 5:45 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi Matt,
I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174230Z/outputs/iso/
Best Regards,
Xu, Chenjie
From: Xu, Chenjie
Sent: Wednesday, March 27, 2019 10:38 AM
To: 'Peters, Matt' <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi Matt,
Sorry for misleading! I mean “NUMA sections are still missing”.
I have multiple systems experiencing this issue:
1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals
2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals
3. 0305 ISO image on 4 bare metals
4. 0315 ISO image one 1 bare metals
The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal.
I will try 0322 ISO image on 4 bare metals.
Best Regards,
Xu, Chenjie
From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Tuesday, March 26, 2019 11:10 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hello Chenjie,
Can you clarify this statement? “NUMA sections missing bug still exits”.
Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing?
Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology? I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue.
As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load.
-Matt
From: Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Tuesday, March 26, 2019 at 9:28 AM
To: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>, "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare.
Ada,
As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated?
Thanks,
Ghada
From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Tuesday, March 26, 2019 7:09 AM
To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi Matt,
I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits:
https://bugs.launchpad.net/starlingx/+bug/1820378
This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions?
Best Regards,
Xu, Chenjie
From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Friday, March 22, 2019 8:31 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hello Chenjie,
Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images. Since some of them align, I think you might be running a different set of images. Can you confirm what the build baseline is that you pulled the images from?
I have attached a sample list from our labs for comparison.
-Matt
From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Friday, March 22, 2019 at 7:25 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi Matt,
The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image.
Best Regards,
Xu, Chenjie
From: Martinez Monroy, Elio [mailto:elio.martinez.monroy at intel.com]
Sent: Thursday, March 21, 2019 11:20 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Perfect, I will be cheking my mail for that new ISO.
Adding Richo , he is the one that is going to execute the tests.
Thanks
BR
Elio
From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Thursday, March 21, 2019 9:15 AM
To: Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi Shuicheng,
We discussed this in the networking bi-weekly team meeting today.
Please re-enable the mlx pmd – re-apply the following patch:
https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/centos/meta_patches/0005-enable-mlx-pmds.patch
Then please rebase to the latest master and build a new load for Ada’s team to start testing
Hi Elio,
One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates.
Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade). Please note that none of these issues are reproducible in WR labs:
https://bugs.launchpad.net/starlingx/+bug/1821150
https://bugs.launchpad.net/starlingx/+bug/1821135
https://bugs.launchpad.net/starlingx/+bug/1820378
Thanks,
Ghada
From: Khalil, Ghada
Sent: Thursday, March 21, 2019 9:12 AM
To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi Shuicheng,
There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well.
Can you confirm that the load being used has the mellanox pmd enabled in it?
Thanks,
Ghada
From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
Sent: Thursday, March 21, 2019 9:03 AM
To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Thanks Chenjie for the testing.
Hi all,
Does it mean greenlight for code merge, or any other test is needed?
Here is the patch list:
https://review.openstack.org/642672
https://review.openstack.org/642673
Best Regards
Shuicheng
From: Xu, Chenjie
Sent: Thursday, March 21, 2019 5:46 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Peng, Peng <Peng.Peng at windriver.com<mailto:Peng.Peng at windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing
Hi team,
I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries:
Bug Summary:
No bug related to OVSDPDK Upgrade.
3 bugs related to cutting over to container and configuration.
https://bugs.launchpad.net/starlingx/+bug/1821150
https://bugs.launchpad.net/starlingx/+bug/1821135
https://bugs.launchpad.net/starlingx/+bug/1820378
The following features have been tested:
N/S traffic for FLAT/VLAN network
E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts)
QOS
Security Group
MTU
Best Regards,
Xu, Chenjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/8e562cfc/attachment-0001.html>
More information about the Starlingx-discuss
mailing list