[Starlingx-discuss] OVS-DPDK Upgrade Testing

Martinez Monroy, Elio elio.martinez.monroy at intel.com
Wed Mar 27 18:47:13 UTC 2019


Hi Matt

Just exercising same stuff, please check my outputs from compute-0, please let me know if something else is needed

sudo /usr/sbin/dmdecode
Password:
sudo: /usr/sbin/dmdecode: command not found
compute-0:~$ sudo /usr/sbin/dmidecode
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 419 bytes.
Table at 0x000F6460.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: SeaBIOS
        Version: Ubuntu-1.8.2-1ubuntu1
        Release Date: 04/01/2014
        Address: 0xE8000
        Runtime Size: 96 kB
        ROM Size: 64 kB
        Characteristics:
                BIOS characteristics not supported
                Targeted content distribution is supported
        BIOS Revision: 0.0

Handle 0x0100, DMI type 1, 27 bytes
System Information
        Manufacturer: QEMU
        Product Name: Standard PC (i440FX + PIIX, 1996)
        Version: pc-i440fx-2.5
        Serial Number: Not Specified
        UUID: 71d54a5d-b569-4e1c-83e3-9babf169e824
        Wake-up Type: Power Switch
        SKU Number: Not Specified
        Family: Not Specified

Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
        Manufacturer: QEMU
        Type: Other
        Lock: Not Present
        Version: pc-i440fx-2.5
        Serial Number: Not Specified
        Asset Tag: Not Specified
        Boot-up State: Safe
        Power Supply State: Safe
        Thermal State: Safe
        Security Status: Unknown
        OEM Information: 0x00000000
        Height: Unspecified
        Number Of Power Cords: Unspecified
        Contained Elements: 0

Handle 0x0400, DMI type 4, 42 bytes
Processor Information
        Socket Designation: CPU 0
        Type: Central Processor
        Family: Other
        Manufacturer: QEMU
        ID: A3 06 01 00 FF FB 8B 07
        Version: pc-i440fx-2.5
        Voltage: Unknown
        External Clock: Unknown
        Max Speed: 2000 MHz
        Current Speed: 2000 MHz
        Status: Populated, Enabled
        Upgrade: Other
        L1 Cache Handle: Not Provided
        L2 Cache Handle: Not Provided
        L3 Cache Handle: Not Provided
        Serial Number: Not Specified
        Asset Tag: Not Specified
        Part Number: Not Specified
        Core Count: 6
        Core Enabled: 6
        Thread Count: 1
        Characteristics: None

Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
        Location: Other
        Use: System Memory
        Error Correction Type: Multi-bit ECC
        Maximum Capacity: 10 GB
        Error Information Handle: Not Provided
        Number Of Devices: 1

Handle 0x1100, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x1000
        Error Information Handle: Not Provided
        Total Width: Unknown
        Data Width: Unknown
        Size: 10240 MB
        Form Factor: DIMM
        Set: None
        Locator: DIMM 0
        Bank Locator: Not Specified
        Type: RAM
        Type Detail: Other
        Speed: Unknown
        Manufacturer: QEMU
        Serial Number: Not Specified
        Asset Tag: Not Specified
        Part Number: Not Specified
        Rank: Unknown
        Configured Clock Speed: Unknown
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
        Starting Address: 0x00000000000
        Ending Address: 0x000BFFFFFFF
        Range Size: 3 GB
        Physical Array Handle: 0x1000
        Partition Width: 1

Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
        Starting Address: 0x00100000000
        Ending Address: 0x002BFFFFFFF
        Range Size: 7 GB
        Physical Array Handle: 0x1000
        Partition Width: 1

Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
        Status: No errors detected

Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

compute-0:~$ virsh nodeinfo
CPU model:           x86_64
CPU(s):              6
CPU frequency:       3792 MHz
CPU socket(s):       1
Core(s) per socket:  6
Thread(s) per core:  1
NUMA cell(s):        1
Memory size:         10485236 KiB

compute-0:~$ /usr/bin/topology
TOPOLOGY:
    logical cpus :     6
         sockets :     1
   cores_per_pkg :     6
threads_per_core :     1
      numa_nodes :     1
    total_memory :  9.61 GiB
memory_per_node : 10.00 GiB

LOGICAL CPU TOPOLOGY:
   cpu_id :   0   1   2   3   4   5
socket_id :   0   0   0   0   0   0
  core_id :   0   1   2   3   4   5
thread_id :   0   0   0   0   0   0

CORE TOPOLOGY:
cpu_id socket_id core_id thread_id affinity
     0         0       0         0 0x1
     1         0       1         0 0x2
     2         0       2         0 0x4
     3         0       3         0 0x8
     4         0       4         0 0x10
     5         0       5         0 0x20
compute-0:~$ grep -i numa /var/log/dmesg
[    0.000000] No NUMA configuration found

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Wednesday, March 27, 2019 6:10 AM
To: Xu, Chenjie <chenjie.xu at intel.com>; Khalil, Ghada <Ghada.Khalil at windriver.com>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com>; Lin, Shuicheng <shuicheng.lin at intel.com>; Cabrales, Ada <ada.cabrales at intel.com>; Perez, Ricardo O <ricardo.o.perez at intel.com>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>; Zhao, Forrest <forrest.zhao at intel.com>; Rowsell, Brent <Brent.Rowsell at windriver.com>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hello Chenjie,
Can you run the following set of commands on your test system compute hosts and provide the output for each?

sudo /usr/sbin/dmidecode
virsh nodeinfo
/usr/bin/topology
grep -i numa /var/log/dmesg

From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Wednesday, March 27, 2019 at 5:45 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
I tried standard 0322 ISO image on 4 bare metals and “NUMA sections are still missing”. The docker images versions and build baseline have been attached. The ISO image link is following:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190322T174230Z/outputs/iso/

Best Regards,
Xu, Chenjie

From: Xu, Chenjie
Sent: Wednesday, March 27, 2019 10:38 AM
To: 'Peters, Matt' <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
Sorry for misleading! I mean “NUMA sections are still missing”.

I have multiple systems experiencing this issue:
1. 0306 ISO image for OVSDPDK Upgrade Testing on 4 bare metals
2. 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals
3. 0305 ISO image on 4 bare metals
4. 0315 ISO image one 1 bare metals
The item 1, 2 and 3 use same bare metals. The item 4 use a separate bare metal.

I will try 0322 ISO image on 4 bare metals.

Best Regards,
Xu, Chenjie

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Tuesday, March 26, 2019 11:10 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hello Chenjie,
Can you clarify this statement? “NUMA sections missing bug still exits”.
Did you mean to say that the missing NUMA sections in the domain XML are now present, but you still can’t ping over the vhost port, or are you saying the NUMA sections are still missing?

Do you have multiple systems that are experiencing this issue, or is it isolated to a single system that you are having problems with the NUMA topology?  I believe you mentioned that you saw the same connectivity issues on a standard load from the mirror, so that still points to a test system / environment issue.

As Ghada mentioned, I think the next steps are to get results from the sanity systems with the same load baseline and to test with your custom load on a system that has been proven to work on the standard load.

-Matt

From: Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Tuesday, March 26, 2019 at 9:28 AM
To: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>, "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Has Elio or Rich from the test team tried the new load as well on their systems? Do they see the same issues you reported? I think we need some data for another system to compare.

Ada,
As part of this thread, I also requested that a VM connectivity test be added to regular sanity so that there is data from the daily cengn builds w/o the new version of ovs-dpdk. Has the sanity suite been updated?

Thanks,
Ghada

From: Xu, Chenjie [mailto:chenjie.xu at intel.com]
Sent: Tuesday, March 26, 2019 7:09 AM
To: Peters, Matt; Martinez Monroy, Elio; Khalil, Ghada; Lin, Shuicheng; Cabrales, Ada; Perez, Ricardo O
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
I installed StarlingX using the new 0322 ISO image for OVSDPDK Upgrade Testing on 4 bare metals. NUMA sections missing bug still exits:
https://bugs.launchpad.net/starlingx/+bug/1820378

This time I update the docker images to latest before installing StarlingX. And the docker images versions and build baseline have been attached. Could you please help review the docker image versions?

Best Regards,
Xu, Chenjie

From: Peters, Matt [mailto:Matt.Peters at windriver.com]
Sent: Friday, March 22, 2019 8:31 PM
To: Xu, Chenjie <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hello Chenjie,
Comparing the image list to our labs shows a difference in the image IDs (sha), but I’m not sure if that is just a difference in build baselines or an actual difference in the images.  Since some of them align, I think you might be running a different set of images.  Can you confirm what the build baseline is that you pulled the images from?

I have attached a sample list from our labs for comparison.

-Matt

From: "Xu, Chenjie" <chenjie.xu at intel.com<mailto:chenjie.xu at intel.com>>
Date: Friday, March 22, 2019 at 7:25 AM
To: "Peters, Matt" <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>, "Martinez Monroy, Elio" <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>, Ghada Khalil <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>, "Lin, Shuicheng" <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>, "Cabrales, Ada" <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>, "Perez, Ricardo O" <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: "'starlingx-discuss at lists.starlingx.io'" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>, "Zhao, Forrest" <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Matt,
The docker image versions in my environment and helm-charts-manifest-no-tests.tgz has been attached. The helm-charts-manifest-no-tests.tgz is built by Shuicheng for the OVSDPDK Upgrade Testing ISO image.

Best Regards,
Xu, Chenjie

From: Martinez Monroy, Elio [mailto:elio.martinez.monroy at intel.com]
Sent: Thursday, March 21, 2019 11:20 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Perez, Ricardo O <ricardo.o.perez at intel.com<mailto:ricardo.o.perez at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: Re: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Perfect, I will be cheking my mail  for that new ISO.

Adding Richo , he is the one that is going to execute the tests.

Thanks

BR

Elio

From: Khalil, Ghada [mailto:Ghada.Khalil at windriver.com]
Sent: Thursday, March 21, 2019 9:15 AM
To: Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Shuicheng,
We discussed this in the networking bi-weekly team meeting today.

Please re-enable the mlx pmd – re-apply the following patch:
https://github.com/openstack/stx-integ/blob/master/networking/openvswitch/centos/meta_patches/0005-enable-mlx-pmds.patch

Then please rebase to the latest master and build a new load for Ada’s team to start testing

Hi Elio,
One a new build is available, please start testing as per our previous plans. You need to run a networking regression on various NICs – including the mellanox. Testing was planned for about two weeks with the target for the code to merge after testing is complete by April 5. Please provide regular updates.

Please also verify if you see the 3 bugs that Chenjie reports below (please test on both a cengn load as well as the load with the ovs-dpdk upgrade).  Please note that none of these issues are reproducible in WR labs:
https://bugs.launchpad.net/starlingx/+bug/1821150
https://bugs.launchpad.net/starlingx/+bug/1821135
https://bugs.launchpad.net/starlingx/+bug/1820378

Thanks,
Ghada

From: Khalil, Ghada
Sent: Thursday, March 21, 2019 9:12 AM
To: 'Lin, Shuicheng'; Xu, Chenjie; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi Shuicheng,
There is still testing planned by Ada’s team before merging the code related to networking regression. This will cover the mellanox NIC as well.

Can you confirm that the load being used has the mellanox pmd enabled in it?

Thanks,
Ghada

From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
Sent: Thursday, March 21, 2019 9:03 AM
To: Xu, Chenjie; Khalil, Ghada; Liu, ZhipengS; Xie, Cindy; Cabrales, Ada
Cc: 'starlingx-discuss at lists.starlingx.io'; Zhao, Forrest; Peters, Matt; Richard, Joseph; Winnicki, Chris; Peng, Peng; Martinez Monroy, Elio; Jones, Bruce E; Qin, Kailun; Guo, Ruijing; Le, Huifeng
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Thanks Chenjie for the testing.

Hi all,
Does it mean greenlight for code merge, or any other test is needed?
Here is the patch list:
https://review.openstack.org/642672
https://review.openstack.org/642673


Best Regards
Shuicheng

From: Xu, Chenjie
Sent: Thursday, March 21, 2019 5:46 PM
To: Khalil, Ghada <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>; Liu, ZhipengS <zhipengs.liu at intel.com<mailto:zhipengs.liu at intel.com>>; Xie, Cindy <cindy.xie at intel.com<mailto:cindy.xie at intel.com>>; Cabrales, Ada <ada.cabrales at intel.com<mailto:ada.cabrales at intel.com>>
Cc: 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>; Zhao, Forrest <forrest.zhao at intel.com<mailto:forrest.zhao at intel.com>>; Peters, Matt <Matt.Peters at windriver.com<mailto:Matt.Peters at windriver.com>>; Richard, Joseph <Joseph.Richard at windriver.com<mailto:Joseph.Richard at windriver.com>>; Winnicki, Chris <Chris.Winnicki at windriver.com<mailto:Chris.Winnicki at windriver.com>>; Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Peng, Peng <Peng.Peng at windriver.com<mailto:Peng.Peng at windriver.com>>; Martinez Monroy, Elio <elio.martinez.monroy at intel.com<mailto:elio.martinez.monroy at intel.com>>; Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; Qin, Kailun <kailun.qin at intel.com<mailto:kailun.qin at intel.com>>; Guo, Ruijing <ruijing.guo at intel.com<mailto:ruijing.guo at intel.com>>; Le, Huifeng <huifeng.le at intel.com<mailto:huifeng.le at intel.com>>
Subject: RE: [Starlingx-discuss] OVS-DPDK Upgrade Testing

Hi team,
I have finished the Basic Functional Testing for OVSDPDK Upgrade. Please check following summaries:

Bug Summary:
No bug related to OVSDPDK Upgrade.
3 bugs related to cutting over to container and configuration.
    https://bugs.launchpad.net/starlingx/+bug/1821150
                   https://bugs.launchpad.net/starlingx/+bug/1821135
                   https://bugs.launchpad.net/starlingx/+bug/1820378

The following features have been tested:
N/S traffic for FLAT/VLAN network
E/W traffic for FLAT/VLAN/VXLAN network (E/W traffic within same host and across hosts)
QOS
Security Group
MTU

Best Regards,
Xu, Chenjie


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/8e562cfc/attachment-0001.html>


More information about the Starlingx-discuss mailing list