Re: [Starlingx-discuss] Unable to bring up controller-0

3 Dec 2018

      Hi Xuyun:
BMC5720  is not in DPDK list [1]. You can find whole list from [2].

[1] http://doc.dpdk.org/guides/nics/bnxt.html
[2] https://core.dpdk.org/supported/

Thanks.
BR
Austin Sun.

From: 徐蕴 [mailto:xuyun@jxresearch.com]
Sent: Tuesday, December 4, 2018 10:23 AM
To: Sun, Austin <austin.sun@intel.com>
Cc: MacDonald, Eric <Eric.MacDonald@windriver.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] Unable to bring up controller-0

Hi Austin,

I’m deploying an all in one node in a bare metal (an old DELL server), maybe the NIC doesn’t support DPDK.
I cannot see eno4 in the output:

controller-0:/var/log/puppet# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:02:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' drv=vfio-pci unused=

Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno1 drv=tg3 unused=vfio-pci *Active*
0000:01:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno2 drv=tg3 unused=vfio-pci
0000:02:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=eno3 drv=tg3 unused=vfio-pci

Other Network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other Crypto devices
====================
<none>

Eventdev devices using DPDK-compatible driver
=============================================
<none>

Eventdev devices using kernel driver
====================================
<none>

Other Eventdev devices
======================
<none>

Mempool devices using DPDK-compatible driver
============================================
<none>

Mempool devices using kernel driver
===================================
<none>

Other Mempool devices
=====================
<none>

controller-0:/var/log/puppet# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.3/24 brd 127.168.204.255 scope host lo
       valid_lft forever preferred_lft forever
    inet 169.254.202.2/24 scope global lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.2/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.5/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.6/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet 127.168.204.8/24 scope host secondary lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b0:83:fe:d6:78:fb brd ff:ff:ff:ff:ff:ff
    inet 111.111.111.1/24 brd 111.111.111.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::b283:feff:fed6:78fb/64 scope link
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether b0:83:fe:d6:78:fc brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether b0:83:fe:d6:78:fd brd ff:ff:ff:ff:ff:ff
6: br-eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b283:feff:fed6:78fe/64 scope link
       valid_lft forever preferred_lft forever
7: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 3a:66:f9:32:13:a3 brd ff:ff:ff:ff:ff:ff
8: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether e6:e9:8a:8a:8a:4b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e4e9:8aff:fe8a:8a4b/64 scope link
       valid_lft forever preferred_lft forever
9: lldpd2a87771-fd: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 6a:c6:ce:ff:c0:07 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::68c6:ceff:feff:c007/64 scope link
       valid_lft forever preferred_lft forever
10: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 4a:0c:5c:b2:bf:4e brd ff:ff:ff:ff:ff:ff

controller-0:/var/log/puppet# sudo ovs-vsctl show
a98562f6-3223-4eab-a663-68cb4d8e9ece
    Manager "ptcp:6639:127.0.0.1"
        is_connected: true
    Bridge "br-phy0"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "lldpd2a87771-fd"
            Interface "lldpd2a87771-fd"
                type: internal
        Port "phy-br-phy0"
            Interface "phy-br-phy0"
                type: patch
                options: {peer="int-br-phy0"}
        Port "eth0"
            Interface "eth0"
                type: dpdk
                options: {dpdk-devargs="0000:02:00.1", n_rxq="1"}
                error: "Error attaching device '0000:02:00.1' to DPDK"
        Port "br-phy0"
            Interface "br-phy0"
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "int-br-phy0"
            Interface "int-br-phy0"
                type: patch
                options: {peer="phy-br-phy0"}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: “2.9.0"

BR,
Xu Yun

在 2018年12月4日，上午10:13，Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> 写道：

Hi xuyun:
Eth0 is using for dpdk , which is virtual port , not real NIC name.

Could you run “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” to check what type of  eno4 is ?

Are you deploy in a virtual machine with libvirt ? or bare metal system ?

Thanks.
BR
Austin Sun.

From: 徐蕴 [mailto:xuyun@jxresearch.com]
Sent: Tuesday, December 4, 2018 9:31 AM
To: MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>>
Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>
Subject: Re: [Starlingx-discuss] Unable to bring up controller-0

Hello,

Thanks for Eric's hint. I doubted this problem is related to OVS bridge setup yesterday

controller-0:/var/log/puppet# ovs-vsctl show
a98562f6-3223-4eab-a663-68cb4d8e9ece
    Manager "ptcp:6639:127.0.0.1"
        is_connected: true
    Bridge "br-phy0"
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "lldpd2a87771-fd"
            Interface "lldpd2a87771-fd"
                type: internal
        Port "phy-br-phy0"
            Interface "phy-br-phy0"
                type: patch
                options: {peer="int-br-phy0"}
        Port "eth0"
            Interface "eth0"
                type: dpdk
                options: {dpdk-devargs="0000:02:00.1", n_rxq="1"}
                error: "Error attaching device '0000:02:00.1' to DPDK"
        Port "br-phy0"
            Interface "br-phy0"
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "int-br-phy0"
            Interface "int-br-phy0"
                type: patch
                options: {peer="phy-br-phy0"}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: “2.9.0"

I don’t know why ‘eth0’ was added into the br-phy0 bridge, since there is no ‘eth0’ in my machine. I configured an interface named ‘eno4’ for my data plane.
[wrsroot@controller-0 log(keystone_admin)]$ system host-if-list -a 1
+--------------------------------------+------+----------+----------+---------+-----------+----------+-------------+----------------------------+-------------------+
| uuid                                 | name | class    | type     | vlan id | ports     | uses i/f | used by i/f | attributes                 | provider networks |
+--------------------------------------+------+----------+----------+---------+-----------+----------+-------------+----------------------------+-------------------+
| 548477c1-038f-4280-a4b1-d7c11d724b09 | eno4 | data     | ethernet | None    | [u'eno4'] | []       | []          | MTU=1500,accelerated=False | providernet-a     |
| 5c4760f5-a32b-49b9-91d1-73a72fdcabfb | eno3 | None     | ethernet | None    | [u'eno3'] | []       | []          | MTU=1500                   | None              |
| 7b85e213-f435-4693-a13a-364de1e06a35 | eno1 | platform | ethernet | None    | [u'eno1'] | []       | []          | MTU=1500                   | None              |
| 96e1e865-2f7c-486e-8e9e-88fcb1c71686 | lo   | platform | virtual  | None    | []        | []       | []          | MTU=1500                   | None              |
| c87ccb52-a1c7-4332-87f9-5966e6599ee6 | eno2 | None     | ethernet | None    | [u'eno2'] | []       | []          | MTU=1500                   | None              |
+--------------------------------------+------+----------+----------+---------+-----------+----------+-------------+----------------------------+—————————+

And also the puppet log seems confirm that ovs setup failure because of ‘eth0':

controller-0:/var/log/puppet# grep -r /var/log/puppet/ -e Error -e Warning
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:16.348 Notice: 2018-12-03 16:07:16 +0800 /Stage[main]/Platform::Vswitch::Ovs/Platform::Vswitch::Ovs::Port[eth0]/Exec[ovs-add-port: eth0]/returns: ovs-vsctl: Error detected while setting up 'eth0': Error attaching device '0000:02:00.1' to DPDK.  See ovs-vswitchd log for details.
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:16.443 Error: 2018-12-03 16:07:16 +0800 ovs-ofctl add-flow br-phy0 dl_dst=01:80:c2:00:00:0e,dl_type=0x88cc,hard_timeout=0,idle_timeout=0,in_port=eth0,actions=output:lldpd2a87771-fd returned 1 instead of one of [0]
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:16.678 Error: 2018-12-03 16:07:16 +0800 /Stage[main]/Platform::Vswitch::Ovs/Platform::Vswitch::Ovs::Flow[eth0]/Exec[ovs-add-flow: eth0]/returns: change from notrun to 0 failed: ovs-ofctl add-flow br-phy0 dl_dst=01:80:c2:00:00:0e,dl_type=0x88cc,hard_timeout=0,idle_timeout=0,in_port=eth0,actions=output:lldpd2a87771-fd returned 1 instead of one of [0]
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:28.800 Warning: 2018-12-03 16:07:28 +0800 /Stage[post]/Platform::Ceph::Post/File[/var/run/.ceph_started]: Skipping because of failed dependencies
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:28.806 Warning: 2018-12-03 16:07:28 +0800 /Stage[post]/Platform::Config::Post/Service[crond]: Skipping because of failed dependencies
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:28.817 Warning: 2018-12-03 16:07:28 +0800 /Stage[post]/Platform::Config::Compute::Post/File[/etc/platform/.initial_compute_config_complete]: Skipping because of failed dependencies
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:28.824 Warning: 2018-12-03 16:07:28 +0800 /Stage[post]/Platform::Config::Compute::Post/File[/var/run/.compute_config_complete]: Skipping because of failed dependencies
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:28.833 Warning: 2018-12-03 16:07:28 +0800 /Stage[post]/Platform::Config::Post/File[/etc/platform/.initial_config_complete]: Skipping because of failed dependencies
/var/log/puppet/2018-12-03-08-05-45_compute/puppet.log:2018-12-03T08:07:28.839 Warning: 2018-12-03 16:07:28 +0800 /Stage[post]/Platform::Config::Post/File[/etc/platform/.config_applied]: Skipping because of failed dependencies

Any further suggestion how to fix this problem? Thank you.

Br,
Xu Yun

在 2018年12月3日，下午11:02，MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>> 写道：

300.004 is a dependency alarm that should go away once controller-0 is unlocked-enabled.

Looks like an All-In-One system your trying to provision.
Therefore this could be a controller or compute function configuration error.

I would look for the words Error and Warning in the puppet logs and see what that shows.

sudo grep จCr /var/log/puppet จCe Error จCe Warning

Eric.

From: ะ์ิฬ [mailto:xuyun@jxresearch.com]
Sent: Monday, December 03, 2018 4:26 AM
To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>
Subject: [Starlingx-discuss] Unable to bring up controller-0

Hi,

Iกฏm trying to deploy a simplex on a bare metal server. After unlocking controller-0 following the installation_guide, two error events are reported and this node was degraded:

200.011<http://111.111.111.1/admin/fault_management/58857199-137d-41dc-a8ab-063024695f76/detail/>

controller-0 experienced a configuration failure.

host=controller-0

300.004<http://111.111.111.1/admin/fault_management/5be8292e-c1c1-4068-bce3-84b4d43d923b/detail/>

No enabled compute host with connectivity to provider network.

service=networking.providernet=e542cf30-a07d-41f7-be7b-cc5a4e14b0d7

Would you please give me some hints how to debug this problem? Thank you!

Br,
Xu Yun