[Starlingx-discuss] Deployment Option (error: compute boot in loop)
Himanshu Goyal
himanshugoyal500 at gmail.com
Fri Jan 25 14:31:20 UTC 2019
Hi Eric,
Below are required outputs of commands:
*fm alarm-list *
######################################################
[root at controller-0 wrsroot(keystone_admin)]# fm alarm-list
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| Alarm | Reason Text
| Entity ID | Severity | Time
Stamp |
| ID |
| | |
|
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| 200. | compute-0 was administratively locked to take it out-of-service.
| host=compute-0 | warning |
2019-01-25T15: |
| 001 |
| | |
00:12.159130 |
| |
| | |
|
| 300. | No enabled compute host with connectivity to provider network.
| service=networking.providernet= | major |
2019-01-25T14: |
| 004 |
| 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | |
58:56.711665 |
| |
| | |
|
| 400. | Service group cloud-services loss of redundancy; expected 1
standby member but | service_domain=controller. | major |
2019-01-22T19: |
| 002 | no standby members available
| service_group=cloud-services | |
12:07.245142 |
| |
| | |
|
| 400. | Service group controller-services loss of redundancy; expected 1
standby member | service_domain=controller. | major |
2019-01-22T19: |
| 002 | but no standby members available
| service_group=controller-services | |
11:40.174144 |
| |
| | |
|
| 400. | Service group vim-services loss of redundancy; expected 1 standby
member but no | service_domain=controller. | major |
2019-01-22T19: |
| 002 | standby members available
| service_group=vim-services | |
11:39.052152 |
| |
| | |
|
| 400. | Service group oam-services loss of redundancy; expected 1 standby
member but no | service_domain=controller. | major |
2019-01-22T19: |
| 002 | standby members available
| service_group=oam-services | |
11:38.728175 |
| |
| | |
|
| 400. | Service group patching-services loss of redundancy; expected 1
standby member | service_domain=controller. | major |
2019-01-22T19: |
| 002 | but no standby members available
| service_group=patching-services | |
11:37.756169 |
| |
| | |
|
| 400. | Service group directory-services loss of redundancy; expected 2
active members | service_domain=controller. | major |
2019-01-22T19: |
| 002 | but only 1 active member available
| service_group=directory-services | |
11:37.513178 |
| |
| | |
|
| 400. | Service group web-services loss of redundancy; expected 2 active
members but | service_domain=controller. | major |
2019-01-22T19: |
| 002 | only 1 active member available
| service_group=web-services | |
11:37.189196 |
| |
| | |
|
| 400. | Communication failure detected with peer over port bond0 on host
controller-0 | host=controller-0.network=mgmt | major |
2019-01-22T19: |
| 005 |
| | |
11:36.690138 |
| |
| | |
|
| 400. | Communication failure detected with peer over port bond1 on host
controller-0 | host=controller-0.network=oam | major |
2019-01-22T19: |
| 005 |
| | |
11:36.427073 |
| |
| | |
|
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
[root at controller-0 wrsroot(keystone_admin)]#
#########################################################################################################
Please find attached file for fgrep Error /var/log/mtcAgent.log output
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric <
Eric.MacDonald at windriver.com> wrote:
> fgrep Error /var/log/mtcAgent.log
>
> fm alarm-list
>
>
>
> *From:* Peters, Matt [mailto:Matt.Peters at windriver.com]
> *Sent:* Friday, January 25, 2019 8:05 AM
> *To:* Himanshu Goyal
> *Cc:* starlingx-discuss at lists.starlingx.io
> *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot
> in loop)
>
>
>
> The DPDK and OVS status look fine now.
>
> I would check the /var/log/mtcAgent.log on the active controller host to
> determine what is being reported for the reboot.
>
>
>
> *From: *Himanshu Goyal <himanshugoyal500 at gmail.com>
> *Date: *Friday, January 25, 2019 at 3:56 AM
> *To: *"Peters, Matt" <Matt.Peters at windriver.com>
> *Cc: *"Hu, Yong" <yong.hu at intel.com>, "
> starlingx-discuss at lists.starlingx.io" <
> starlingx-discuss at lists.starlingx.io>
> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot
> in loop)
>
>
>
> Hi Matt,
>
>
>
> Please find below output of dpdk-devbind.py --status also:
>
>
>
> compute-0:/var/log/puppet/latest# python
> /usr/share/openvswitch/scripts/dpdk-devbind.py --status
>
>
>
>
> ##########################################################################################################
>
> python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
>
> Network devices using DPDK-compatible driver
>
> ============================================
>
> 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
>
>
>
> Network devices using kernel driver
>
> ===================================
>
> 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=ens513f1 drv=ixgbe unused=vfio-pci
>
> 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb
> unused=vfio-pci *Active*
>
>
>
> Other Network devices
>
> =====================
>
> 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> unused=vfio-pci
>
>
>
> Crypto devices using DPDK-compatible driver
>
> ===========================================
>
> <none>
>
>
>
> Crypto devices using kernel driver
>
> ==================================
>
> <none>
>
>
>
> Other Crypto devices
>
> ====================
>
> <none>
>
>
>
> Eventdev devices using DPDK-compatible driver
>
> =============================================
>
> <none>
>
>
>
> Eventdev devices using kernel driver
>
> ====================================
>
> <none>
>
>
>
> Other Eventdev devices
>
> ======================
>
> <none>
>
>
>
> Mempool devices using DPDK-compatible driver
>
> ============================================
>
> <none>
>
>
>
> Mempool devices using kernel driver
>
> ===================================
>
> <none>
>
>
>
> Other Mempool devices
>
> =====================
>
> <none>
>
> compute-0:/var/log/puppet/latest#
>
>
> ################################################################################
>
>
>
> Regards,
>
> Himanshu Goyal
>
>
>
> On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500 at gmail.com>
> wrote:
>
> Hi Matt,
>
>
>
> Sorry, interrupt remapping was disabled in my BIOS, I tried again after
> enable that, Now in dmesg & ovs-vsct there is no errors.* But compute
> still rebooting in loop*.
>
> Please find attached dmesg log output for reference.
>
>
>
> #######################################################################
>
> compute-0:/home/wrsroot#* sudo ovs-vsctl show*
>
> 8a1f7179-2014-4b4e-ad23-2065b8024964
>
> Manager "ptcp:6640:127.0.0.1"
>
> is_connected: true
>
> Bridge "br-phy0"
>
> Controller "tcp:127.0.0.1:6633"
>
> is_connected: true
>
> fail_mode: secure
>
> Port "lldp64416a6d-7d"
>
> Interface "lldp64416a6d-7d"
>
> type: internal
>
> Port "phy-br-phy0"
>
> Interface "phy-br-phy0"
>
> type: patch
>
> options: {peer="int-br-phy0"}
>
> Port "br-phy0"
>
> Interface "br-phy0"
>
> type: internal
>
> Port "eth0"
>
> Interface "eth0"
>
> type: dpdk
>
> options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
>
> Bridge br-int
>
> Controller "tcp:127.0.0.1:6633"
>
> is_connected: true
>
> fail_mode: secure
>
> Port br-int
>
> Interface br-int
>
> type: internal
>
> Port "int-br-phy0"
>
> Interface "int-br-phy0"
>
> type: patch
>
> options: {peer="phy-br-phy0"}
>
> ovs_version: "2.9.0"
>
> compute-0:/home/wrsroot#
>
>
> #############################################################################
>
>
>
> Please suggest me any other logs that i can check.
>
>
>
> Regards,
>
> Himanshu Goyal
>
>
>
> On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <
> himanshugoyal500 at gmail.com> wrote:
>
> Thanks Matt,
>
>
>
> Below are the details of my server BIOS, Currently in my Server Bios
> Version: SE5C610.86B.01.01.0014.121820151719,
>
> Please suggest which version of Bios we can use & this version is
> up-gradable or not with required version.
>
>
>
> #################################################################
>
> compute-0:/home/wrsroot# dmidecode -t1
>
> # dmidecode 3.0
>
> Getting SMBIOS data from sysfs.
>
> SMBIOS 2.7 present.
>
>
>
> Handle 0x0001, DMI type 1, 27 bytes
>
> System Information
>
> Manufacturer: Intel Corporation
>
> Product Name: S2600WT2
>
> Version: ....................
>
> Serial Number: ............
>
> UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
>
> Wake-up Type: Power Switch
>
> SKU Number: SKU Number
>
> Family: Family
>
>
> #############################################################################
>
>
>
>
> ################################################################################
>
> compute-0:/home/wrsroot# sudo dmidecode --type bios
>
> # dmidecode 3.0
>
> Getting SMBIOS data from sysfs.
>
> SMBIOS 2.7 present.
>
>
>
> Handle 0x0000, DMI type 0, 24 bytes
>
> BIOS Information
>
> Vendor: Intel Corporation
>
> Version: SE5C610.86B.01.01.0014.121820151719
>
> Release Date: 12/18/2015
>
> Address: 0xF0000
>
> Runtime Size: 64 kB
>
> ROM Size: 16384 kB
>
> Characteristics:
>
> PCI is supported
>
> PNP is supported
>
> BIOS is upgradeable
>
> BIOS shadowing is allowed
>
> Boot from CD is supported
>
> Selectable boot is supported
>
> EDD is supported
>
> 5.25"/1.2 MB floppy services are supported (int 13h)
>
> 3.5"/720 kB floppy services are supported (int 13h)
>
> 3.5"/2.88 MB floppy services are supported (int 13h)
>
> Print screen service is supported (int 5h)
>
> 8042 keyboard services are supported (int 9h)
>
> Serial services are supported (int 14h)
>
> Printer services are supported (int 17h)
>
> CGA/mono video services are supported (int 10h)
>
> ACPI is supported
>
> USB legacy is supported
>
> LS-120 boot is supported
>
> ATAPI Zip drive boot is supported
>
> BIOS boot specification is supported
>
> Function key-initiated network boot is supported
>
> Targeted content distribution is supported
>
> UEFI is supported
>
> BIOS Revision: 0.0
>
> Firmware Revision: 0.0
>
>
>
> Handle 0x000B, DMI type 13, 22 bytes
>
> BIOS Language Information
>
> Language Description Format: Abbreviated
>
> Installable Languages: 1
>
> enUS
>
> Currently Installed Language: enUS
>
>
> ########################################################################################################
>
>
>
> Regards,
>
> Himanshu Goyal
>
>
>
> On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters at windriver.com>
> wrote:
>
> Based on the logs, it looks like you have an older system that doesn’t
> fully support VT-d IOMMU remapping. You can try updating your BIOS if
> there is a newer version available. We don’t currently support setting
> arbitrary module parameters, so we don’t have a way to implement the
> workaround module param specified in the logs (i.e. disabling interrupt
> remapping).
>
>
>
> Maybe there are others on the distribution list that could offer
> additional suggestions.
>
>
>
> *From: *Himanshu Goyal <himanshugoyal500 at gmail.com>
> *Date: *Thursday, January 24, 2019 at 11:32 AM
> *To: *"Peters, Matt" <Matt.Peters at windriver.com>
> *Cc: *"Hu, Yong" <yong.hu at intel.com>, "
> starlingx-discuss at lists.starlingx.io" <
> starlingx-discuss at lists.starlingx.io>
> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot
> in loop)
>
>
>
> Hi Peters,
>
>
>
> Please find output of required command in attached file.
>
>
>
> Regards,
>
> Himanshu Goyal
>
>
>
> Regards,
>
> Himanshu Goyal
>
> On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters at windriver.com>
> wrote:
>
> Can you confirm that VT-d is enabled properly by running the following?
>
>
>
> dmesg | grep -i -e DMAR -e IOMMU
>
>
>
> *From: *Himanshu Goyal <himanshugoyal500 at gmail.com>
> *Date: *Thursday, January 24, 2019 at 8:19 AM
> *To: *"Peters, Matt" <Matt.Peters at windriver.com>
> *Cc: *"Hu, Yong" <yong.hu at intel.com>, "
> starlingx-discuss at lists.starlingx.io" <
> starlingx-discuss at lists.starlingx.io>
> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot
> in loop)
>
>
>
> Hi Peters,
>
>
>
> Error logs ovs-vswitched.log:
>
> #######################################
>
> 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on
> NUMA socket 0
>
> 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb
> net_ixgbe
>
> 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to
> select IOMMU type
>
> 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the
> device (0000:02:00.0)
>
> 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device
> '0000:02:00.0' to DPDK
>
> 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set
> configuration (Invalid argument)
>
> ########################################
>
>
>
> Please find attached log file also.
>
>
>
> Regards,
>
> Himanshu Goyal
>
>
>
> On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters at windriver.com>
> wrote:
>
> The name of eth0 is correct. It is just an assigned name by configuration
> management.
>
>
>
> What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
>
>
>
> *From: *Himanshu Goyal <himanshugoyal500 at gmail.com>
> *Date: *Thursday, January 24, 2019 at 6:52 AM
> *To: *"Hu, Yong" <yong.hu at intel.com>
> *Cc: *"starlingx-discuss at lists.starlingx.io" <
> starlingx-discuss at lists.starlingx.io>
> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot
> in loop)
>
>
>
> Thanks a lot Yong,
>
>
>
> Checked OVS/DPDK status, It is giving me the error as output shown below:
>
> In br-phy0 is attaching a port name as eth0, but my compute port name
> is ens513f0, i think that may be the issue.
>
>
>
>
> #################################################################################
>
> *ovs-vsctl show output:*
>
> compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
>
> 543d08f4-ff1e-4a8d-8e48-11f01356750d
>
> Manager "ptcp:6640:127.0.0.1"
>
> is_connected: true
>
> Bridge "br-phy0"
>
> Controller "tcp:127.0.0.1:6633"
>
> is_connected: true
>
> fail_mode: secure
>
> Port "phy-br-phy0"
>
> Interface "phy-br-phy0"
>
> type: patch
>
> options: {peer="int-br-phy0"}
>
> Port "br-phy0"
>
> Interface "br-phy0"
>
> type: internal
>
> Port "lldpabeb30a6-6c"
>
> Interface "lldpabeb30a6-6c"
>
> type: internal
>
> Port "eth0"
>
> * Interface "eth0"*
>
> type: dpdk
>
> options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
>
> *error: "Error attaching device '0000:02:00.0' to DPDK"*
>
> Bridge br-int
>
> Controller "tcp:127.0.0.1:6633"
>
> is_connected: true
>
> fail_mode: secure
>
> Port "int-br-phy0"
>
> Interface "int-br-phy0"
>
> type: patch
>
> options: {peer="phy-br-phy0"}
>
> Port br-int
>
> Interface br-int
>
> type: internal
>
> ovs_version: "2.9.0"
>
>
> #####################################################################################################
>
>
>
>
> ##############################################################################
>
> *Output of dpdk-devbind.py:*
>
>
>
> compute-0:/usr/share/openvswitch/scripts$ python
> /usr/share/openvswitch/scripts/dpdk-devbind.py --status
>
>
>
> Network devices using DPDK-compatible driver
>
> ============================================
>
> 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> drv=vfio-pci unused=
>
>
>
> Network devices using kernel driver
>
> ===================================
>
> 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
>
> 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb
> unused=vfio-pci
>
> 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb
> unused=vfio-pci
>
>
>
> Other Network devices
>
> =====================
>
> <none>
>
>
>
> Crypto devices using DPDK-compatible driver
>
> ===========================================
>
> <none>
>
>
>
> Crypto devices using kernel driver
>
> ==================================
>
> <none>
>
>
>
> Other Crypto devices
>
> ====================
>
> <none>
>
>
>
> Eventdev devices using DPDK-compatible driver
>
> =============================================
>
> <none>
>
>
>
> Eventdev devices using kernel driver
>
> ====================================
>
> <none>
>
>
>
> Other Eventdev devices
>
> ======================
>
> <none>
>
>
>
> Mempool devices using DPDK-compatible driver
>
> ============================================
>
> <none>
>
>
>
> Mempool devices using kernel driver
>
> ===================================
>
> <none>
>
>
>
> Other Mempool devices
>
> =====================
>
> <none>
>
> compute-0:/usr/share/openvswitch/scripts$
>
> #############################################################
>
>
>
> Both management and OAM ports support DPDK.
>
>
>
> *lspci output:*
>
> *##################################################*
>
> compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
>
> 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit
> SFI/SFP+ Network Connection [8086:10fb] (rev 01)
>
> 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit
> SFI/SFP+ Network Connection [8086:10fb] (rev 01)
>
> 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network
> Connection [8086:1521] (rev 01)
>
> 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network
> Connection [8086:1521] (rev 01)
>
> ###############################################################
>
> compute-0:/usr/share/openvswitch/scripts$
>
>
>
>
>
> Many Thanks,
>
> Himanshu Goyal
>
>
>
>
>
> On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu at intel.com> wrote:
>
> I saw you enabled VT-D, how about VT-x in BIOS?
>
>
>
> Another possibility is that your ethernet NIC is not compatible with
> OVS/DPDK.
>
> Please follow these steps, to dig out more info
>
> 1. lock your compute-node
>
> 2. ssh to your compute-node and check OVS/DPDK status by “sudo
> ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py
> --status”
>
>
>
> *From: *Himanshu Goyal <himanshugoyal500 at gmail.com>
> *Date: *Thursday, 24 January 2019 at 4:46 PM
> *To: *"Hu, Yong" <yong.hu at intel.com>
> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso at intel.com>, "
> starlingx-discuss at lists.starlingx.io" <
> starlingx-discuss at lists.starlingx.io>
> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot
> in loop)
>
> Hi Yong,
>
>
>
> I checked server BIOS setting Intel VT-D is enabled in my compute machine
>
> compute machine is rebooting only after unlocking, Before reboot it came
> in unlocked, enabled & online.
>
>
>
> Please suggest how can i debug this.
>
>
>
> Many Thanks,
>
> Himanshu Goyal
>
>
>
> On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu at intel.com> wrote:
>
> Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
>
> There are other 2, like VT-x and VT-d flags, to be enabled. Please have a
> look in BIOS.
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190125/37fd3341/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fgrep_error_mtcAgent.log
Type: application/octet-stream
Size: 76313 bytes
Desc: not available
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190125/37fd3341/attachment-0001.obj>
More information about the Starlingx-discuss
mailing list