fgrep Error /var/log/mtcAgent.log fm alarm-list From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, January 25, 2019 8:05 AM To: Himanshu Goyal Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) The DPDK and OVS status look fine now. I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Friday, January 25, 2019 at 3:56 AM To: "Peters, Matt" <Matt.Peters@windriver.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Matt, Please find below output of dpdk-devbind.py --status also: compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status ########################################################################################################## python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active* Other Network devices ===================== 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/var/log/puppet/latest# ################################################################################ Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi Matt, Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors. But compute still rebooting in loop. Please find attached dmesg log output for reference. ####################################################################### compute-0:/home/wrsroot# sudo ovs-vsctl show 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot# ############################################################################# Please suggest me any other logs that i can check. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Thanks Matt, Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version. ################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family ############################################################################# ################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0 Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS ######################################################################################################## Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping). Maybe there are others on the distribution list that could offer additional suggestions. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 11:32 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.