[Starlingx-discuss] Deployment Option
Hi, Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines). Many Thanks, Himanshu Goyal
Yes, you can. Though, you’d better have 2 disks on each compute. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Tuesday, 1 January 2019 at 6:06 PM To: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Deployment Option Hi, Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines). Many Thanks, Himanshu Goyal
Hi, I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task* waiting for service activation*...... with Error: "*Configuration failed: Timeout waiting for service enable*" I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Please suggest us the procedure to debug that & how i can re-run the config_controller again. Many Thanks, Himanshu Goyal On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi,
Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines).
Many Thanks, Himanshu Goyal
Hi, On what step of config_controller it is failing? Can you provide the logs? Are you deploying manually or automatic? To apply the config_controller again I think you need to start over the installation process. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com] Sent: Thursday, January 17, 2019 7:22 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option Hi, I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task waiting for service activation...... with Error: "Configuration failed: Timeout waiting for service enable" I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Please suggest us the procedure to debug that & how i can re-run the config_controller again. Many Thanks, Himanshu Goyal On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi, Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines). Many Thanks, Himanshu Goyal
Thanks Juan, Error is resolved after again boot the server. The reason for that my management interface was not up. I have some Questions: 1) As I'm deploying starlingX with only 1 controller & 1 compute Machine.while executing "sudo config_controller" it is taking the IP Addresses of controller 0, controller 1 & floating IP Address. As i have only one controller, So how can i avoid those type of configurations..? 2) Is there any specific installation guide for that(1 controller & 1 compute) type installation currently I'm following the guide: https://docs.starlingx.io/installation_guide/controller_storage.html#control.... Please suggest me the need full changes i have to done for that. 3) my controller machine have one Disk of around 3725GB, But when I'm trying to configure cinder on same disk host-disk-list command shows me available disk is as 0 GB. please suggest me if there anyway to use that same disk with cinder. *snapshot of host-disk-list:* [image: image.png] Regards, Himanshu Goyal On Thu, Jan 17, 2019 at 8:41 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
On what step of config_controller it is failing? Can you provide the logs?
Are you deploying manually or automatic?
To apply the config_controller again I think you need to start over the installation process.
Regards.
Juan Carlos Alonso
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Thursday, January 17, 2019 7:22 AM *To:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi,
I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task* waiting for service activation* ......
with Error: "*Configuration failed: Timeout waiting for service enable*"
I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou...
Please suggest us the procedure to debug that & how i can re-run the config_controller again.
Many Thanks,
Himanshu Goyal
On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi,
Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines).
Many Thanks,
Himanshu Goyal
Hi, Good that you could resolve config_controller issue. I am not sure if you can avoid/skip extra configuration for hosts you don’t want during config_controller. I think you need to follow the installation steps normally but provision only the host you want to use. Yesterday I asked to my team and it is possible to deploy 1 controller and 1 compute only, please refer to https://github.com/xe1gyq/starlingx/blob/master/ControllerStorage.md To configure cinder on controller-0 you need to have a partition with space available to be added to cinder-volume. By default should be two partitions on each host, one with available space. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com] Sent: Thursday, January 17, 2019 11:16 PM To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Error is resolved after again boot the server. The reason for that my management interface was not up. I have some Questions: 1) As I'm deploying starlingX with only 1 controller & 1 compute Machine.while executing "sudo config_controller" it is taking the IP Addresses of controller 0, controller 1 & floating IP Address. As i have only one controller, So how can i avoid those type of configurations..? 2) Is there any specific installation guide for that(1 controller & 1 compute) type installation currently I'm following the guide: https://docs.starlingx.io/installation_guide/controller_storage.html#control.... Please suggest me the need full changes i have to done for that. 3) my controller machine have one Disk of around 3725GB, But when I'm trying to configure cinder on same disk host-disk-list command shows me available disk is as 0 GB. please suggest me if there anyway to use that same disk with cinder. snapshot of host-disk-list: [image.png] Regards, Himanshu Goyal On Thu, Jan 17, 2019 at 8:41 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, On what step of config_controller it is failing? Can you provide the logs? Are you deploying manually or automatic? To apply the config_controller again I think you need to start over the installation process. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>] Sent: Thursday, January 17, 2019 7:22 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Hi, I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task waiting for service activation...... with Error: "Configuration failed: Timeout waiting for service enable" I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Please suggest us the procedure to debug that & how i can re-run the config_controller again. Many Thanks, Himanshu Goyal On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi, Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines). Many Thanks, Himanshu Goyal
Thanks Juan, Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "*system host-list*" command. my controller machine is directly connected to compute machine. I'm following the below steps Steps: *1) system host-unlock controller-0* *2) system host-list* Output:: [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 3) power on my compute machine. And give option to boot from PXE my compute machine is directly connected with controller with mgmt port. But not able to see host in "system host-list". 4) i tried with system host-add command also, but it is giving below error: *Error:* [wrsroot@controller-0 ~(keystone_admin)]$ system host-add Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. Please suggest me the needful change. Regards, Himanshu Goyal On Fri, Jan 18, 2019 at 10:01 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
Good that you could resolve config_controller issue.
I am not sure if you can avoid/skip extra configuration for hosts you don’t want during config_controller. I think you need to follow the installation steps normally but provision only the host you want to use.
Yesterday I asked to my team and it is possible to deploy 1 controller and 1 compute only, please refer to https://github.com/xe1gyq/starlingx/blob/master/ControllerStorage.md
To configure cinder on controller-0 you need to have a partition with space available to be added to cinder-volume. By default should be two partitions on each host, one with available space.
Regards.
Juan Carlos Alonso
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Thursday, January 17, 2019 11:16 PM *To:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Thanks Juan,
Error is resolved after again boot the server. The reason for that my management interface was not up.
I have some Questions:
1) As I'm deploying starlingX with only 1 controller & 1 compute Machine.while executing "sudo config_controller" it is taking the IP Addresses of controller 0, controller 1 & floating IP Address. As i have only one controller, So how can i avoid those type of configurations..?
2) Is there any specific installation guide for that(1 controller & 1 compute) type installation currently I'm following the guide: https://docs.starlingx.io/installation_guide/controller_storage.html#control.... Please suggest me the need full changes i have to done for that.
3) my controller machine have one Disk of around 3725GB, But when I'm trying to configure cinder on same disk host-disk-list command shows me available disk is as 0 GB. please suggest me if there anyway to use that same disk with cinder.
*snapshot of host-disk-list:*
[image: image.png]
Regards,
Himanshu Goyal
On Thu, Jan 17, 2019 at 8:41 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
On what step of config_controller it is failing? Can you provide the logs?
Are you deploying manually or automatic?
To apply the config_controller again I think you need to start over the installation process.
Regards.
Juan Carlos Alonso
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Thursday, January 17, 2019 7:22 AM *To:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi,
I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task* waiting for service activation* ......
with Error: "*Configuration failed: Timeout waiting for service enable*"
I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou...
Please suggest us the procedure to debug that & how i can re-run the config_controller again.
Many Thanks,
Himanshu Goyal
On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi,
Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines).
Many Thanks,
Himanshu Goyal
Hi Himanshu, “system host-list” doesn’t “see” your compute node and LLDP won’t work, because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub). Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd: # system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac> Regards, yong From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Monday, 21 January 2019 at 8:14 PM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "system host-list" command. my controller machine is directly connected to compute machine. I'm following the below steps Steps: 1) system host-unlock controller-0 2) system host-list Output:: [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 3) power on my compute machine. And give option to boot from PXE my compute machine is directly connected with controller with mgmt port. But not able to see host in "system host-list". 4) i tried with system host-add command also, but it is giving below error: Error: [wrsroot@controller-0 ~(keystone_admin)]$ system host-add Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. Please suggest me the needful change. Regards, Himanshu Goyal On Fri, Jan 18, 2019 at 10:01 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, Good that you could resolve config_controller issue. I am not sure if you can avoid/skip extra configuration for hosts you don’t want during config_controller. I think you need to follow the installation steps normally but provision only the host you want to use. Yesterday I asked to my team and it is possible to deploy 1 controller and 1 compute only, please refer to https://github.com/xe1gyq/starlingx/blob/master/ControllerStorage.md To configure cinder on controller-0 you need to have a partition with space available to be added to cinder-volume. By default should be two partitions on each host, one with available space. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>] Sent: Thursday, January 17, 2019 11:16 PM To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Error is resolved after again boot the server. The reason for that my management interface was not up. I have some Questions: 1) As I'm deploying starlingX with only 1 controller & 1 compute Machine.while executing "sudo config_controller" it is taking the IP Addresses of controller 0, controller 1 & floating IP Address. As i have only one controller, So how can i avoid those type of configurations..? 2) Is there any specific installation guide for that(1 controller & 1 compute) type installation currently I'm following the guide: https://docs.starlingx.io/installation_guide/controller_storage.html#control.... Please suggest me the need full changes i have to done for that. 3) my controller machine have one Disk of around 3725GB, But when I'm trying to configure cinder on same disk host-disk-list command shows me available disk is as 0 GB. please suggest me if there anyway to use that same disk with cinder. snapshot of host-disk-list: [image.png] Regards, Himanshu Goyal On Thu, Jan 17, 2019 at 8:41 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, On what step of config_controller it is failing? Can you provide the logs? Are you deploying manually or automatic? To apply the config_controller again I think you need to start over the installation process. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>] Sent: Thursday, January 17, 2019 7:22 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Hi, I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task waiting for service activation...... with Error: "Configuration failed: Timeout waiting for service enable" I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Please suggest us the procedure to debug that & how i can re-run the config_controller again. Many Thanks, Himanshu Goyal On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi, Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines). Many Thanks, Himanshu Goyal
Hi, The personality of computes changed to “worker”, so the command should be: system host-add -n compute-0 -p worker -m ${mac_address} Regards. Juan Carlos Alonso From: Hu, Yong Sent: Monday, January 21, 2019 8:51 AM To: Himanshu Goyal <himanshugoyal500@gmail.com>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option Hi Himanshu, “system host-list” doesn’t “see” your compute node and LLDP won’t work, because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub). Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd: # system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac> Regards, yong From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Monday, 21 January 2019 at 8:14 PM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "system host-list" command. my controller machine is directly connected to compute machine. I'm following the below steps Steps: 1) system host-unlock controller-0 2) system host-list Output:: [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 3) power on my compute machine. And give option to boot from PXE my compute machine is directly connected with controller with mgmt port. But not able to see host in "system host-list". 4) i tried with system host-add command also, but it is giving below error: Error: [wrsroot@controller-0 ~(keystone_admin)]$ system host-add Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. Please suggest me the needful change. Regards, Himanshu Goyal On Fri, Jan 18, 2019 at 10:01 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, Good that you could resolve config_controller issue. I am not sure if you can avoid/skip extra configuration for hosts you don’t want during config_controller. I think you need to follow the installation steps normally but provision only the host you want to use. Yesterday I asked to my team and it is possible to deploy 1 controller and 1 compute only, please refer to https://github.com/xe1gyq/starlingx/blob/master/ControllerStorage.md To configure cinder on controller-0 you need to have a partition with space available to be added to cinder-volume. By default should be two partitions on each host, one with available space. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>] Sent: Thursday, January 17, 2019 11:16 PM To: Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Error is resolved after again boot the server. The reason for that my management interface was not up. I have some Questions: 1) As I'm deploying starlingX with only 1 controller & 1 compute Machine.while executing "sudo config_controller" it is taking the IP Addresses of controller 0, controller 1 & floating IP Address. As i have only one controller, So how can i avoid those type of configurations..? 2) Is there any specific installation guide for that(1 controller & 1 compute) type installation currently I'm following the guide: https://docs.starlingx.io/installation_guide/controller_storage.html#control.... Please suggest me the need full changes i have to done for that. 3) my controller machine have one Disk of around 3725GB, But when I'm trying to configure cinder on same disk host-disk-list command shows me available disk is as 0 GB. please suggest me if there anyway to use that same disk with cinder. snapshot of host-disk-list: [image.png] Regards, Himanshu Goyal On Thu, Jan 17, 2019 at 8:41 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, On what step of config_controller it is failing? Can you provide the logs? Are you deploying manually or automatic? To apply the config_controller again I think you need to start over the installation process. Regards. Juan Carlos Alonso From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>] Sent: Thursday, January 17, 2019 7:22 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Hi, I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task waiting for service activation...... with Error: "Configuration failed: Timeout waiting for service enable" I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Please suggest us the procedure to debug that & how i can re-run the config_controller again. Many Thanks, Himanshu Goyal On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi, Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines). Many Thanks, Himanshu Goyal
Thanks Juan ,Yong I tried both the commands output shows as below: 1) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe* usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>] [-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>] [-T <bm_type>] [-U <bm_username>] [-P <bm_password>] [-b <boot_device>] [-r <rootfs_device>] [-o <install_output>] [-c <console>] [-v <vsc_controllers>] [-l <location>] [-D <true/false>] system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile') [wrsroot@controller-0 ~(keystone_admin)]$ [wrsroot@controller-0 ~(keystone_admin)]$ 2) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe* Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. [wrsroot@controller-0 ~(keystone_admin)]$ Regards, Himanshu Goyal On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
The personality of computes changed to “worker”, so the command should be:
system host-add -n compute-0 -p worker -m ${mac_address}
Regards.
Juan Carlos Alonso
*From:* Hu, Yong *Sent:* Monday, January 21, 2019 8:51 AM *To:* Himanshu Goyal <himanshugoyal500@gmail.com>; Alonso, Juan Carlos < juan.carlos.alonso@intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi Himanshu,
“system host-list” doesn’t “see” your compute node and LLDP won’t work,
because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub).
Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd:
# system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac>
Regards,
yong
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Monday, 21 January 2019 at 8:14 PM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan,
Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "*system host-list*" command.
my controller machine is directly connected to compute machine.
I'm following the below steps
Steps:
*1) system host-unlock controller-0*
*2) system host-list*
Output::
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
3) power on my compute machine. And give option to boot from PXE
my compute machine is directly connected with controller with mgmt port.
But not able to see host in "system host-list".
4) i tried with system host-add command also, but it is giving below error:
*Error:*
[wrsroot@controller-0 ~(keystone_admin)]$ system host-add
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
Please suggest me the needful change.
Regards,
Himanshu Goyal
On Fri, Jan 18, 2019 at 10:01 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
Good that you could resolve config_controller issue.
I am not sure if you can avoid/skip extra configuration for hosts you don’t want during config_controller. I think you need to follow the installation steps normally but provision only the host you want to use.
Yesterday I asked to my team and it is possible to deploy 1 controller and 1 compute only, please refer to https://github.com/xe1gyq/starlingx/blob/master/ControllerStorage.md
To configure cinder on controller-0 you need to have a partition with space available to be added to cinder-volume. By default should be two partitions on each host, one with available space.
Regards.
Juan Carlos Alonso
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Thursday, January 17, 2019 11:16 PM *To:* Alonso, Juan Carlos <juan.carlos.alonso@intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Thanks Juan,
Error is resolved after again boot the server. The reason for that my management interface was not up.
I have some Questions:
1) As I'm deploying starlingX with only 1 controller & 1 compute Machine.while executing "sudo config_controller" it is taking the IP Addresses of controller 0, controller 1 & floating IP Address. As i have only one controller, So how can i avoid those type of configurations..?
2) Is there any specific installation guide for that(1 controller & 1 compute) type installation currently I'm following the guide: https://docs.starlingx.io/installation_guide/controller_storage.html#control.... Please suggest me the need full changes i have to done for that.
3) my controller machine have one Disk of around 3725GB, But when I'm trying to configure cinder on same disk host-disk-list command shows me available disk is as 0 GB. please suggest me if there anyway to use that same disk with cinder.
*snapshot of host-disk-list:*
[image: image.png]
Regards,
Himanshu Goyal
On Thu, Jan 17, 2019 at 8:41 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
On what step of config_controller it is failing? Can you provide the logs?
Are you deploying manually or automatic?
To apply the config_controller again I think you need to start over the installation process.
Regards.
Juan Carlos Alonso
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Thursday, January 17, 2019 7:22 AM *To:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi,
I'm trying to install StarlingX Controller 0 on Physical Machine, But it is failing in config_controller at task* waiting for service activation* ......
with Error: "*Configuration failed: Timeout waiting for service enable*"
I'm using ISO available at the path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou...
Please suggest us the procedure to debug that & how i can re-run the config_controller again.
Many Thanks,
Himanshu Goyal
On Tue, Jan 1, 2019 at 3:34 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi,
Can we deploy starlingX with 2 Machines 1 controller & 1 Compute Node (Both nodes on different physical Machines).
Many Thanks,
Himanshu Goyal
Himanshu, Could you have a try with a hub which 2 mgt ports (from controller and compute) are plugged into? Let’s assure the normal setup works first, and then figure out why the direct linkage of cable doesn’t work. BTW: “worker” and “compute” are just different “personality” names in different STX version. On your current setup “compute” will do, supposedly. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Tuesday, 22 January 2019 at 12:32 AM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan ,Yong I tried both the commands output shows as below: 1) [wrsroot@controller-0 ~(keystone_admin)]$ system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>] [-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>] [-T <bm_type>] [-U <bm_username>] [-P <bm_password>] [-b <boot_device>] [-r <rootfs_device>] [-o <install_output>] [-c <console>] [-v <vsc_controllers>] [-l <location>] [-D <true/false>] system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile') [wrsroot@controller-0 ~(keystone_admin)]$ [wrsroot@controller-0 ~(keystone_admin)]$ 2) [wrsroot@controller-0 ~(keystone_admin)]$ system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. [wrsroot@controller-0 ~(keystone_admin)]$ Regards, Himanshu Goyal On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, The personality of computes changed to “worker”, so the command should be: system host-add -n compute-0 -p worker -m ${mac_address} Regards. Juan Carlos Alonso From: Hu, Yong Sent: Monday, January 21, 2019 8:51 AM To: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Hi Himanshu, “system host-list” doesn’t “see” your compute node and LLDP won’t work, because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub). Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd: # system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac> Regards, yong From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Monday, 21 January 2019 at 8:14 PM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "system host-list" command. my controller machine is directly connected to compute machine. I'm following the below steps Steps: 1) system host-unlock controller-0 2) system host-list Output:: [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 3) power on my compute machine. And give option to boot from PXE my compute machine is directly connected with controller with mgmt port. But not able to see host in "system host-list". 4) i tried with system host-add command also, but it is giving below error: Error: [wrsroot@controller-0 ~(keystone_admin)]$ system host-add Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. Please suggest me the needful change. Regards, Himanshu Goyal
Thanks a lot Juan and Yong, Able to see compute host in "system-host-list" after connecting with a hub. starlingX Compute installation has been done. But after unlocking the compute nodes the compute node come into an endless boot loop. dmesg log shows below error: [ 20.967137] iTCO_vendor_support: vendor-support=0 [ 20.968785] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 20.968840] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 21.066607] device-mapper: uevent: version 1.0.3 [ 21.067124] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com Checked the BIOS setting those are as mentioned in installation document. I'm using ISO Image available at below path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Regards, Himanshu Goyal On Tue, Jan 22, 2019 at 12:43 PM Hu, Yong <yong.hu@intel.com> wrote:
Himanshu,
Could you have a try with a hub which 2 mgt ports (from controller and compute) are plugged into?
Let’s assure the normal setup works first, and then figure out why the direct linkage of cable doesn’t work.
BTW: “worker” and “compute” are just different “personality” names in different STX version.
On your current setup “compute” will do, supposedly.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Tuesday, 22 January 2019 at 12:32 AM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan ,Yong
I tried both the commands output shows as below:
1) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe*
usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>]
[-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>]
[-T <bm_type>] [-U <bm_username>] [-P <bm_password>]
[-b <boot_device>] [-r <rootfs_device>]
[-o <install_output>] [-c <console>]
[-v <vsc_controllers>] [-l <location>]
[-D <true/false>]
system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile')
[wrsroot@controller-0 ~(keystone_admin)]$
[wrsroot@controller-0 ~(keystone_admin)]$
2) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe*
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
[wrsroot@controller-0 ~(keystone_admin)]$
Regards,
Himanshu Goyal
On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
The personality of computes changed to “worker”, so the command should be:
system host-add -n compute-0 -p worker -m ${mac_address}
Regards.
Juan Carlos Alonso
*From:* Hu, Yong *Sent:* Monday, January 21, 2019 8:51 AM *To:* Himanshu Goyal <himanshugoyal500@gmail.com>; Alonso, Juan Carlos < juan.carlos.alonso@intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi Himanshu,
“system host-list” doesn’t “see” your compute node and LLDP won’t work,
because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub).
Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd:
# system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac>
Regards,
yong
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Monday, 21 January 2019 at 8:14 PM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan,
Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "*system host-list*" command.
my controller machine is directly connected to compute machine.
I'm following the below steps
Steps:
*1) system host-unlock controller-0*
*2) system host-list*
Output::
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
3) power on my compute machine. And give option to boot from PXE
my compute machine is directly connected with controller with mgmt port.
But not able to see host in "system host-list".
4) i tried with system host-add command also, but it is giving below error:
*Error:*
[wrsroot@controller-0 ~(keystone_admin)]$ system host-add
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
Please suggest me the needful change.
Regards,
Himanshu Goyal
Hi Himanshu, For compute/worker node, please make sure Virtualization settings (such as VT-X and VT-D) ENABLED in BIOS. They are mandatory requirements for compute node. Regards, Yong From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Wednesday, 23 January 2019 at 6:33 PM To: "Hu, Yong" <yong.hu@intel.com> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Juan and Yong, Able to see compute host in "system-host-list" after connecting with a hub. starlingX Compute installation has been done. But after unlocking the compute nodes the compute node come into an endless boot loop. dmesg log shows below error: [ 20.967137] iTCO_vendor_support: vendor-support=0 [ 20.968785] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 20.968840] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 21.066607] device-mapper: uevent: version 1.0.3 [ 21.067124] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com<mailto:dm-devel@redhat.com> Checked the BIOS setting those are as mentioned in installation document. I'm using ISO Image available at below path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Regards, Himanshu Goyal On Tue, Jan 22, 2019 at 12:43 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Himanshu, Could you have a try with a hub which 2 mgt ports (from controller and compute) are plugged into? Let’s assure the normal setup works first, and then figure out why the direct linkage of cable doesn’t work. BTW: “worker” and “compute” are just different “personality” names in different STX version. On your current setup “compute” will do, supposedly. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Tuesday, 22 January 2019 at 12:32 AM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan ,Yong I tried both the commands output shows as below: 1) [wrsroot@controller-0 ~(keystone_admin)]$ system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>] [-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>] [-T <bm_type>] [-U <bm_username>] [-P <bm_password>] [-b <boot_device>] [-r <rootfs_device>] [-o <install_output>] [-c <console>] [-v <vsc_controllers>] [-l <location>] [-D <true/false>] system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile') [wrsroot@controller-0 ~(keystone_admin)]$ [wrsroot@controller-0 ~(keystone_admin)]$ 2) [wrsroot@controller-0 ~(keystone_admin)]$ system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. [wrsroot@controller-0 ~(keystone_admin)]$ Regards, Himanshu Goyal On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, The personality of computes changed to “worker”, so the command should be: system host-add -n compute-0 -p worker -m ${mac_address} Regards. Juan Carlos Alonso From: Hu, Yong Sent: Monday, January 21, 2019 8:51 AM To: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Hi Himanshu, “system host-list” doesn’t “see” your compute node and LLDP won’t work, because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub). Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd: # system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac> Regards, yong From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Monday, 21 January 2019 at 8:14 PM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "system host-list" command. my controller machine is directly connected to compute machine. I'm following the below steps Steps: 1) system host-unlock controller-0 2) system host-list Output:: [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 3) power on my compute machine. And give option to boot from PXE my compute machine is directly connected with controller with mgmt port. But not able to see host in "system host-list". 4) i tried with system host-add command also, but it is giving below error: Error: [wrsroot@controller-0 ~(keystone_admin)]$ system host-add Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. Please suggest me the needful change. Regards, Himanshu Goyal
Hi Yong, Yes, virtualization setting are enabled in BIOS. Below are the dmesg logs for IOMMU [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.36.tis.x86_64 root=UUID=8c9ba8fa-ecae-4d66-98ac-a77fb66faae2 ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 hugepagesz=1G hugepages=2 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1* intel_iommu=on *user_namespace.enable=1 hugepagesz=2M hugepages=0 default_hugepagesz=2M isolcpus=1,2 rcu_nocbs=1-35 kthread_cpus=0 irqaffinity=0 nopti nospectre_v2 [ 0.000000] audit: disabled (until reboot) *[ 0.000000] DMAR: IOMMU enabled* [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 7:43 PM Hu, Yong <yong.hu@intel.com> wrote:
Hi Himanshu,
For compute/worker node, please make sure Virtualization settings (such as VT-X and VT-D) ENABLED in BIOS.
They are mandatory requirements for compute node.
Regards,
Yong
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Wednesday, 23 January 2019 at 6:33 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Juan and Yong,
Able to see compute host in "system-host-list" after connecting with a hub. starlingX Compute installation has been done.
But after unlocking the compute nodes the compute node come into an endless boot loop.
dmesg log shows below error:
[ 20.967137] iTCO_vendor_support: vendor-support=0 [ 20.968785] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 20.968840] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 21.066607] device-mapper: uevent: version 1.0.3 [ 21.067124] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com
Checked the BIOS setting those are as mentioned in installation document.
I'm using ISO Image available at below path:
http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou...
Regards,
Himanshu Goyal
On Tue, Jan 22, 2019 at 12:43 PM Hu, Yong <yong.hu@intel.com> wrote:
Himanshu,
Could you have a try with a hub which 2 mgt ports (from controller and compute) are plugged into?
Let’s assure the normal setup works first, and then figure out why the direct linkage of cable doesn’t work.
BTW: “worker” and “compute” are just different “personality” names in different STX version.
On your current setup “compute” will do, supposedly.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Tuesday, 22 January 2019 at 12:32 AM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan ,Yong
I tried both the commands output shows as below:
1) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe*
usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>]
[-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>]
[-T <bm_type>] [-U <bm_username>] [-P <bm_password>]
[-b <boot_device>] [-r <rootfs_device>]
[-o <install_output>] [-c <console>]
[-v <vsc_controllers>] [-l <location>]
[-D <true/false>]
system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile')
[wrsroot@controller-0 ~(keystone_admin)]$
[wrsroot@controller-0 ~(keystone_admin)]$
2) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe*
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
[wrsroot@controller-0 ~(keystone_admin)]$
Regards,
Himanshu Goyal
On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
The personality of computes changed to “worker”, so the command should be:
system host-add -n compute-0 -p worker -m ${mac_address}
Regards.
Juan Carlos Alonso
*From:* Hu, Yong *Sent:* Monday, January 21, 2019 8:51 AM *To:* Himanshu Goyal <himanshugoyal500@gmail.com>; Alonso, Juan Carlos < juan.carlos.alonso@intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi Himanshu,
“system host-list” doesn’t “see” your compute node and LLDP won’t work,
because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub).
Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd:
# system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac>
Regards,
yong
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Monday, 21 January 2019 at 8:14 PM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan,
Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "*system host-list*" command.
my controller machine is directly connected to compute machine.
I'm following the below steps
Steps:
*1) system host-unlock controller-0*
*2) system host-list*
Output::
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
3) power on my compute machine. And give option to boot from PXE
my compute machine is directly connected with controller with mgmt port.
But not able to see host in "system host-list".
4) i tried with system host-add command also, but it is giving below error:
*Error:*
[wrsroot@controller-0 ~(keystone_admin)]$ system host-add
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
Please suggest me the needful change.
Regards,
Himanshu Goyal
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Thursday, 24 January 2019 at 12:07 AM To: "Hu, Yong" <yong.hu@intel.com> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, Yes, virtualization setting are enabled in BIOS. Below are the dmesg logs for IOMMU [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.36.tis.x86_64 root=UUID=8c9ba8fa-ecae-4d66-98ac-a77fb66faae2 ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 hugepagesz=1G hugepages=2 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on user_namespace.enable=1 hugepagesz=2M hugepages=0 default_hugepagesz=2M isolcpus=1,2 rcu_nocbs=1-35 kthread_cpus=0 irqaffinity=0 nopti nospectre_v2 [ 0.000000] audit: disabled (until reboot) [ 0.000000] DMAR: IOMMU enabled [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 7:43 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Hi Himanshu, For compute/worker node, please make sure Virtualization settings (such as VT-X and VT-D) ENABLED in BIOS. They are mandatory requirements for compute node. Regards, Yong From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Wednesday, 23 January 2019 at 6:33 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Juan and Yong, Able to see compute host in "system-host-list" after connecting with a hub. starlingX Compute installation has been done. But after unlocking the compute nodes the compute node come into an endless boot loop. dmesg log shows below error: [ 20.967137] iTCO_vendor_support: vendor-support=0 [ 20.968785] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 20.968840] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 21.066607] device-mapper: uevent: version 1.0.3 [ 21.067124] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com<mailto:dm-devel@redhat.com> Checked the BIOS setting those are as mentioned in installation document. I'm using ISO Image available at below path: http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou... Regards, Himanshu Goyal On Tue, Jan 22, 2019 at 12:43 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Himanshu, Could you have a try with a hub which 2 mgt ports (from controller and compute) are plugged into? Let’s assure the normal setup works first, and then figure out why the direct linkage of cable doesn’t work. BTW: “worker” and “compute” are just different “personality” names in different STX version. On your current setup “compute” will do, supposedly. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Tuesday, 22 January 2019 at 12:32 AM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan ,Yong I tried both the commands output shows as below: 1) [wrsroot@controller-0 ~(keystone_admin)]$ system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>] [-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>] [-T <bm_type>] [-U <bm_username>] [-P <bm_password>] [-b <boot_device>] [-r <rootfs_device>] [-o <install_output>] [-c <console>] [-v <vsc_controllers>] [-l <location>] [-D <true/false>] system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile') [wrsroot@controller-0 ~(keystone_admin)]$ [wrsroot@controller-0 ~(keystone_admin)]$ 2) [wrsroot@controller-0 ~(keystone_admin)]$ system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. [wrsroot@controller-0 ~(keystone_admin)]$ Regards, Himanshu Goyal On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> wrote: Hi, The personality of computes changed to “worker”, so the command should be: system host-add -n compute-0 -p worker -m ${mac_address} Regards. Juan Carlos Alonso From: Hu, Yong Sent: Monday, January 21, 2019 8:51 AM To: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>>; Alonso, Juan Carlos <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option Hi Himanshu, “system host-list” doesn’t “see” your compute node and LLDP won’t work, because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub). Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd: # system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac> Regards, yong From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Monday, 21 January 2019 at 8:14 PM To: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option Thanks Juan, Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "system host-list" command. my controller machine is directly connected to compute machine. I'm following the below steps Steps: 1) system host-unlock controller-0 2) system host-list Output:: [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | +----+--------------+-------------+----------------+-------------+--------------+ 3) power on my compute machine. And give option to boot from PXE my compute machine is directly connected with controller with mgmt port. But not able to see host in "system host-list". 4) i tried with system host-add command also, but it is giving below error: Error: [wrsroot@controller-0 ~(keystone_admin)]$ system host-add Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured. Please suggest me the needful change. Regards, Himanshu Goyal
Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 12:07 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
Yes, virtualization setting are enabled in BIOS.
Below are the dmesg logs for IOMMU
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.36.tis.x86_64 root=UUID=8c9ba8fa-ecae-4d66-98ac-a77fb66faae2 ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 hugepagesz=1G hugepages=2 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1* intel_iommu=on *user_namespace.enable=1 hugepagesz=2M hugepages=0 default_hugepagesz=2M isolcpus=1,2 rcu_nocbs=1-35 kthread_cpus=0 irqaffinity=0 nopti nospectre_v2
[ 0.000000] audit: disabled (until reboot)
*[ 0.000000] DMAR: IOMMU enabled*
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 7:43 PM Hu, Yong <yong.hu@intel.com> wrote:
Hi Himanshu,
For compute/worker node, please make sure Virtualization settings (such as VT-X and VT-D) ENABLED in BIOS.
They are mandatory requirements for compute node.
Regards,
Yong
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Wednesday, 23 January 2019 at 6:33 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Juan and Yong,
Able to see compute host in "system-host-list" after connecting with a hub. starlingX Compute installation has been done.
But after unlocking the compute nodes the compute node come into an endless boot loop.
dmesg log shows below error:
[ 20.967137] iTCO_vendor_support: vendor-support=0 [ 20.968785] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 20.968840] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 21.066607] device-mapper: uevent: version 1.0.3 [ 21.067124] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com
Checked the BIOS setting those are as mentioned in installation document.
I'm using ISO Image available at below path:
http://mirror.starlingx.cengn.ca/mirror/starlingx/centos/2018.10/20181110/ou...
Regards,
Himanshu Goyal
On Tue, Jan 22, 2019 at 12:43 PM Hu, Yong <yong.hu@intel.com> wrote:
Himanshu,
Could you have a try with a hub which 2 mgt ports (from controller and compute) are plugged into?
Let’s assure the normal setup works first, and then figure out why the direct linkage of cable doesn’t work.
BTW: “worker” and “compute” are just different “personality” names in different STX version.
On your current setup “compute” will do, supposedly.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Tuesday, 22 January 2019 at 12:32 AM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan ,Yong
I tried both the commands output shows as below:
1) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p worker -m 00:1e:67:fd:3d:fe*
usage: system host-add [-n <hostname>] [-p <personality>] [-s <subfunctions>]
[-m <mgmt_mac>] [-i <mgmt_ip>] [-I <bm_ip>]
[-T <bm_type>] [-U <bm_username>] [-P <bm_password>]
[-b <boot_device>] [-r <rootfs_device>]
[-o <install_output>] [-c <console>]
[-v <vsc_controllers>] [-l <location>]
[-D <true/false>]
system host-add: error: argument -p/--personality: invalid choice: 'worker' (choose from 'controller', 'compute', 'storage', 'network', 'profile')
[wrsroot@controller-0 ~(keystone_admin)]$
[wrsroot@controller-0 ~(keystone_admin)]$
2) [wrsroot@controller-0 ~(keystone_admin)]$ *system host-add -n compute-0 -p compute -m 00:1e:67:fd:3d:fe*
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
[wrsroot@controller-0 ~(keystone_admin)]$
Regards,
Himanshu Goyal
On Mon, Jan 21, 2019 at 8:42 PM Alonso, Juan Carlos < juan.carlos.alonso@intel.com> wrote:
Hi,
The personality of computes changed to “worker”, so the command should be:
system host-add -n compute-0 -p worker -m ${mac_address}
Regards.
Juan Carlos Alonso
*From:* Hu, Yong *Sent:* Monday, January 21, 2019 8:51 AM *To:* Himanshu Goyal <himanshugoyal500@gmail.com>; Alonso, Juan Carlos < juan.carlos.alonso@intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option
Hi Himanshu,
“system host-list” doesn’t “see” your compute node and LLDP won’t work,
because the mgt port on compute node directly connects to mgt port on controller-0 (rather than both connecting to a hub).
Anyway, given you know the MAC of mgt port on compute node, you can have a try to run the following cmd:
# system host-add -n compute-0 -p compute -m <compute_node_mgt_port_mac>
Regards,
yong
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Monday, 21 January 2019 at 8:14 PM *To: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option
Thanks Juan,
Able to unlock my controller node. But facing Issue in PXE boot of compute node. After unlocking of controller machine not able to see compute host in "*system host-list*" command.
my controller machine is directly connected to compute machine.
I'm following the below steps
Steps:
*1) system host-unlock controller-0*
*2) system host-list*
Output::
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
3) power on my compute machine. And give option to boot from PXE
my compute machine is directly connected with controller with mgmt port.
But not able to see host in "system host-list".
4) i tried with system host-add command also, but it is giving below error:
*Error:*
[wrsroot@controller-0 ~(keystone_admin)]$ system host-add
Host-add Rejected: Cannot add a compute host without specifying a mgmt_ip when static address allocation is configured.
Please suggest me the needful change.
Regards,
Himanshu Goyal
I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# *ovs-vsctl show output:* compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" * Interface "eth0"* type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} *error: "Error attaching device '0000:02:00.0' to DPDK"* Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## *Output of dpdk-devbind.py:* compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. *lspci output:* *##################################################* compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com> Cc: "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Peters, Output of command : #################################################### compute-0:~$ dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 000000007b69d000 000E0 (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.298584] DMAR: Host address width 46 [ 0.302872] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.308808] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.317647] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.323581] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.332427] DMAR: RMRR base: 0x0000007a3dd000 end: 0x0000007a3dffff [ 0.339426] DMAR: ATSR flags: 0x0 [ 11.329277] DMAR: dmar0: Using Queued invalidation [ 11.334645] DMAR: dmar1: Using Queued invalidation [ 11.342303] DMAR: Hardware identity mapping for device 0000:ff:08.0 [ 11.349304] DMAR: Hardware identity mapping for device 0000:ff:08.2 [ 11.356299] DMAR: Hardware identity mapping for device 0000:ff:08.3 [ 11.363296] DMAR: Hardware identity mapping for device 0000:ff:09.0 [ 11.370291] DMAR: Hardware identity mapping for device 0000:ff:09.2 [ 11.377286] DMAR: Hardware identity mapping for device 0000:ff:09.3 [ 11.384281] DMAR: Hardware identity mapping for device 0000:ff:0b.0 [ 11.391275] DMAR: Hardware identity mapping for device 0000:ff:0b.1 [ 11.398270] DMAR: Hardware identity mapping for device 0000:ff:0b.2 [ 11.405265] DMAR: Hardware identity mapping for device 0000:ff:0c.0 [ 11.412260] DMAR: Hardware identity mapping for device 0000:ff:0c.1 [ 11.419255] DMAR: Hardware identity mapping for device 0000:ff:0c.2 [ 11.426250] DMAR: Hardware identity mapping for device 0000:ff:0c.3 [ 11.433244] DMAR: Hardware identity mapping for device 0000:ff:0c.4 [ 11.440239] DMAR: Hardware identity mapping for device 0000:ff:0c.5 [ 11.447234] DMAR: Hardware identity mapping for device 0000:ff:0c.6 [ 11.454229] DMAR: Hardware identity mapping for device 0000:ff:0c.7 [ 11.461223] DMAR: Hardware identity mapping for device 0000:ff:0d.0 [ 11.468218] DMAR: Hardware identity mapping for device 0000:ff:0d.1 [ 11.475212] DMAR: Hardware identity mapping for device 0000:ff:0d.2 [ 11.482208] DMAR: Hardware identity mapping for device 0000:ff:0d.3 [ 11.489202] DMAR: Hardware identity mapping for device 0000:ff:0d.4 [ 11.496197] DMAR: Hardware identity mapping for device 0000:ff:0d.5 [ 11.503192] DMAR: Hardware identity mapping for device 0000:ff:0d.6 [ 11.510186] DMAR: Hardware identity mapping for device 0000:ff:0d.7 [ 11.517181] DMAR: Hardware identity mapping for device 0000:ff:0e.0 [ 11.524177] DMAR: Hardware identity mapping for device 0000:ff:0e.1 [ 11.531172] DMAR: Hardware identity mapping for device 0000:ff:0f.0 [ 11.538166] DMAR: Hardware identity mapping for device 0000:ff:0f.1 [ 11.545160] DMAR: Hardware identity mapping for device 0000:ff:0f.2 [ 11.552155] DMAR: Hardware identity mapping for device 0000:ff:0f.3 [ 11.559149] DMAR: Hardware identity mapping for device 0000:ff:0f.4 [ 11.566144] DMAR: Hardware identity mapping for device 0000:ff:0f.5 [ 11.573138] DMAR: Hardware identity mapping for device 0000:ff:0f.6 [ 11.580133] DMAR: Hardware identity mapping for device 0000:ff:10.0 [ 11.587128] DMAR: Hardware identity mapping for device 0000:ff:10.1 [ 11.594123] DMAR: Hardware identity mapping for device 0000:ff:10.5 [ 11.601117] DMAR: Hardware identity mapping for device 0000:ff:10.6 [ 11.608111] DMAR: Hardware identity mapping for device 0000:ff:10.7 [ 11.615106] DMAR: Hardware identity mapping for device 0000:ff:12.0 [ 11.622101] DMAR: Hardware identity mapping for device 0000:ff:12.1 [ 11.629095] DMAR: Hardware identity mapping for device 0000:ff:12.4 [ 11.636090] DMAR: Hardware identity mapping for device 0000:ff:12.5 [ 11.643085] DMAR: Hardware identity mapping for device 0000:ff:13.0 [ 11.650079] DMAR: Hardware identity mapping for device 0000:ff:13.1 [ 11.657074] DMAR: Hardware identity mapping for device 0000:ff:13.2 [ 11.664069] DMAR: Hardware identity mapping for device 0000:ff:13.3 [ 11.671063] DMAR: Hardware identity mapping for device 0000:ff:13.6 [ 11.678058] DMAR: Hardware identity mapping for device 0000:ff:13.7 [ 11.685052] DMAR: Hardware identity mapping for device 0000:ff:14.0 [ 11.692047] DMAR: Hardware identity mapping for device 0000:ff:14.1 [ 11.699044] DMAR: Hardware identity mapping for device 0000:ff:14.2 [ 11.706039] DMAR: Hardware identity mapping for device 0000:ff:14.3 [ 11.713034] DMAR: Hardware identity mapping for device 0000:ff:14.4 [ 11.720028] DMAR: Hardware identity mapping for device 0000:ff:14.5 [ 11.727023] DMAR: Hardware identity mapping for device 0000:ff:14.6 [ 11.734018] DMAR: Hardware identity mapping for device 0000:ff:14.7 [ 11.741012] DMAR: Hardware identity mapping for device 0000:ff:16.0 [ 11.748008] DMAR: Hardware identity mapping for device 0000:ff:16.1 [ 11.755002] DMAR: Hardware identity mapping for device 0000:ff:16.2 [ 11.761997] DMAR: Hardware identity mapping for device 0000:ff:16.3 [ 11.768992] DMAR: Hardware identity mapping for device 0000:ff:16.6 [ 11.775987] DMAR: Hardware identity mapping for device 0000:ff:16.7 [ 11.782981] DMAR: Hardware identity mapping for device 0000:ff:17.0 [ 11.789976] DMAR: Hardware identity mapping for device 0000:ff:17.1 [ 11.796971] DMAR: Hardware identity mapping for device 0000:ff:17.2 [ 11.803966] DMAR: Hardware identity mapping for device 0000:ff:17.3 [ 11.810960] DMAR: Hardware identity mapping for device 0000:ff:17.4 [ 11.817955] DMAR: Hardware identity mapping for device 0000:ff:17.5 [ 11.824949] DMAR: Hardware identity mapping for device 0000:ff:17.6 [ 11.831944] DMAR: Hardware identity mapping for device 0000:ff:17.7 [ 11.838939] DMAR: Hardware identity mapping for device 0000:ff:1e.0 [ 11.845933] DMAR: Hardware identity mapping for device 0000:ff:1e.1 [ 11.852928] DMAR: Hardware identity mapping for device 0000:ff:1e.2 [ 11.859923] DMAR: Hardware identity mapping for device 0000:ff:1e.3 [ 11.866918] DMAR: Hardware identity mapping for device 0000:ff:1e.4 [ 11.873913] DMAR: Hardware identity mapping for device 0000:ff:1f.0 [ 11.880908] DMAR: Hardware identity mapping for device 0000:ff:1f.2 [ 11.887905] DMAR: Hardware identity mapping for device 0000:7f:08.0 [ 11.894899] DMAR: Hardware identity mapping for device 0000:7f:08.2 [ 11.901895] DMAR: Hardware identity mapping for device 0000:7f:08.3 [ 11.908890] DMAR: Hardware identity mapping for device 0000:7f:09.0 [ 11.915885] DMAR: Hardware identity mapping for device 0000:7f:09.2 [ 11.922879] DMAR: Hardware identity mapping for device 0000:7f:09.3 [ 11.929875] DMAR: Hardware identity mapping for device 0000:7f:0b.0 [ 11.936870] DMAR: Hardware identity mapping for device 0000:7f:0b.1 [ 11.943865] DMAR: Hardware identity mapping for device 0000:7f:0b.2 [ 11.950860] DMAR: Hardware identity mapping for device 0000:7f:0c.0 [ 11.957855] DMAR: Hardware identity mapping for device 0000:7f:0c.1 [ 11.964849] DMAR: Hardware identity mapping for device 0000:7f:0c.2 [ 11.971844] DMAR: Hardware identity mapping for device 0000:7f:0c.3 [ 11.978839] DMAR: Hardware identity mapping for device 0000:7f:0c.4 [ 11.985833] DMAR: Hardware identity mapping for device 0000:7f:0c.5 [ 11.992828] DMAR: Hardware identity mapping for device 0000:7f:0c.6 [ 11.999823] DMAR: Hardware identity mapping for device 0000:7f:0c.7 [ 12.006818] DMAR: Hardware identity mapping for device 0000:7f:0d.0 [ 12.013814] DMAR: Hardware identity mapping for device 0000:7f:0d.1 [ 12.020801] DMAR: Hardware identity mapping for device 0000:7f:0d.2 [ 12.027796] DMAR: Hardware identity mapping for device 0000:7f:0d.3 [ 12.034790] DMAR: Hardware identity mapping for device 0000:7f:0d.4 [ 12.041785] DMAR: Hardware identity mapping for device 0000:7f:0d.5 [ 12.048781] DMAR: Hardware identity mapping for device 0000:7f:0d.6 [ 12.055776] DMAR: Hardware identity mapping for device 0000:7f:0d.7 [ 12.062771] DMAR: Hardware identity mapping for device 0000:7f:0e.0 [ 12.069766] DMAR: Hardware identity mapping for device 0000:7f:0e.1 [ 12.076761] DMAR: Hardware identity mapping for device 0000:7f:0f.0 [ 12.083756] DMAR: Hardware identity mapping for device 0000:7f:0f.1 [ 12.090752] DMAR: Hardware identity mapping for device 0000:7f:0f.2 [ 12.097747] DMAR: Hardware identity mapping for device 0000:7f:0f.3 [ 12.104741] DMAR: Hardware identity mapping for device 0000:7f:0f.4 [ 12.111736] DMAR: Hardware identity mapping for device 0000:7f:0f.5 [ 12.118731] DMAR: Hardware identity mapping for device 0000:7f:0f.6 [ 12.125726] DMAR: Hardware identity mapping for device 0000:7f:10.0 [ 12.132721] DMAR: Hardware identity mapping for device 0000:7f:10.1 [ 12.139716] DMAR: Hardware identity mapping for device 0000:7f:10.5 [ 12.146713] DMAR: Hardware identity mapping for device 0000:7f:10.6 [ 12.153708] DMAR: Hardware identity mapping for device 0000:7f:10.7 [ 12.160703] DMAR: Hardware identity mapping for device 0000:7f:12.0 [ 12.167698] DMAR: Hardware identity mapping for device 0000:7f:12.1 [ 12.174693] DMAR: Hardware identity mapping for device 0000:7f:12.4 [ 12.181688] DMAR: Hardware identity mapping for device 0000:7f:12.5 [ 12.188683] DMAR: Hardware identity mapping for device 0000:7f:13.0 [ 12.195678] DMAR: Hardware identity mapping for device 0000:7f:13.1 [ 12.202673] DMAR: Hardware identity mapping for device 0000:7f:13.2 [ 12.209668] DMAR: Hardware identity mapping for device 0000:7f:13.3 [ 12.216663] DMAR: Hardware identity mapping for device 0000:7f:13.6 [ 12.223658] DMAR: Hardware identity mapping for device 0000:7f:13.7 [ 12.230653] DMAR: Hardware identity mapping for device 0000:7f:14.0 [ 12.237648] DMAR: Hardware identity mapping for device 0000:7f:14.1 [ 12.244643] DMAR: Hardware identity mapping for device 0000:7f:14.2 [ 12.251638] DMAR: Hardware identity mapping for device 0000:7f:14.3 [ 12.258634] DMAR: Hardware identity mapping for device 0000:7f:14.4 [ 12.265629] DMAR: Hardware identity mapping for device 0000:7f:14.5 [ 12.272624] DMAR: Hardware identity mapping for device 0000:7f:14.6 [ 12.279620] DMAR: Hardware identity mapping for device 0000:7f:14.7 [ 12.286615] DMAR: Hardware identity mapping for device 0000:7f:16.0 [ 12.293610] DMAR: Hardware identity mapping for device 0000:7f:16.1 [ 12.300605] DMAR: Hardware identity mapping for device 0000:7f:16.2 [ 12.307600] DMAR: Hardware identity mapping for device 0000:7f:16.3 [ 12.314595] DMAR: Hardware identity mapping for device 0000:7f:16.6 [ 12.321590] DMAR: Hardware identity mapping for device 0000:7f:16.7 [ 12.328585] DMAR: Hardware identity mapping for device 0000:7f:17.0 [ 12.335580] DMAR: Hardware identity mapping for device 0000:7f:17.1 [ 12.342575] DMAR: Hardware identity mapping for device 0000:7f:17.2 [ 12.349570] DMAR: Hardware identity mapping for device 0000:7f:17.3 [ 12.356565] DMAR: Hardware identity mapping for device 0000:7f:17.4 [ 12.363564] DMAR: Hardware identity mapping for device 0000:7f:17.5 [ 12.370563] DMAR: Hardware identity mapping for device 0000:7f:17.6 [ 12.377562] DMAR: Hardware identity mapping for device 0000:7f:17.7 [ 12.384561] DMAR: Hardware identity mapping for device 0000:7f:1e.0 [ 12.391560] DMAR: Hardware identity mapping for device 0000:7f:1e.1 [ 12.398555] DMAR: Hardware identity mapping for device 0000:7f:1e.2 [ 12.405550] DMAR: Hardware identity mapping for device 0000:7f:1e.3 [ 12.412545] DMAR: Hardware identity mapping for device 0000:7f:1e.4 [ 12.419541] DMAR: Hardware identity mapping for device 0000:7f:1f.0 [ 12.426536] DMAR: Hardware identity mapping for device 0000:7f:1f.2 [ 12.433533] DMAR: Hardware identity mapping for device 0000:00:00.0 [ 12.440529] DMAR: Hardware identity mapping for device 0000:00:01.0 [ 12.447523] DMAR: Hardware identity mapping for device 0000:00:02.0 [ 12.454519] DMAR: Hardware identity mapping for device 0000:00:02.2 [ 12.461514] DMAR: Hardware identity mapping for device 0000:00:03.0 [ 12.468509] DMAR: Hardware identity mapping for device 0000:00:04.0 [ 12.475504] DMAR: Hardware identity mapping for device 0000:00:04.1 [ 12.482499] DMAR: Hardware identity mapping for device 0000:00:04.2 [ 12.489495] DMAR: Hardware identity mapping for device 0000:00:04.3 [ 12.496490] DMAR: Hardware identity mapping for device 0000:00:04.4 [ 12.503486] DMAR: Hardware identity mapping for device 0000:00:04.5 [ 12.510481] DMAR: Hardware identity mapping for device 0000:00:04.6 [ 12.517479] DMAR: Hardware identity mapping for device 0000:00:04.7 [ 12.524474] DMAR: Hardware identity mapping for device 0000:00:05.0 [ 12.531469] DMAR: Hardware identity mapping for device 0000:00:05.1 [ 12.538465] DMAR: Hardware identity mapping for device 0000:00:05.2 [ 12.545461] DMAR: Hardware identity mapping for device 0000:00:05.4 [ 12.552456] DMAR: Hardware identity mapping for device 0000:00:11.0 [ 12.559451] DMAR: Hardware identity mapping for device 0000:00:11.1 [ 12.566449] DMAR: Hardware identity mapping for device 0000:00:11.4 [ 12.573444] DMAR: Hardware identity mapping for device 0000:00:14.0 [ 12.580440] DMAR: Hardware identity mapping for device 0000:00:16.0 [ 12.587436] DMAR: Hardware identity mapping for device 0000:00:16.1 [ 12.594433] DMAR: Hardware identity mapping for device 0000:00:1a.0 [ 12.601429] DMAR: Hardware identity mapping for device 0000:00:1c.0 [ 12.608424] DMAR: Hardware identity mapping for device 0000:00:1c.3 [ 12.615419] DMAR: Hardware identity mapping for device 0000:00:1d.0 [ 12.622415] DMAR: Hardware identity mapping for device 0000:00:1f.0 [ 12.629410] DMAR: Hardware identity mapping for device 0000:00:1f.2 [ 12.636405] DMAR: Hardware identity mapping for device 0000:00:1f.3 [ 12.643402] DMAR: Hardware identity mapping for device 0000:02:00.0 [ 12.650397] DMAR: Hardware identity mapping for device 0000:02:00.1 [ 12.657394] DMAR: Hardware identity mapping for device 0000:04:00.0 [ 12.664390] DMAR: Hardware identity mapping for device 0000:04:00.3 [ 12.671390] DMAR: Hardware identity mapping for device 0000:08:00.0 [ 12.678388] DMAR: Hardware identity mapping for device 0000:80:04.0 [ 12.685384] DMAR: Hardware identity mapping for device 0000:80:04.1 [ 12.692371] DMAR: Hardware identity mapping for device 0000:80:04.2 [ 12.699358] DMAR: Hardware identity mapping for device 0000:80:04.3 [ 12.706346] DMAR: Hardware identity mapping for device 0000:80:04.4 [ 12.713341] DMAR: Hardware identity mapping for device 0000:80:04.5 [ 12.720336] DMAR: Hardware identity mapping for device 0000:80:04.6 [ 12.727331] DMAR: Hardware identity mapping for device 0000:80:04.7 [ 12.734331] DMAR: Hardware identity mapping for device 0000:80:05.0 [ 12.741327] DMAR: Hardware identity mapping for device 0000:80:05.1 [ 12.748314] DMAR: Hardware identity mapping for device 0000:80:05.2 [ 12.755301] DMAR: Hardware identity mapping for device 0000:80:05.4 [ 12.762287] DMAR: Setting RMRR: [ 12.765793] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0x7a3dd000 - 0x7a3dffff] [ 12.776376] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x7a3dd000 - 0x7a3dffff] [ 12.786965] DMAR: Prepare 0-16MiB unity mapping for LPC [ 12.792797] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff] [ 12.802607] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 14.693008] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping [ 14.702309] DMAR: Setting identity map for device 0000:00:1a.0 [0x7a3dd000 - 0x7a3dffff] [ 14.762084] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping [ 14.770843] DMAR: Setting identity map for device 0000:00:1d.0 [0x7a3dd000 - 0x7a3dffff] compute-0:~$ compute-0:~$ #################################################################################### Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Sorry, can you do the command again with the -i dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Thursday, January 24, 2019 at 9:10 AM To: "Peters, Matt" <Matt.Peters@windriver.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Output of command : #################################################### compute-0:~$ dmesg | grep -e DMAR -e IOMMU [ 0.000000] ACPI: DMAR 000000007b69d000 000E0 (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.000000] DMAR: IOMMU enabled [ 0.298584] DMAR: Host address width 46 [ 0.302872] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.308808] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.317647] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.323581] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.332427] DMAR: RMRR base: 0x0000007a3dd000 end: 0x0000007a3dffff [ 0.339426] DMAR: ATSR flags: 0x0 [ 11.329277] DMAR: dmar0: Using Queued invalidation [ 11.334645] DMAR: dmar1: Using Queued invalidation [ 11.342303] DMAR: Hardware identity mapping for device 0000:ff:08.0 [ 11.349304] DMAR: Hardware identity mapping for device 0000:ff:08.2 [ 11.356299] DMAR: Hardware identity mapping for device 0000:ff:08.3 [ 11.363296] DMAR: Hardware identity mapping for device 0000:ff:09.0 [ 11.370291] DMAR: Hardware identity mapping for device 0000:ff:09.2 [ 11.377286] DMAR: Hardware identity mapping for device 0000:ff:09.3 [ 11.384281] DMAR: Hardware identity mapping for device 0000:ff:0b.0 [ 11.391275] DMAR: Hardware identity mapping for device 0000:ff:0b.1 [ 11.398270] DMAR: Hardware identity mapping for device 0000:ff:0b.2 [ 11.405265] DMAR: Hardware identity mapping for device 0000:ff:0c.0 [ 11.412260] DMAR: Hardware identity mapping for device 0000:ff:0c.1 [ 11.419255] DMAR: Hardware identity mapping for device 0000:ff:0c.2 [ 11.426250] DMAR: Hardware identity mapping for device 0000:ff:0c.3 [ 11.433244] DMAR: Hardware identity mapping for device 0000:ff:0c.4 [ 11.440239] DMAR: Hardware identity mapping for device 0000:ff:0c.5 [ 11.447234] DMAR: Hardware identity mapping for device 0000:ff:0c.6 [ 11.454229] DMAR: Hardware identity mapping for device 0000:ff:0c.7 [ 11.461223] DMAR: Hardware identity mapping for device 0000:ff:0d.0 [ 11.468218] DMAR: Hardware identity mapping for device 0000:ff:0d.1 [ 11.475212] DMAR: Hardware identity mapping for device 0000:ff:0d.2 [ 11.482208] DMAR: Hardware identity mapping for device 0000:ff:0d.3 [ 11.489202] DMAR: Hardware identity mapping for device 0000:ff:0d.4 [ 11.496197] DMAR: Hardware identity mapping for device 0000:ff:0d.5 [ 11.503192] DMAR: Hardware identity mapping for device 0000:ff:0d.6 [ 11.510186] DMAR: Hardware identity mapping for device 0000:ff:0d.7 [ 11.517181] DMAR: Hardware identity mapping for device 0000:ff:0e.0 [ 11.524177] DMAR: Hardware identity mapping for device 0000:ff:0e.1 [ 11.531172] DMAR: Hardware identity mapping for device 0000:ff:0f.0 [ 11.538166] DMAR: Hardware identity mapping for device 0000:ff:0f.1 [ 11.545160] DMAR: Hardware identity mapping for device 0000:ff:0f.2 [ 11.552155] DMAR: Hardware identity mapping for device 0000:ff:0f.3 [ 11.559149] DMAR: Hardware identity mapping for device 0000:ff:0f.4 [ 11.566144] DMAR: Hardware identity mapping for device 0000:ff:0f.5 [ 11.573138] DMAR: Hardware identity mapping for device 0000:ff:0f.6 [ 11.580133] DMAR: Hardware identity mapping for device 0000:ff:10.0 [ 11.587128] DMAR: Hardware identity mapping for device 0000:ff:10.1 [ 11.594123] DMAR: Hardware identity mapping for device 0000:ff:10.5 [ 11.601117] DMAR: Hardware identity mapping for device 0000:ff:10.6 [ 11.608111] DMAR: Hardware identity mapping for device 0000:ff:10.7 [ 11.615106] DMAR: Hardware identity mapping for device 0000:ff:12.0 [ 11.622101] DMAR: Hardware identity mapping for device 0000:ff:12.1 [ 11.629095] DMAR: Hardware identity mapping for device 0000:ff:12.4 [ 11.636090] DMAR: Hardware identity mapping for device 0000:ff:12.5 [ 11.643085] DMAR: Hardware identity mapping for device 0000:ff:13.0 [ 11.650079] DMAR: Hardware identity mapping for device 0000:ff:13.1 [ 11.657074] DMAR: Hardware identity mapping for device 0000:ff:13.2 [ 11.664069] DMAR: Hardware identity mapping for device 0000:ff:13.3 [ 11.671063] DMAR: Hardware identity mapping for device 0000:ff:13.6 [ 11.678058] DMAR: Hardware identity mapping for device 0000:ff:13.7 [ 11.685052] DMAR: Hardware identity mapping for device 0000:ff:14.0 [ 11.692047] DMAR: Hardware identity mapping for device 0000:ff:14.1 [ 11.699044] DMAR: Hardware identity mapping for device 0000:ff:14.2 [ 11.706039] DMAR: Hardware identity mapping for device 0000:ff:14.3 [ 11.713034] DMAR: Hardware identity mapping for device 0000:ff:14.4 [ 11.720028] DMAR: Hardware identity mapping for device 0000:ff:14.5 [ 11.727023] DMAR: Hardware identity mapping for device 0000:ff:14.6 [ 11.734018] DMAR: Hardware identity mapping for device 0000:ff:14.7 [ 11.741012] DMAR: Hardware identity mapping for device 0000:ff:16.0 [ 11.748008] DMAR: Hardware identity mapping for device 0000:ff:16.1 [ 11.755002] DMAR: Hardware identity mapping for device 0000:ff:16.2 [ 11.761997] DMAR: Hardware identity mapping for device 0000:ff:16.3 [ 11.768992] DMAR: Hardware identity mapping for device 0000:ff:16.6 [ 11.775987] DMAR: Hardware identity mapping for device 0000:ff:16.7 [ 11.782981] DMAR: Hardware identity mapping for device 0000:ff:17.0 [ 11.789976] DMAR: Hardware identity mapping for device 0000:ff:17.1 [ 11.796971] DMAR: Hardware identity mapping for device 0000:ff:17.2 [ 11.803966] DMAR: Hardware identity mapping for device 0000:ff:17.3 [ 11.810960] DMAR: Hardware identity mapping for device 0000:ff:17.4 [ 11.817955] DMAR: Hardware identity mapping for device 0000:ff:17.5 [ 11.824949] DMAR: Hardware identity mapping for device 0000:ff:17.6 [ 11.831944] DMAR: Hardware identity mapping for device 0000:ff:17.7 [ 11.838939] DMAR: Hardware identity mapping for device 0000:ff:1e.0 [ 11.845933] DMAR: Hardware identity mapping for device 0000:ff:1e.1 [ 11.852928] DMAR: Hardware identity mapping for device 0000:ff:1e.2 [ 11.859923] DMAR: Hardware identity mapping for device 0000:ff:1e.3 [ 11.866918] DMAR: Hardware identity mapping for device 0000:ff:1e.4 [ 11.873913] DMAR: Hardware identity mapping for device 0000:ff:1f.0 [ 11.880908] DMAR: Hardware identity mapping for device 0000:ff:1f.2 [ 11.887905] DMAR: Hardware identity mapping for device 0000:7f:08.0 [ 11.894899] DMAR: Hardware identity mapping for device 0000:7f:08.2 [ 11.901895] DMAR: Hardware identity mapping for device 0000:7f:08.3 [ 11.908890] DMAR: Hardware identity mapping for device 0000:7f:09.0 [ 11.915885] DMAR: Hardware identity mapping for device 0000:7f:09.2 [ 11.922879] DMAR: Hardware identity mapping for device 0000:7f:09.3 [ 11.929875] DMAR: Hardware identity mapping for device 0000:7f:0b.0 [ 11.936870] DMAR: Hardware identity mapping for device 0000:7f:0b.1 [ 11.943865] DMAR: Hardware identity mapping for device 0000:7f:0b.2 [ 11.950860] DMAR: Hardware identity mapping for device 0000:7f:0c.0 [ 11.957855] DMAR: Hardware identity mapping for device 0000:7f:0c.1 [ 11.964849] DMAR: Hardware identity mapping for device 0000:7f:0c.2 [ 11.971844] DMAR: Hardware identity mapping for device 0000:7f:0c.3 [ 11.978839] DMAR: Hardware identity mapping for device 0000:7f:0c.4 [ 11.985833] DMAR: Hardware identity mapping for device 0000:7f:0c.5 [ 11.992828] DMAR: Hardware identity mapping for device 0000:7f:0c.6 [ 11.999823] DMAR: Hardware identity mapping for device 0000:7f:0c.7 [ 12.006818] DMAR: Hardware identity mapping for device 0000:7f:0d.0 [ 12.013814] DMAR: Hardware identity mapping for device 0000:7f:0d.1 [ 12.020801] DMAR: Hardware identity mapping for device 0000:7f:0d.2 [ 12.027796] DMAR: Hardware identity mapping for device 0000:7f:0d.3 [ 12.034790] DMAR: Hardware identity mapping for device 0000:7f:0d.4 [ 12.041785] DMAR: Hardware identity mapping for device 0000:7f:0d.5 [ 12.048781] DMAR: Hardware identity mapping for device 0000:7f:0d.6 [ 12.055776] DMAR: Hardware identity mapping for device 0000:7f:0d.7 [ 12.062771] DMAR: Hardware identity mapping for device 0000:7f:0e.0 [ 12.069766] DMAR: Hardware identity mapping for device 0000:7f:0e.1 [ 12.076761] DMAR: Hardware identity mapping for device 0000:7f:0f.0 [ 12.083756] DMAR: Hardware identity mapping for device 0000:7f:0f.1 [ 12.090752] DMAR: Hardware identity mapping for device 0000:7f:0f.2 [ 12.097747] DMAR: Hardware identity mapping for device 0000:7f:0f.3 [ 12.104741] DMAR: Hardware identity mapping for device 0000:7f:0f.4 [ 12.111736] DMAR: Hardware identity mapping for device 0000:7f:0f.5 [ 12.118731] DMAR: Hardware identity mapping for device 0000:7f:0f.6 [ 12.125726] DMAR: Hardware identity mapping for device 0000:7f:10.0 [ 12.132721] DMAR: Hardware identity mapping for device 0000:7f:10.1 [ 12.139716] DMAR: Hardware identity mapping for device 0000:7f:10.5 [ 12.146713] DMAR: Hardware identity mapping for device 0000:7f:10.6 [ 12.153708] DMAR: Hardware identity mapping for device 0000:7f:10.7 [ 12.160703] DMAR: Hardware identity mapping for device 0000:7f:12.0 [ 12.167698] DMAR: Hardware identity mapping for device 0000:7f:12.1 [ 12.174693] DMAR: Hardware identity mapping for device 0000:7f:12.4 [ 12.181688] DMAR: Hardware identity mapping for device 0000:7f:12.5 [ 12.188683] DMAR: Hardware identity mapping for device 0000:7f:13.0 [ 12.195678] DMAR: Hardware identity mapping for device 0000:7f:13.1 [ 12.202673] DMAR: Hardware identity mapping for device 0000:7f:13.2 [ 12.209668] DMAR: Hardware identity mapping for device 0000:7f:13.3 [ 12.216663] DMAR: Hardware identity mapping for device 0000:7f:13.6 [ 12.223658] DMAR: Hardware identity mapping for device 0000:7f:13.7 [ 12.230653] DMAR: Hardware identity mapping for device 0000:7f:14.0 [ 12.237648] DMAR: Hardware identity mapping for device 0000:7f:14.1 [ 12.244643] DMAR: Hardware identity mapping for device 0000:7f:14.2 [ 12.251638] DMAR: Hardware identity mapping for device 0000:7f:14.3 [ 12.258634] DMAR: Hardware identity mapping for device 0000:7f:14.4 [ 12.265629] DMAR: Hardware identity mapping for device 0000:7f:14.5 [ 12.272624] DMAR: Hardware identity mapping for device 0000:7f:14.6 [ 12.279620] DMAR: Hardware identity mapping for device 0000:7f:14.7 [ 12.286615] DMAR: Hardware identity mapping for device 0000:7f:16.0 [ 12.293610] DMAR: Hardware identity mapping for device 0000:7f:16.1 [ 12.300605] DMAR: Hardware identity mapping for device 0000:7f:16.2 [ 12.307600] DMAR: Hardware identity mapping for device 0000:7f:16.3 [ 12.314595] DMAR: Hardware identity mapping for device 0000:7f:16.6 [ 12.321590] DMAR: Hardware identity mapping for device 0000:7f:16.7 [ 12.328585] DMAR: Hardware identity mapping for device 0000:7f:17.0 [ 12.335580] DMAR: Hardware identity mapping for device 0000:7f:17.1 [ 12.342575] DMAR: Hardware identity mapping for device 0000:7f:17.2 [ 12.349570] DMAR: Hardware identity mapping for device 0000:7f:17.3 [ 12.356565] DMAR: Hardware identity mapping for device 0000:7f:17.4 [ 12.363564] DMAR: Hardware identity mapping for device 0000:7f:17.5 [ 12.370563] DMAR: Hardware identity mapping for device 0000:7f:17.6 [ 12.377562] DMAR: Hardware identity mapping for device 0000:7f:17.7 [ 12.384561] DMAR: Hardware identity mapping for device 0000:7f:1e.0 [ 12.391560] DMAR: Hardware identity mapping for device 0000:7f:1e.1 [ 12.398555] DMAR: Hardware identity mapping for device 0000:7f:1e.2 [ 12.405550] DMAR: Hardware identity mapping for device 0000:7f:1e.3 [ 12.412545] DMAR: Hardware identity mapping for device 0000:7f:1e.4 [ 12.419541] DMAR: Hardware identity mapping for device 0000:7f:1f.0 [ 12.426536] DMAR: Hardware identity mapping for device 0000:7f:1f.2 [ 12.433533] DMAR: Hardware identity mapping for device 0000:00:00.0 [ 12.440529] DMAR: Hardware identity mapping for device 0000:00:01.0 [ 12.447523] DMAR: Hardware identity mapping for device 0000:00:02.0 [ 12.454519] DMAR: Hardware identity mapping for device 0000:00:02.2 [ 12.461514] DMAR: Hardware identity mapping for device 0000:00:03.0 [ 12.468509] DMAR: Hardware identity mapping for device 0000:00:04.0 [ 12.475504] DMAR: Hardware identity mapping for device 0000:00:04.1 [ 12.482499] DMAR: Hardware identity mapping for device 0000:00:04.2 [ 12.489495] DMAR: Hardware identity mapping for device 0000:00:04.3 [ 12.496490] DMAR: Hardware identity mapping for device 0000:00:04.4 [ 12.503486] DMAR: Hardware identity mapping for device 0000:00:04.5 [ 12.510481] DMAR: Hardware identity mapping for device 0000:00:04.6 [ 12.517479] DMAR: Hardware identity mapping for device 0000:00:04.7 [ 12.524474] DMAR: Hardware identity mapping for device 0000:00:05.0 [ 12.531469] DMAR: Hardware identity mapping for device 0000:00:05.1 [ 12.538465] DMAR: Hardware identity mapping for device 0000:00:05.2 [ 12.545461] DMAR: Hardware identity mapping for device 0000:00:05.4 [ 12.552456] DMAR: Hardware identity mapping for device 0000:00:11.0 [ 12.559451] DMAR: Hardware identity mapping for device 0000:00:11.1 [ 12.566449] DMAR: Hardware identity mapping for device 0000:00:11.4 [ 12.573444] DMAR: Hardware identity mapping for device 0000:00:14.0 [ 12.580440] DMAR: Hardware identity mapping for device 0000:00:16.0 [ 12.587436] DMAR: Hardware identity mapping for device 0000:00:16.1 [ 12.594433] DMAR: Hardware identity mapping for device 0000:00:1a.0 [ 12.601429] DMAR: Hardware identity mapping for device 0000:00:1c.0 [ 12.608424] DMAR: Hardware identity mapping for device 0000:00:1c.3 [ 12.615419] DMAR: Hardware identity mapping for device 0000:00:1d.0 [ 12.622415] DMAR: Hardware identity mapping for device 0000:00:1f.0 [ 12.629410] DMAR: Hardware identity mapping for device 0000:00:1f.2 [ 12.636405] DMAR: Hardware identity mapping for device 0000:00:1f.3 [ 12.643402] DMAR: Hardware identity mapping for device 0000:02:00.0 [ 12.650397] DMAR: Hardware identity mapping for device 0000:02:00.1 [ 12.657394] DMAR: Hardware identity mapping for device 0000:04:00.0 [ 12.664390] DMAR: Hardware identity mapping for device 0000:04:00.3 [ 12.671390] DMAR: Hardware identity mapping for device 0000:08:00.0 [ 12.678388] DMAR: Hardware identity mapping for device 0000:80:04.0 [ 12.685384] DMAR: Hardware identity mapping for device 0000:80:04.1 [ 12.692371] DMAR: Hardware identity mapping for device 0000:80:04.2 [ 12.699358] DMAR: Hardware identity mapping for device 0000:80:04.3 [ 12.706346] DMAR: Hardware identity mapping for device 0000:80:04.4 [ 12.713341] DMAR: Hardware identity mapping for device 0000:80:04.5 [ 12.720336] DMAR: Hardware identity mapping for device 0000:80:04.6 [ 12.727331] DMAR: Hardware identity mapping for device 0000:80:04.7 [ 12.734331] DMAR: Hardware identity mapping for device 0000:80:05.0 [ 12.741327] DMAR: Hardware identity mapping for device 0000:80:05.1 [ 12.748314] DMAR: Hardware identity mapping for device 0000:80:05.2 [ 12.755301] DMAR: Hardware identity mapping for device 0000:80:05.4 [ 12.762287] DMAR: Setting RMRR: [ 12.765793] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0x7a3dd000 - 0x7a3dffff] [ 12.776376] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x7a3dd000 - 0x7a3dffff] [ 12.786965] DMAR: Prepare 0-16MiB unity mapping for LPC [ 12.792797] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff] [ 12.802607] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 14.693008] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping [ 14.702309] DMAR: Setting identity map for device 0000:00:1a.0 [0x7a3dd000 - 0x7a3dffff] [ 14.762084] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping [ 14.770843] DMAR: Setting identity map for device 0000:00:1d.0 [0x7a3dd000 - 0x7a3dffff] compute-0:~$ compute-0:~$ #################################################################################### Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 8:31 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Sorry, can you do the command again with the -i
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 9:10 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Output of command :
####################################################
compute-0:~$ dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 000000007b69d000 000E0 (v01 INTEL S2600WT 00000001 INTL 20091013)
[ 0.000000] DMAR: IOMMU enabled
[ 0.298584] DMAR: Host address width 46
[ 0.302872] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[ 0.308808] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[ 0.317647] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1
[ 0.323581] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df
[ 0.332427] DMAR: RMRR base: 0x0000007a3dd000 end: 0x0000007a3dffff
[ 0.339426] DMAR: ATSR flags: 0x0
[ 11.329277] DMAR: dmar0: Using Queued invalidation
[ 11.334645] DMAR: dmar1: Using Queued invalidation
[ 11.342303] DMAR: Hardware identity mapping for device 0000:ff:08.0
[ 11.349304] DMAR: Hardware identity mapping for device 0000:ff:08.2
[ 11.356299] DMAR: Hardware identity mapping for device 0000:ff:08.3
[ 11.363296] DMAR: Hardware identity mapping for device 0000:ff:09.0
[ 11.370291] DMAR: Hardware identity mapping for device 0000:ff:09.2
[ 11.377286] DMAR: Hardware identity mapping for device 0000:ff:09.3
[ 11.384281] DMAR: Hardware identity mapping for device 0000:ff:0b.0
[ 11.391275] DMAR: Hardware identity mapping for device 0000:ff:0b.1
[ 11.398270] DMAR: Hardware identity mapping for device 0000:ff:0b.2
[ 11.405265] DMAR: Hardware identity mapping for device 0000:ff:0c.0
[ 11.412260] DMAR: Hardware identity mapping for device 0000:ff:0c.1
[ 11.419255] DMAR: Hardware identity mapping for device 0000:ff:0c.2
[ 11.426250] DMAR: Hardware identity mapping for device 0000:ff:0c.3
[ 11.433244] DMAR: Hardware identity mapping for device 0000:ff:0c.4
[ 11.440239] DMAR: Hardware identity mapping for device 0000:ff:0c.5
[ 11.447234] DMAR: Hardware identity mapping for device 0000:ff:0c.6
[ 11.454229] DMAR: Hardware identity mapping for device 0000:ff:0c.7
[ 11.461223] DMAR: Hardware identity mapping for device 0000:ff:0d.0
[ 11.468218] DMAR: Hardware identity mapping for device 0000:ff:0d.1
[ 11.475212] DMAR: Hardware identity mapping for device 0000:ff:0d.2
[ 11.482208] DMAR: Hardware identity mapping for device 0000:ff:0d.3
[ 11.489202] DMAR: Hardware identity mapping for device 0000:ff:0d.4
[ 11.496197] DMAR: Hardware identity mapping for device 0000:ff:0d.5
[ 11.503192] DMAR: Hardware identity mapping for device 0000:ff:0d.6
[ 11.510186] DMAR: Hardware identity mapping for device 0000:ff:0d.7
[ 11.517181] DMAR: Hardware identity mapping for device 0000:ff:0e.0
[ 11.524177] DMAR: Hardware identity mapping for device 0000:ff:0e.1
[ 11.531172] DMAR: Hardware identity mapping for device 0000:ff:0f.0
[ 11.538166] DMAR: Hardware identity mapping for device 0000:ff:0f.1
[ 11.545160] DMAR: Hardware identity mapping for device 0000:ff:0f.2
[ 11.552155] DMAR: Hardware identity mapping for device 0000:ff:0f.3
[ 11.559149] DMAR: Hardware identity mapping for device 0000:ff:0f.4
[ 11.566144] DMAR: Hardware identity mapping for device 0000:ff:0f.5
[ 11.573138] DMAR: Hardware identity mapping for device 0000:ff:0f.6
[ 11.580133] DMAR: Hardware identity mapping for device 0000:ff:10.0
[ 11.587128] DMAR: Hardware identity mapping for device 0000:ff:10.1
[ 11.594123] DMAR: Hardware identity mapping for device 0000:ff:10.5
[ 11.601117] DMAR: Hardware identity mapping for device 0000:ff:10.6
[ 11.608111] DMAR: Hardware identity mapping for device 0000:ff:10.7
[ 11.615106] DMAR: Hardware identity mapping for device 0000:ff:12.0
[ 11.622101] DMAR: Hardware identity mapping for device 0000:ff:12.1
[ 11.629095] DMAR: Hardware identity mapping for device 0000:ff:12.4
[ 11.636090] DMAR: Hardware identity mapping for device 0000:ff:12.5
[ 11.643085] DMAR: Hardware identity mapping for device 0000:ff:13.0
[ 11.650079] DMAR: Hardware identity mapping for device 0000:ff:13.1
[ 11.657074] DMAR: Hardware identity mapping for device 0000:ff:13.2
[ 11.664069] DMAR: Hardware identity mapping for device 0000:ff:13.3
[ 11.671063] DMAR: Hardware identity mapping for device 0000:ff:13.6
[ 11.678058] DMAR: Hardware identity mapping for device 0000:ff:13.7
[ 11.685052] DMAR: Hardware identity mapping for device 0000:ff:14.0
[ 11.692047] DMAR: Hardware identity mapping for device 0000:ff:14.1
[ 11.699044] DMAR: Hardware identity mapping for device 0000:ff:14.2
[ 11.706039] DMAR: Hardware identity mapping for device 0000:ff:14.3
[ 11.713034] DMAR: Hardware identity mapping for device 0000:ff:14.4
[ 11.720028] DMAR: Hardware identity mapping for device 0000:ff:14.5
[ 11.727023] DMAR: Hardware identity mapping for device 0000:ff:14.6
[ 11.734018] DMAR: Hardware identity mapping for device 0000:ff:14.7
[ 11.741012] DMAR: Hardware identity mapping for device 0000:ff:16.0
[ 11.748008] DMAR: Hardware identity mapping for device 0000:ff:16.1
[ 11.755002] DMAR: Hardware identity mapping for device 0000:ff:16.2
[ 11.761997] DMAR: Hardware identity mapping for device 0000:ff:16.3
[ 11.768992] DMAR: Hardware identity mapping for device 0000:ff:16.6
[ 11.775987] DMAR: Hardware identity mapping for device 0000:ff:16.7
[ 11.782981] DMAR: Hardware identity mapping for device 0000:ff:17.0
[ 11.789976] DMAR: Hardware identity mapping for device 0000:ff:17.1
[ 11.796971] DMAR: Hardware identity mapping for device 0000:ff:17.2
[ 11.803966] DMAR: Hardware identity mapping for device 0000:ff:17.3
[ 11.810960] DMAR: Hardware identity mapping for device 0000:ff:17.4
[ 11.817955] DMAR: Hardware identity mapping for device 0000:ff:17.5
[ 11.824949] DMAR: Hardware identity mapping for device 0000:ff:17.6
[ 11.831944] DMAR: Hardware identity mapping for device 0000:ff:17.7
[ 11.838939] DMAR: Hardware identity mapping for device 0000:ff:1e.0
[ 11.845933] DMAR: Hardware identity mapping for device 0000:ff:1e.1
[ 11.852928] DMAR: Hardware identity mapping for device 0000:ff:1e.2
[ 11.859923] DMAR: Hardware identity mapping for device 0000:ff:1e.3
[ 11.866918] DMAR: Hardware identity mapping for device 0000:ff:1e.4
[ 11.873913] DMAR: Hardware identity mapping for device 0000:ff:1f.0
[ 11.880908] DMAR: Hardware identity mapping for device 0000:ff:1f.2
[ 11.887905] DMAR: Hardware identity mapping for device 0000:7f:08.0
[ 11.894899] DMAR: Hardware identity mapping for device 0000:7f:08.2
[ 11.901895] DMAR: Hardware identity mapping for device 0000:7f:08.3
[ 11.908890] DMAR: Hardware identity mapping for device 0000:7f:09.0
[ 11.915885] DMAR: Hardware identity mapping for device 0000:7f:09.2
[ 11.922879] DMAR: Hardware identity mapping for device 0000:7f:09.3
[ 11.929875] DMAR: Hardware identity mapping for device 0000:7f:0b.0
[ 11.936870] DMAR: Hardware identity mapping for device 0000:7f:0b.1
[ 11.943865] DMAR: Hardware identity mapping for device 0000:7f:0b.2
[ 11.950860] DMAR: Hardware identity mapping for device 0000:7f:0c.0
[ 11.957855] DMAR: Hardware identity mapping for device 0000:7f:0c.1
[ 11.964849] DMAR: Hardware identity mapping for device 0000:7f:0c.2
[ 11.971844] DMAR: Hardware identity mapping for device 0000:7f:0c.3
[ 11.978839] DMAR: Hardware identity mapping for device 0000:7f:0c.4
[ 11.985833] DMAR: Hardware identity mapping for device 0000:7f:0c.5
[ 11.992828] DMAR: Hardware identity mapping for device 0000:7f:0c.6
[ 11.999823] DMAR: Hardware identity mapping for device 0000:7f:0c.7
[ 12.006818] DMAR: Hardware identity mapping for device 0000:7f:0d.0
[ 12.013814] DMAR: Hardware identity mapping for device 0000:7f:0d.1
[ 12.020801] DMAR: Hardware identity mapping for device 0000:7f:0d.2
[ 12.027796] DMAR: Hardware identity mapping for device 0000:7f:0d.3
[ 12.034790] DMAR: Hardware identity mapping for device 0000:7f:0d.4
[ 12.041785] DMAR: Hardware identity mapping for device 0000:7f:0d.5
[ 12.048781] DMAR: Hardware identity mapping for device 0000:7f:0d.6
[ 12.055776] DMAR: Hardware identity mapping for device 0000:7f:0d.7
[ 12.062771] DMAR: Hardware identity mapping for device 0000:7f:0e.0
[ 12.069766] DMAR: Hardware identity mapping for device 0000:7f:0e.1
[ 12.076761] DMAR: Hardware identity mapping for device 0000:7f:0f.0
[ 12.083756] DMAR: Hardware identity mapping for device 0000:7f:0f.1
[ 12.090752] DMAR: Hardware identity mapping for device 0000:7f:0f.2
[ 12.097747] DMAR: Hardware identity mapping for device 0000:7f:0f.3
[ 12.104741] DMAR: Hardware identity mapping for device 0000:7f:0f.4
[ 12.111736] DMAR: Hardware identity mapping for device 0000:7f:0f.5
[ 12.118731] DMAR: Hardware identity mapping for device 0000:7f:0f.6
[ 12.125726] DMAR: Hardware identity mapping for device 0000:7f:10.0
[ 12.132721] DMAR: Hardware identity mapping for device 0000:7f:10.1
[ 12.139716] DMAR: Hardware identity mapping for device 0000:7f:10.5
[ 12.146713] DMAR: Hardware identity mapping for device 0000:7f:10.6
[ 12.153708] DMAR: Hardware identity mapping for device 0000:7f:10.7
[ 12.160703] DMAR: Hardware identity mapping for device 0000:7f:12.0
[ 12.167698] DMAR: Hardware identity mapping for device 0000:7f:12.1
[ 12.174693] DMAR: Hardware identity mapping for device 0000:7f:12.4
[ 12.181688] DMAR: Hardware identity mapping for device 0000:7f:12.5
[ 12.188683] DMAR: Hardware identity mapping for device 0000:7f:13.0
[ 12.195678] DMAR: Hardware identity mapping for device 0000:7f:13.1
[ 12.202673] DMAR: Hardware identity mapping for device 0000:7f:13.2
[ 12.209668] DMAR: Hardware identity mapping for device 0000:7f:13.3
[ 12.216663] DMAR: Hardware identity mapping for device 0000:7f:13.6
[ 12.223658] DMAR: Hardware identity mapping for device 0000:7f:13.7
[ 12.230653] DMAR: Hardware identity mapping for device 0000:7f:14.0
[ 12.237648] DMAR: Hardware identity mapping for device 0000:7f:14.1
[ 12.244643] DMAR: Hardware identity mapping for device 0000:7f:14.2
[ 12.251638] DMAR: Hardware identity mapping for device 0000:7f:14.3
[ 12.258634] DMAR: Hardware identity mapping for device 0000:7f:14.4
[ 12.265629] DMAR: Hardware identity mapping for device 0000:7f:14.5
[ 12.272624] DMAR: Hardware identity mapping for device 0000:7f:14.6
[ 12.279620] DMAR: Hardware identity mapping for device 0000:7f:14.7
[ 12.286615] DMAR: Hardware identity mapping for device 0000:7f:16.0
[ 12.293610] DMAR: Hardware identity mapping for device 0000:7f:16.1
[ 12.300605] DMAR: Hardware identity mapping for device 0000:7f:16.2
[ 12.307600] DMAR: Hardware identity mapping for device 0000:7f:16.3
[ 12.314595] DMAR: Hardware identity mapping for device 0000:7f:16.6
[ 12.321590] DMAR: Hardware identity mapping for device 0000:7f:16.7
[ 12.328585] DMAR: Hardware identity mapping for device 0000:7f:17.0
[ 12.335580] DMAR: Hardware identity mapping for device 0000:7f:17.1
[ 12.342575] DMAR: Hardware identity mapping for device 0000:7f:17.2
[ 12.349570] DMAR: Hardware identity mapping for device 0000:7f:17.3
[ 12.356565] DMAR: Hardware identity mapping for device 0000:7f:17.4
[ 12.363564] DMAR: Hardware identity mapping for device 0000:7f:17.5
[ 12.370563] DMAR: Hardware identity mapping for device 0000:7f:17.6
[ 12.377562] DMAR: Hardware identity mapping for device 0000:7f:17.7
[ 12.384561] DMAR: Hardware identity mapping for device 0000:7f:1e.0
[ 12.391560] DMAR: Hardware identity mapping for device 0000:7f:1e.1
[ 12.398555] DMAR: Hardware identity mapping for device 0000:7f:1e.2
[ 12.405550] DMAR: Hardware identity mapping for device 0000:7f:1e.3
[ 12.412545] DMAR: Hardware identity mapping for device 0000:7f:1e.4
[ 12.419541] DMAR: Hardware identity mapping for device 0000:7f:1f.0
[ 12.426536] DMAR: Hardware identity mapping for device 0000:7f:1f.2
[ 12.433533] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 12.440529] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 12.447523] DMAR: Hardware identity mapping for device 0000:00:02.0
[ 12.454519] DMAR: Hardware identity mapping for device 0000:00:02.2
[ 12.461514] DMAR: Hardware identity mapping for device 0000:00:03.0
[ 12.468509] DMAR: Hardware identity mapping for device 0000:00:04.0
[ 12.475504] DMAR: Hardware identity mapping for device 0000:00:04.1
[ 12.482499] DMAR: Hardware identity mapping for device 0000:00:04.2
[ 12.489495] DMAR: Hardware identity mapping for device 0000:00:04.3
[ 12.496490] DMAR: Hardware identity mapping for device 0000:00:04.4
[ 12.503486] DMAR: Hardware identity mapping for device 0000:00:04.5
[ 12.510481] DMAR: Hardware identity mapping for device 0000:00:04.6
[ 12.517479] DMAR: Hardware identity mapping for device 0000:00:04.7
[ 12.524474] DMAR: Hardware identity mapping for device 0000:00:05.0
[ 12.531469] DMAR: Hardware identity mapping for device 0000:00:05.1
[ 12.538465] DMAR: Hardware identity mapping for device 0000:00:05.2
[ 12.545461] DMAR: Hardware identity mapping for device 0000:00:05.4
[ 12.552456] DMAR: Hardware identity mapping for device 0000:00:11.0
[ 12.559451] DMAR: Hardware identity mapping for device 0000:00:11.1
[ 12.566449] DMAR: Hardware identity mapping for device 0000:00:11.4
[ 12.573444] DMAR: Hardware identity mapping for device 0000:00:14.0
[ 12.580440] DMAR: Hardware identity mapping for device 0000:00:16.0
[ 12.587436] DMAR: Hardware identity mapping for device 0000:00:16.1
[ 12.594433] DMAR: Hardware identity mapping for device 0000:00:1a.0
[ 12.601429] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 12.608424] DMAR: Hardware identity mapping for device 0000:00:1c.3
[ 12.615419] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 12.622415] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 12.629410] DMAR: Hardware identity mapping for device 0000:00:1f.2
[ 12.636405] DMAR: Hardware identity mapping for device 0000:00:1f.3
[ 12.643402] DMAR: Hardware identity mapping for device 0000:02:00.0
[ 12.650397] DMAR: Hardware identity mapping for device 0000:02:00.1
[ 12.657394] DMAR: Hardware identity mapping for device 0000:04:00.0
[ 12.664390] DMAR: Hardware identity mapping for device 0000:04:00.3
[ 12.671390] DMAR: Hardware identity mapping for device 0000:08:00.0
[ 12.678388] DMAR: Hardware identity mapping for device 0000:80:04.0
[ 12.685384] DMAR: Hardware identity mapping for device 0000:80:04.1
[ 12.692371] DMAR: Hardware identity mapping for device 0000:80:04.2
[ 12.699358] DMAR: Hardware identity mapping for device 0000:80:04.3
[ 12.706346] DMAR: Hardware identity mapping for device 0000:80:04.4
[ 12.713341] DMAR: Hardware identity mapping for device 0000:80:04.5
[ 12.720336] DMAR: Hardware identity mapping for device 0000:80:04.6
[ 12.727331] DMAR: Hardware identity mapping for device 0000:80:04.7
[ 12.734331] DMAR: Hardware identity mapping for device 0000:80:05.0
[ 12.741327] DMAR: Hardware identity mapping for device 0000:80:05.1
[ 12.748314] DMAR: Hardware identity mapping for device 0000:80:05.2
[ 12.755301] DMAR: Hardware identity mapping for device 0000:80:05.4
[ 12.762287] DMAR: Setting RMRR:
[ 12.765793] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0x7a3dd000 - 0x7a3dffff]
[ 12.776376] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0x7a3dd000 - 0x7a3dffff]
[ 12.786965] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 12.792797] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 12.802607] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 14.693008] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
[ 14.702309] DMAR: Setting identity map for device 0000:00:1a.0 [0x7a3dd000 - 0x7a3dffff]
[ 14.762084] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[ 14.770843] DMAR: Setting identity map for device 0000:00:1d.0 [0x7a3dd000 - 0x7a3dffff]
compute-0:~$
compute-0:~$
####################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping). Maybe there are others on the distribution list that could offer additional suggestions. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Thursday, January 24, 2019 at 11:32 AM To: "Peters, Matt" <Matt.Peters@windriver.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Thanks Matt, Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version. ################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family ############################################################################# ################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0 Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS ######################################################################################################## Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Matt, Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*. Please find attached dmesg log output for reference. ####################################################################### compute-0:/home/wrsroot#* sudo ovs-vsctl show* 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot# ############################################################################# Please suggest me any other logs that i can check. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family
#############################################################################
################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS
########################################################################################################
Regards, Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Matt, Please find below output of dpdk-devbind.py --status also: compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status ########################################################################################################## python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active* Other Network devices ===================== 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/var/log/puppet/latest# ################################################################################ Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*. Please find attached dmesg log output for reference.
####################################################################### compute-0:/home/wrsroot#* sudo ovs-vsctl show* 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards, Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family
#############################################################################
################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS
########################################################################################################
Regards, Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
The DPDK and OVS status look fine now. I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Friday, January 25, 2019 at 3:56 AM To: "Peters, Matt" <Matt.Peters@windriver.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Matt, Please find below output of dpdk-devbind.py --status also: compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status ########################################################################################################## python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active* Other Network devices ===================== 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/var/log/puppet/latest# ################################################################################ Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi Matt, Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors. But compute still rebooting in loop. Please find attached dmesg log output for reference. ####################################################################### compute-0:/home/wrsroot# sudo ovs-vsctl show 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot# ############################################################################# Please suggest me any other logs that i can check. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Thanks Matt, Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version. ################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family ############################################################################# ################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0 Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS ######################################################################################################## Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping). Maybe there are others on the distribution list that could offer additional suggestions. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 11:32 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Thanks Matt, Please find attached mtcAgent.log Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 6:36 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The DPDK and OVS status look fine now.
I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Friday, January 25, 2019 at 3:56 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Matt,
Please find below output of dpdk-devbind.py --status also:
compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
##########################################################################################################
python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active*
Other Network devices
=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/var/log/puppet/latest#
################################################################################
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*.
Please find attached dmesg log output for reference.
#######################################################################
compute-0:/home/wrsroot#* sudo ovs-vsctl show*
8a1f7179-2014-4b4e-ad23-2065b8024964
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "lldp64416a6d-7d"
Interface "lldp64416a6d-7d"
type: internal
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "eth0"
Interface "eth0"
type: dpdk
options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
ovs_version: "2.9.0"
compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719,
Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
#################################################################
compute-0:/home/wrsroot# dmidecode -t1
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Intel Corporation
Product Name: S2600WT2
Version: ....................
Serial Number: ............
UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
Wake-up Type: Power Switch
SKU Number: SKU Number
Family: Family
#############################################################################
################################################################################
compute-0:/home/wrsroot# sudo dmidecode --type bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Intel Corporation
Version: SE5C610.86B.01.01.0014.121820151719
Release Date: 12/18/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16384 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 0.0
Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Abbreviated
Installable Languages: 1
enUS
Currently Installed Language: enUS
########################################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
fgrep Error /var/log/mtcAgent.log fm alarm-list From: Peters, Matt [mailto:Matt.Peters@windriver.com] Sent: Friday, January 25, 2019 8:05 AM To: Himanshu Goyal Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) The DPDK and OVS status look fine now. I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot. From: Himanshu Goyal <himanshugoyal500@gmail.com> Date: Friday, January 25, 2019 at 3:56 AM To: "Peters, Matt" <Matt.Peters@windriver.com> Cc: "Hu, Yong" <yong.hu@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Matt, Please find below output of dpdk-devbind.py --status also: compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status ########################################################################################################## python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active* Other Network devices ===================== 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/var/log/puppet/latest# ################################################################################ Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi Matt, Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors. But compute still rebooting in loop. Please find attached dmesg log output for reference. ####################################################################### compute-0:/home/wrsroot# sudo ovs-vsctl show 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot# ############################################################################# Please suggest me any other logs that i can check. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Thanks Matt, Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version. ################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family ############################################################################# ################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0 Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS ######################################################################################################## Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping). Maybe there are others on the distribution list that could offer additional suggestions. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 11:32 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Eric, Below are required outputs of commands: *fm alarm-list * ###################################################### [root@controller-0 wrsroot(keystone_admin)]# fm alarm-list +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ | Alarm | Reason Text | Entity ID | Severity | Time Stamp | | ID | | | | | +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ | 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: | | 001 | | | | 00:12.159130 | | | | | | | | 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: | | 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 | | | | | | | | 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: | | 002 | no standby members available | service_group=cloud-services | | 12:07.245142 | | | | | | | | 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: | | 002 | but no standby members available | service_group=controller-services | | 11:40.174144 | | | | | | | | 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: | | 002 | standby members available | service_group=vim-services | | 11:39.052152 | | | | | | | | 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: | | 002 | standby members available | service_group=oam-services | | 11:38.728175 | | | | | | | | 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: | | 002 | but no standby members available | service_group=patching-services | | 11:37.756169 | | | | | | | | 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: | | 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 | | | | | | | | 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: | | 002 | only 1 active member available | service_group=web-services | | 11:37.189196 | | | | | | | | 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: | | 005 | | | | 11:36.690138 | | | | | | | | 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: | | 005 | | | | 11:36.427073 | | | | | | | +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ [root@controller-0 wrsroot(keystone_admin)]# ######################################################################################################### Please find attached file for fgrep Error /var/log/mtcAgent.log output Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
fgrep Error /var/log/mtcAgent.log
fm alarm-list
*From:* Peters, Matt [mailto:Matt.Peters@windriver.com] *Sent:* Friday, January 25, 2019 8:05 AM *To:* Himanshu Goyal *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
The DPDK and OVS status look fine now.
I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Friday, January 25, 2019 at 3:56 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Matt,
Please find below output of dpdk-devbind.py --status also:
compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
##########################################################################################################
python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active*
Other Network devices
=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/var/log/puppet/latest#
################################################################################
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*.
Please find attached dmesg log output for reference.
#######################################################################
compute-0:/home/wrsroot#* sudo ovs-vsctl show*
8a1f7179-2014-4b4e-ad23-2065b8024964
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "lldp64416a6d-7d"
Interface "lldp64416a6d-7d"
type: internal
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "eth0"
Interface "eth0"
type: dpdk
options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
ovs_version: "2.9.0"
compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719,
Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
#################################################################
compute-0:/home/wrsroot# dmidecode -t1
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Intel Corporation
Product Name: S2600WT2
Version: ....................
Serial Number: ............
UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
Wake-up Type: Power Switch
SKU Number: SKU Number
Family: Family
#############################################################################
################################################################################
compute-0:/home/wrsroot# sudo dmidecode --type bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Intel Corporation
Version: SE5C610.86B.01.01.0014.121820151719
Release Date: 12/18/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16384 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 0.0
Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Abbreviated
Installable Languages: 1
enUS
Currently Installed Language: enUS
########################################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node
2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Himanshu, The mtcAgent.log shows that compute-0 has experienced the following errors several times over the span of almost 3 days. Configuration failures 2019-01-24T12:30:34.764 [44890.04799] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1046) enable_handler :Error : compute-0 configuration incomplete or failed (oob:2:1) 2019-01-24T12:30:34.764 [44890.04800] controller-0 mtcAgent --- nodeClass.cpp (1606) alarm_config_failure :Error : compute-0 critical config failure 2019-01-24T12:30:34.764 [44890.04801] controller-0 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : compute-0 setting critical 'Configuration' failure alarm (200.011 Out-Of-Service GoEnable Test Failures 2019-01-24T12:00:56.186 [44890.04700] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1163) enable_handler :Error : compute-0 got GOENABLED Experiencing Management network heartbeat failures. 2019-01-23T15:03:44.535 [44890.01949] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure 2019-01-23T15:03:45.688 [44890.01967] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss ** Since compute-0 is locked the in-service alarms against it will be suppressed. Since there are many errors over time I would recommend tailing the mtcAgent.log file to a new small file you can send to me. This file would contain all the logs that are produced for one cycle of an unlock of compute-0 This way we can focus on one case at a time. Eric. From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com] Sent: Friday, January 25, 2019 9:31 AM To: MacDonald, Eric Cc: Peters, Matt; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Importance: High Hi Eric, Below are required outputs of commands: fm alarm-list ###################################################### [root@controller-0 wrsroot(keystone_admin)]# fm alarm-list +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ | Alarm | Reason Text | Entity ID | Severity | Time Stamp | | ID | | | | | +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ | 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: | | 001 | | | | 00:12.159130 | | | | | | | | 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: | | 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 | | | | | | | | 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: | | 002 | no standby members available | service_group=cloud-services | | 12:07.245142 | | | | | | | | 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: | | 002 | but no standby members available | service_group=controller-services | | 11:40.174144 | | | | | | | | 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: | | 002 | standby members available | service_group=vim-services | | 11:39.052152 | | | | | | | | 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: | | 002 | standby members available | service_group=oam-services | | 11:38.728175 | | | | | | | | 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: | | 002 | but no standby members available | service_group=patching-services | | 11:37.756169 | | | | | | | | 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: | | 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 | | | | | | | | 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: | | 002 | only 1 active member available | service_group=web-services | | 11:37.189196 | | | | | | | | 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: | | 005 | | | | 11:36.690138 | | | | | | | | 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: | | 005 | | | | 11:36.427073 | | | | | | | +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ [root@controller-0 wrsroot(keystone_admin)]# ######################################################################################################### Please find attached file for fgrep Error /var/log/mtcAgent.log output Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>> wrote: fgrep Error /var/log/mtcAgent.log fm alarm-list From: Peters, Matt [mailto:Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>] Sent: Friday, January 25, 2019 8:05 AM To: Himanshu Goyal Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) The DPDK and OVS status look fine now. I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Friday, January 25, 2019 at 3:56 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Matt, Please find below output of dpdk-devbind.py --status also: compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status ########################################################################################################## python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active* Other Network devices ===================== 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/var/log/puppet/latest# ################################################################################ Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi Matt, Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors. But compute still rebooting in loop. Please find attached dmesg log output for reference. ####################################################################### compute-0:/home/wrsroot# sudo ovs-vsctl show 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot# ############################################################################# Please suggest me any other logs that i can check. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Thanks Matt, Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version. ################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family ############################################################################# ################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0 Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS ######################################################################################################## Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping). Maybe there are others on the distribution list that could offer additional suggestions. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 11:32 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Sure Eric, currently i have left for the day, i will get back to you first thing on Monday morning. Many Thanks, Himanshu Goyal On Fri, Jan 25, 2019 at 8:17 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
Hi Himanshu,
The mtcAgent.log shows that compute-0 has experienced the following errors several times over the span of almost 3 days.
*Configuration failures*
2019-01-24T12:30:34.764 [44890.04799] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1046) enable_handler :Error : compute-0 configuration incomplete or failed (oob:2:1)
2019-01-24T12:30:34.764 [44890.04800] controller-0 mtcAgent --- nodeClass.cpp (1606) alarm_config_failure :Error : compute-0 critical config failure
2019-01-24T12:30:34.764 [44890.04801] controller-0 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : compute-0 setting critical 'Configuration' failure alarm (200.011
*Out-Of-Service GoEnable Test Failures*
2019-01-24T12:00:56.186 [44890.04700] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1163) enable_handler :Error : compute-0 got GOENABLED
*Experiencing Management network heartbeat failures.*
2019-01-23T15:03:44.535 [44890.01949] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure
2019-01-23T15:03:45.688 [44890.01967] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss **
Since compute-0 is locked the in-service alarms against it will be suppressed.
Since there are many errors over time I would recommend tailing the mtcAgent.log file to a new small file you can send to me. This file would contain all the logs that are produced for one cycle of an unlock of compute-0
This way we can focus on one case at a time.
Eric.
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Friday, January 25, 2019 9:31 AM *To:* MacDonald, Eric *Cc:* Peters, Matt; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) *Importance:* High
Hi Eric,
Below are required outputs of commands:
*fm alarm-list *
######################################################
[root@controller-0 wrsroot(keystone_admin)]# fm alarm-list
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: |
| 001 | | | | 00:12.159130 |
| | | | | |
| 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: |
| 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 |
| | | | | |
| 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | no standby members available | service_group=cloud-services | | 12:07.245142 |
| | | | | |
| 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=controller-services | | 11:40.174144 |
| | | | | |
| 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=vim-services | | 11:39.052152 |
| | | | | |
| 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=oam-services | | 11:38.728175 |
| | | | | |
| 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=patching-services | | 11:37.756169 |
| | | | | |
| 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 |
| | | | | |
| 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | only 1 active member available | service_group=web-services | | 11:37.189196 |
| | | | | |
| 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: |
| 005 | | | | 11:36.690138 |
| | | | | |
| 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: |
| 005 | | | | 11:36.427073 |
| | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
[root@controller-0 wrsroot(keystone_admin)]#
#########################################################################################################
Please find attached file for fgrep Error /var/log/mtcAgent.log output
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
fgrep Error /var/log/mtcAgent.log
fm alarm-list
*From:* Peters, Matt [mailto:Matt.Peters@windriver.com] *Sent:* Friday, January 25, 2019 8:05 AM *To:* Himanshu Goyal *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
The DPDK and OVS status look fine now.
I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Friday, January 25, 2019 at 3:56 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Matt,
Please find below output of dpdk-devbind.py --status also:
compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
##########################################################################################################
python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active*
Other Network devices
=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/var/log/puppet/latest#
################################################################################
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*.
Please find attached dmesg log output for reference.
#######################################################################
compute-0:/home/wrsroot#* sudo ovs-vsctl show*
8a1f7179-2014-4b4e-ad23-2065b8024964
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "lldp64416a6d-7d"
Interface "lldp64416a6d-7d"
type: internal
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "eth0"
Interface "eth0"
type: dpdk
options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
ovs_version: "2.9.0"
compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719,
Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
#################################################################
compute-0:/home/wrsroot# dmidecode -t1
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Intel Corporation
Product Name: S2600WT2
Version: ....................
Serial Number: ............
UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
Wake-up Type: Power Switch
SKU Number: SKU Number
Family: Family
#############################################################################
################################################################################
compute-0:/home/wrsroot# sudo dmidecode --type bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Intel Corporation
Version: SE5C610.86B.01.01.0014.121820151719
Release Date: 12/18/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16384 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 0.0
Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Abbreviated
Installable Languages: 1
enUS
Currently Installed Language: enUS
########################################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node
2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Eric, Please find attached required mtcAgent.log containing one cycle of compute from PXE boot to unlock. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 10:12 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Sure Eric, currently i have left for the day, i will get back to you first thing on Monday morning.
Many Thanks, Himanshu Goyal
On Fri, Jan 25, 2019 at 8:17 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
Hi Himanshu,
The mtcAgent.log shows that compute-0 has experienced the following errors several times over the span of almost 3 days.
*Configuration failures*
2019-01-24T12:30:34.764 [44890.04799] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1046) enable_handler :Error : compute-0 configuration incomplete or failed (oob:2:1)
2019-01-24T12:30:34.764 [44890.04800] controller-0 mtcAgent --- nodeClass.cpp (1606) alarm_config_failure :Error : compute-0 critical config failure
2019-01-24T12:30:34.764 [44890.04801] controller-0 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : compute-0 setting critical 'Configuration' failure alarm (200.011
*Out-Of-Service GoEnable Test Failures*
2019-01-24T12:00:56.186 [44890.04700] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1163) enable_handler :Error : compute-0 got GOENABLED
*Experiencing Management network heartbeat failures.*
2019-01-23T15:03:44.535 [44890.01949] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure
2019-01-23T15:03:45.688 [44890.01967] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss **
Since compute-0 is locked the in-service alarms against it will be suppressed.
Since there are many errors over time I would recommend tailing the mtcAgent.log file to a new small file you can send to me. This file would contain all the logs that are produced for one cycle of an unlock of compute-0
This way we can focus on one case at a time.
Eric.
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Friday, January 25, 2019 9:31 AM *To:* MacDonald, Eric *Cc:* Peters, Matt; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) *Importance:* High
Hi Eric,
Below are required outputs of commands:
*fm alarm-list *
######################################################
[root@controller-0 wrsroot(keystone_admin)]# fm alarm-list
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: |
| 001 | | | | 00:12.159130 |
| | | | | |
| 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: |
| 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 |
| | | | | |
| 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | no standby members available | service_group=cloud-services | | 12:07.245142 |
| | | | | |
| 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=controller-services | | 11:40.174144 |
| | | | | |
| 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=vim-services | | 11:39.052152 |
| | | | | |
| 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=oam-services | | 11:38.728175 |
| | | | | |
| 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=patching-services | | 11:37.756169 |
| | | | | |
| 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 |
| | | | | |
| 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | only 1 active member available | service_group=web-services | | 11:37.189196 |
| | | | | |
| 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: |
| 005 | | | | 11:36.690138 |
| | | | | |
| 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: |
| 005 | | | | 11:36.427073 |
| | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
[root@controller-0 wrsroot(keystone_admin)]#
#########################################################################################################
Please find attached file for fgrep Error /var/log/mtcAgent.log output
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
fgrep Error /var/log/mtcAgent.log
fm alarm-list
*From:* Peters, Matt [mailto:Matt.Peters@windriver.com] *Sent:* Friday, January 25, 2019 8:05 AM *To:* Himanshu Goyal *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
The DPDK and OVS status look fine now.
I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Friday, January 25, 2019 at 3:56 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Matt,
Please find below output of dpdk-devbind.py --status also:
compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
##########################################################################################################
python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active*
Other Network devices
=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/var/log/puppet/latest#
################################################################################
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*.
Please find attached dmesg log output for reference.
#######################################################################
compute-0:/home/wrsroot#* sudo ovs-vsctl show*
8a1f7179-2014-4b4e-ad23-2065b8024964
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "lldp64416a6d-7d"
Interface "lldp64416a6d-7d"
type: internal
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "eth0"
Interface "eth0"
type: dpdk
options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
ovs_version: "2.9.0"
compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719,
Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
#################################################################
compute-0:/home/wrsroot# dmidecode -t1
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Intel Corporation
Product Name: S2600WT2
Version: ....................
Serial Number: ............
UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
Wake-up Type: Power Switch
SKU Number: SKU Number
Family: Family
#############################################################################
################################################################################
compute-0:/home/wrsroot# sudo dmidecode --type bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Intel Corporation
Version: SE5C610.86B.01.01.0014.121820151719
Release Date: 12/18/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16384 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 0.0
Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Abbreviated
Installable Languages: 1
enUS
Currently Installed Language: enUS
########################################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node
2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Hi Eric, That compute reboot issue is resolved, I connect controller & compute connection directly without switch in between after unlock. I don't know why it is not working with switch after unlock, Please check there is only hear-beat issue on mgmt network or something else, currently i'm using only 1 compute so its fine. Regards, Himanshu Goyal On Mon, Jan 28, 2019 at 12:33 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi Eric,
Please find attached required mtcAgent.log containing one cycle of compute from PXE boot to unlock.
Regards, Himanshu Goyal
On Fri, Jan 25, 2019 at 10:12 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Sure Eric, currently i have left for the day, i will get back to you first thing on Monday morning.
Many Thanks, Himanshu Goyal
On Fri, Jan 25, 2019 at 8:17 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
Hi Himanshu,
The mtcAgent.log shows that compute-0 has experienced the following errors several times over the span of almost 3 days.
*Configuration failures*
2019-01-24T12:30:34.764 [44890.04799] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1046) enable_handler :Error : compute-0 configuration incomplete or failed (oob:2:1)
2019-01-24T12:30:34.764 [44890.04800] controller-0 mtcAgent --- nodeClass.cpp (1606) alarm_config_failure :Error : compute-0 critical config failure
2019-01-24T12:30:34.764 [44890.04801] controller-0 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : compute-0 setting critical 'Configuration' failure alarm (200.011
*Out-Of-Service GoEnable Test Failures*
2019-01-24T12:00:56.186 [44890.04700] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1163) enable_handler :Error : compute-0 got GOENABLED
*Experiencing Management network heartbeat failures.*
2019-01-23T15:03:44.535 [44890.01949] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure
2019-01-23T15:03:45.688 [44890.01967] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss **
Since compute-0 is locked the in-service alarms against it will be suppressed.
Since there are many errors over time I would recommend tailing the mtcAgent.log file to a new small file you can send to me. This file would contain all the logs that are produced for one cycle of an unlock of compute-0
This way we can focus on one case at a time.
Eric.
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Friday, January 25, 2019 9:31 AM *To:* MacDonald, Eric *Cc:* Peters, Matt; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) *Importance:* High
Hi Eric,
Below are required outputs of commands:
*fm alarm-list *
######################################################
[root@controller-0 wrsroot(keystone_admin)]# fm alarm-list
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: |
| 001 | | | | 00:12.159130 |
| | | | | |
| 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: |
| 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 |
| | | | | |
| 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | no standby members available | service_group=cloud-services | | 12:07.245142 |
| | | | | |
| 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=controller-services | | 11:40.174144 |
| | | | | |
| 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=vim-services | | 11:39.052152 |
| | | | | |
| 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=oam-services | | 11:38.728175 |
| | | | | |
| 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=patching-services | | 11:37.756169 |
| | | | | |
| 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 |
| | | | | |
| 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | only 1 active member available | service_group=web-services | | 11:37.189196 |
| | | | | |
| 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: |
| 005 | | | | 11:36.690138 |
| | | | | |
| 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: |
| 005 | | | | 11:36.427073 |
| | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
[root@controller-0 wrsroot(keystone_admin)]#
#########################################################################################################
Please find attached file for fgrep Error /var/log/mtcAgent.log output
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
fgrep Error /var/log/mtcAgent.log
fm alarm-list
*From:* Peters, Matt [mailto:Matt.Peters@windriver.com] *Sent:* Friday, January 25, 2019 8:05 AM *To:* Himanshu Goyal *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
The DPDK and OVS status look fine now.
I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Friday, January 25, 2019 at 3:56 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Matt,
Please find below output of dpdk-devbind.py --status also:
compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
##########################################################################################################
python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active*
Other Network devices
=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/var/log/puppet/latest#
################################################################################
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*.
Please find attached dmesg log output for reference.
#######################################################################
compute-0:/home/wrsroot#* sudo ovs-vsctl show*
8a1f7179-2014-4b4e-ad23-2065b8024964
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "lldp64416a6d-7d"
Interface "lldp64416a6d-7d"
type: internal
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "eth0"
Interface "eth0"
type: dpdk
options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
ovs_version: "2.9.0"
compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719,
Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
#################################################################
compute-0:/home/wrsroot# dmidecode -t1
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Intel Corporation
Product Name: S2600WT2
Version: ....................
Serial Number: ............
UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
Wake-up Type: Power Switch
SKU Number: SKU Number
Family: Family
#############################################################################
################################################################################
compute-0:/home/wrsroot# sudo dmidecode --type bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Intel Corporation
Version: SE5C610.86B.01.01.0014.121820151719
Release Date: 12/18/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16384 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 0.0
Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Abbreviated
Installable Languages: 1
enUS
Currently Installed Language: enUS
########################################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node
2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
From the logs it looks like compute enabled fine this time but then failed shortly after due to intermittent heartbeat loss. The stages are … First mtcAlive with no indication of config failure ; a good thing 2019-01-28T12:22:15.430 [44890.06760] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1056) enable_handler : Info : compute-0 is MTCALIVE (uptime:235 secs) 2019-01-28T12:22:15.430 fmAPI.cpp(471): Enqueue raise alarm request: UUID (c698824d-0d3e-42f0-a7a6-f705403ac8fe) alarm id (200.022) instant id (host=compute-0.status=online) Out-Of-Service In-Test passed ; a good thing 2019-01-28T12:22:15.430 [44890.06761] controller-0 mtcAgent inv mtcInvApi.cpp (1085) mtcInvApi_update_state : Info : compute-0 intest (seq:19) 2019-01-28T12:22:27.557 [44890.06764] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1172) enable_handler : Info : compute-0 got GOENABLED Starting of local host services passed ; a good thing 2019-01-28T12:22:27.562 [44890.06766] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1212) enable_handler : Info : compute-0 Starting Host Services 2019-01-28T12:22:27.628 [44890.06769] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (2586) host_services_handler : Info : compute-0 start compute host services completed Heartbeat client was immediately ready as expected ; a good thing 2019-01-28T12:22:28.253 [44890.06770] controller-0 mtcAgent |-| nodeClass.cpp (4717) declare_service_ready : Info : compute-0 got hbsClient ready event Heartbeat start and 11 second trial soak passed ; a good thing 2019-01-28T12:22:28.253 [44890.06771] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 804) send_hbs_command : Info : compute-0 sending 'start' to heartbeat service 2019-01-28T12:22:28.253 [44890.06772] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1327) enable_handler : Info : compute-0 Starting 11 sec Heartbeat Soak (with ready event) 2019-01-28T12:22:39.253 [44890.06773] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1341) enable_handler : Info : compute-0 heartbeating Compute-0 Enabled ok ; very good thing 2019-01-28T12:22:39.502 [44890.06778] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1488) enable_handler : Info : compute-0 is ENABLE Intermittent heartbeat loss starts. The after ~ 4 minutes compute-0 started experiencing Management network heartbeat loss 2019-01-28T12:26:58.819 [44890.06781] controller-0 mtcAgent --- nodeClass.cpp (4613) manage_heartbeat_degrade: Warn : compute-0 Mgmnt *** Heartbeat Miss *** 2019-01-28T12:26:59.257 [44890.06792] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1637) recovery_handler : Info : compute-0 requesting mtcAlive with 5 sec timeout 2019-01-28T12:26:59.257 [44890.06794] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1655) recovery_handler : Info : compute-0 got requested mtcAlive Tries the 11 second heartbeat soak again after getting the requested mtcAlive message 2019-01-28T12:26:59.257 [44890.06795] controller-0 mtcAgent |-| nodeClass.cpp (2480) stop_offline_handler : Info : compute-0 stopping offline handler (unlocked-enabled-failed) (stage:3) 2019-01-28T12:26:59.257 [44890.06796] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1722) recovery_handler : Warn : compute-0 Connectivity Recovered ; host did not reset 2019-01-28T12:26:59.257 [44890.06797] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1724) recovery_handler : Warn : compute-0 ... continuing with graceful recovery 2019-01-28T12:26:59.257 [44890.06798] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1725) recovery_handler : Warn : compute-0 ... with no affect to host services 2019-01-28T12:26:59.257 [44890.06799] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 804) send_hbs_command : Info : compute-0 sending 'start' to heartbeat service 2019-01-28T12:26:59.257 [44890.06800] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (2389) recovery_handler : Info : compute-0 Starting 11 sec Heartbeat Soak (with ready event) The heartbeat loss is intermittent After first loss the recovery tried the 11 second soak again that failed 2019-01-28T12:26:59.982 [44890.06802] controller-0 mtcAgent --- nodeClass.cpp (4613) manage_heartbeat_degrade: Warn : compute-0 Mgmnt *** Heartbeat Miss *** 2019-01-28T12:27:00.403 [44890.06803] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss *** 2019-01-28T12:27:00.403 [44890.06804] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure 2019-01-28T12:27:00.403 [44890.06805] controller-0 mtcAgent --- nodeClass.cpp (4515) manage_heartbeat_failure: Warn : compute-0 restarting graceful recovery 2019-01-28T12:27:00.403 [44890.06806] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1557) recovery_handler : Info : compute-0 Graceful Recovery (uptime was 519) Heartbeat seems intermittent because the recovery algorithm reached its retry threshold of 3 and force failed the node 2019-01-28T12:27:01.556 [44890.06828] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1592) recovery_handler :Error : compute-0 Graceful Recovery Failed (retries=3 2019-01-28T12:27:01.556 [44890.06829] controller-0 mtcAgent |-| nodeClass.cpp (7193) force_full_enable : Info : compute-0 Forcing Full Enable Sequence 2019-01-28T12:27:01.556 [44890.06830] controller-0 mtcAgent --- nodeClass.cpp (1644) alarm_enabled_failure :Error : compute-0 critical enable failure The force failure caused a reboot and recovery retry where the logs ended. Sequence will likely continue until the heartbeat (potentially multicast) issue is fixed. Based on you saying that with the switch removed heartbeat passed I’d look at how the switch or your management network is handling multicast messaging and if the QOS settings on the network might be dropping packets. Eric. From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com] Sent: Monday, January 28, 2019 3:56 AM To: MacDonald, Eric Cc: Peters, Matt; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Importance: High Hi Eric, That compute reboot issue is resolved, I connect controller & compute connection directly without switch in between after unlock. I don't know why it is not working with switch after unlock, Please check there is only hear-beat issue on mgmt network or something else, currently i'm using only 1 compute so its fine. Regards, Himanshu Goyal On Mon, Jan 28, 2019 at 12:33 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi Eric, Please find attached required mtcAgent.log containing one cycle of compute from PXE boot to unlock. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 10:12 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Sure Eric, currently i have left for the day, i will get back to you first thing on Monday morning. Many Thanks, Himanshu Goyal On Fri, Jan 25, 2019 at 8:17 PM MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>> wrote: Hi Himanshu, The mtcAgent.log shows that compute-0 has experienced the following errors several times over the span of almost 3 days. Configuration failures 2019-01-24T12:30:34.764 [44890.04799] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1046) enable_handler :Error : compute-0 configuration incomplete or failed (oob:2:1) 2019-01-24T12:30:34.764 [44890.04800] controller-0 mtcAgent --- nodeClass.cpp (1606) alarm_config_failure :Error : compute-0 critical config failure 2019-01-24T12:30:34.764 [44890.04801] controller-0 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : compute-0 setting critical 'Configuration' failure alarm (200.011 Out-Of-Service GoEnable Test Failures 2019-01-24T12:00:56.186 [44890.04700] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1163) enable_handler :Error : compute-0 got GOENABLED Experiencing Management network heartbeat failures. 2019-01-23T15:03:44.535 [44890.01949] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure 2019-01-23T15:03:45.688 [44890.01967] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss ** Since compute-0 is locked the in-service alarms against it will be suppressed. Since there are many errors over time I would recommend tailing the mtcAgent.log file to a new small file you can send to me. This file would contain all the logs that are produced for one cycle of an unlock of compute-0 This way we can focus on one case at a time. Eric. From: Himanshu Goyal [mailto:himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>] Sent: Friday, January 25, 2019 9:31 AM To: MacDonald, Eric Cc: Peters, Matt; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Importance: High Hi Eric, Below are required outputs of commands: fm alarm-list ###################################################### [root@controller-0 wrsroot(keystone_admin)]# fm alarm-list +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ | Alarm | Reason Text | Entity ID | Severity | Time Stamp | | ID | | | | | +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ | 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: | | 001 | | | | 00:12.159130 | | | | | | | | 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: | | 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 | | | | | | | | 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: | | 002 | no standby members available | service_group=cloud-services | | 12:07.245142 | | | | | | | | 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: | | 002 | but no standby members available | service_group=controller-services | | 11:40.174144 | | | | | | | | 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: | | 002 | standby members available | service_group=vim-services | | 11:39.052152 | | | | | | | | 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: | | 002 | standby members available | service_group=oam-services | | 11:38.728175 | | | | | | | | 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: | | 002 | but no standby members available | service_group=patching-services | | 11:37.756169 | | | | | | | | 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: | | 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 | | | | | | | | 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: | | 002 | only 1 active member available | service_group=web-services | | 11:37.189196 | | | | | | | | 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: | | 005 | | | | 11:36.690138 | | | | | | | | 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: | | 005 | | | | 11:36.427073 | | | | | | | +-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+ [root@controller-0 wrsroot(keystone_admin)]# ######################################################################################################### Please find attached file for fgrep Error /var/log/mtcAgent.log output Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>> wrote: fgrep Error /var/log/mtcAgent.log fm alarm-list From: Peters, Matt [mailto:Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>] Sent: Friday, January 25, 2019 8:05 AM To: Himanshu Goyal Cc: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) The DPDK and OVS status look fine now. I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Friday, January 25, 2019 at 3:56 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Matt, Please find below output of dpdk-devbind.py --status also: compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status ########################################################################################################## python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active* Other Network devices ===================== 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/var/log/puppet/latest# ################################################################################ Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Hi Matt, Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors. But compute still rebooting in loop. Please find attached dmesg log output for reference. ####################################################################### compute-0:/home/wrsroot# sudo ovs-vsctl show 8a1f7179-2014-4b4e-ad23-2065b8024964 Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "lldp64416a6d-7d" Interface "lldp64416a6d-7d" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:04:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" compute-0:/home/wrsroot# ############################################################################# Please suggest me any other logs that i can check. Regards, Himanshu Goyal On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> wrote: Thanks Matt, Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719, Please suggest which version of Bios we can use & this version is up-gradable or not with required version. ################################################################# compute-0:/home/wrsroot# dmidecode -t1 # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Intel Corporation Product Name: S2600WT2 Version: .................... Serial Number: ............ UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD Wake-up Type: Power Switch SKU Number: SKU Number Family: Family ############################################################################# ################################################################################ compute-0:/home/wrsroot# sudo dmidecode --type bios # dmidecode 3.0 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: Intel Corporation Version: SE5C610.86B.01.01.0014.121820151719 Release Date: 12/18/2015 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16384 kB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported USB legacy is supported LS-120 boot is supported ATAPI Zip drive boot is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 0.0 Firmware Revision: 0.0 Handle 0x000B, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 1 enUS Currently Installed Language: enUS ######################################################################################################## Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping). Maybe there are others on the distribution list that could offer additional suggestions. From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 11:32 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Please find output of required command in attached file. Regards, Himanshu Goyal Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: Can you confirm that VT-d is enabled properly by running the following? dmesg | grep -i -e DMAR -e IOMMU From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 8:19 AM To: "Peters, Matt" <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> Cc: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Peters, Error logs ovs-vswitched.log: ####################################### 2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0 2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe 2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type 2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0) 2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK 2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument) ######################################## Please find attached log file also. Regards, Himanshu Goyal On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com<mailto:Matt.Peters@windriver.com>> wrote: The name of eth0 is correct. It is just an assigned name by configuration management. What err logs are present in /var/log/openvswitch/ovs-vswitchd.log? From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, January 24, 2019 at 6:52 AM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Thanks a lot Yong, Checked OVS/DPDK status, It is giving me the error as output shown below: In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue. ################################################################################# ovs-vsctl show output: compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show 543d08f4-ff1e-4a8d-8e48-11f01356750d Manager "ptcp:6640:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "br-phy0" Interface "br-phy0" type: internal Port "lldpabeb30a6-6c" Interface "lldpabeb30a6-6c" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:02:00.0", n_rxq="2"} error: "Error attaching device '0000:02:00.0' to DPDK" Bridge br-int Controller "tcp:127.0.0.1:6633<http://127.0.0.1:6633>" is_connected: true fail_mode: secure Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} Port br-int Interface br-int type: internal ovs_version: "2.9.0" ##################################################################################################### ############################################################################## Output of dpdk-devbind.py: compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused= Network devices using kernel driver =================================== 0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active* 0000:04:00.0 'I350 Gigabit Network Connection 1521' if=enp4s0f0 drv=igb unused=vfio-pci 0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci Other Network devices ===================== <none> Crypto devices using DPDK-compatible driver =========================================== <none> Crypto devices using kernel driver ================================== <none> Other Crypto devices ==================== <none> Eventdev devices using DPDK-compatible driver ============================================= <none> Eventdev devices using kernel driver ==================================== <none> Other Eventdev devices ====================== <none> Mempool devices using DPDK-compatible driver ============================================ <none> Mempool devices using kernel driver =================================== <none> Other Mempool devices ===================== <none> compute-0:/usr/share/openvswitch/scripts$ ############################################################# Both management and OAM ports support DPDK. lspci output: ################################################## compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth 02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) 04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01) ############################################################### compute-0:/usr/share/openvswitch/scripts$ Many Thanks, Himanshu Goyal On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: I saw you enabled VT-D, how about VT-x in BIOS? Another possibility is that your ethernet NIC is not compatible with OVS/DPDK. Please follow these steps, to dig out more info 1. lock your compute-node 2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status” From: Himanshu Goyal <himanshugoyal500@gmail.com<mailto:himanshugoyal500@gmail.com>> Date: Thursday, 24 January 2019 at 4:46 PM To: "Hu, Yong" <yong.hu@intel.com<mailto:yong.hu@intel.com>> Cc: "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com<mailto:juan.carlos.alonso@intel.com>>, "starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>" <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) Hi Yong, I checked server BIOS setting Intel VT-D is enabled in my compute machine compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online. Please suggest how can i debug this. Many Thanks, Himanshu Goyal On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> wrote: Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know. There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
Thanks a lot Eric, I will check about management switch with our IT Team and revert back to you. Require 1 more help, Need 1 PCI-Passthrough NIC on VM. So please provide the steps to create VM with PCI-Passthrough NIC. Currently I'm following the below steps: Step1: system host-lock compute-0 step2: neutron providernet-create pci_net --type flat step3: system host-if-modify -m 1500 -n pci_0 -p "pci_net" -nt pci-passthrough compute-0 ens513f1 step3 is giving wrong command error, can you please provide me the correct commands for the same. or if there is any other to configure that Regards. Himanshu Goyal On Mon, Jan 28, 2019 at 8:03 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
From the logs it looks like compute enabled fine this time but then failed shortly after due to intermittent heartbeat loss.
The stages are …
*First mtcAlive with no indication of config failure ; a good thing*
2019-01-28T12:22:15.430 [44890.06760] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1056) enable_handler : Info : compute-0 is MTCALIVE (uptime:235 secs)
2019-01-28T12:22:15.430 fmAPI.cpp(471): Enqueue raise alarm request: UUID (c698824d-0d3e-42f0-a7a6-f705403ac8fe) alarm id (200.022) instant id (host=compute-0.status=online)
*Out-Of-Service In-Test passed ; a good thing*
2019-01-28T12:22:15.430 [44890.06761] controller-0 mtcAgent inv mtcInvApi.cpp (1085) mtcInvApi_update_state : Info : compute-0 intest (seq:19)
2019-01-28T12:22:27.557 [44890.06764] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1172) enable_handler : Info : compute-0 got GOENABLED
*Starting of local host services passed ; a good thing*
2019-01-28T12:22:27.562 [44890.06766] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1212) enable_handler : Info : compute-0 Starting Host Services
2019-01-28T12:22:27.628 [44890.06769] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (2586) host_services_handler : Info : compute-0 start compute host services completed
*Heartbeat client was immediately ready as expected ; a good thing*
2019-01-28T12:22:28.253 [44890.06770] controller-0 mtcAgent |-| nodeClass.cpp (4717) declare_service_ready : Info : compute-0 got hbsClient ready event
*Heartbeat start and 11 second trial soak passed ; a good thing*
2019-01-28T12:22:28.253 [44890.06771] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 804) send_hbs_command : Info : compute-0 sending 'start' to heartbeat service
2019-01-28T12:22:28.253 [44890.06772] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1327) enable_handler : Info : compute-0 Starting 11 sec Heartbeat Soak (with ready event)
2019-01-28T12:22:39.253 [44890.06773] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1341) enable_handler : Info : compute-0 heartbeating
*Compute-0 Enabled ok ; very good thing*
2019-01-28T12:22:39.502 [44890.06778] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1488) enable_handler : Info : compute-0 is ENABLE
*Intermittent heartbeat loss starts.*
*The after ~ 4 minutes compute-0 started experiencing Management network heartbeat loss*
2019-01-28T12:26:58.819 [44890.06781] controller-0 mtcAgent --- nodeClass.cpp (4613) manage_heartbeat_degrade: Warn : compute-0 Mgmnt *** Heartbeat Miss ***
2019-01-28T12:26:59.257 [44890.06792] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1637) recovery_handler : Info : compute-0 requesting mtcAlive with 5 sec timeout
2019-01-28T12:26:59.257 [44890.06794] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1655) recovery_handler : Info : compute-0 got requested mtcAlive
*Tries the 11 second heartbeat soak again after getting the requested mtcAlive message*
2019-01-28T12:26:59.257 [44890.06795] controller-0 mtcAgent |-| nodeClass.cpp (2480) stop_offline_handler : Info : compute-0 stopping offline handler (unlocked-enabled-failed) (stage:3)
2019-01-28T12:26:59.257 [44890.06796] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1722) recovery_handler : Warn : compute-0 Connectivity Recovered ; host did not reset
2019-01-28T12:26:59.257 [44890.06797] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1724) recovery_handler : Warn : compute-0 ... continuing with graceful recovery
2019-01-28T12:26:59.257 [44890.06798] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1725) recovery_handler : Warn : compute-0 ... with no affect to host services
2019-01-28T12:26:59.257 [44890.06799] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 804) send_hbs_command : Info : compute-0 sending 'start' to heartbeat service
2019-01-28T12:26:59.257 [44890.06800] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (2389) recovery_handler : Info : compute-0 Starting 11 sec Heartbeat Soak (with ready event)
*The heartbeat loss is intermittent*
*After first loss the recovery tried the 11 second soak again that failed*
2019-01-28T12:26:59.982 [44890.06802] controller-0 mtcAgent --- nodeClass.cpp (4613) manage_heartbeat_degrade: Warn : compute-0 Mgmnt *** Heartbeat Miss ***
2019-01-28T12:27:00.403 [44890.06803] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss ***
2019-01-28T12:27:00.403 [44890.06804] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure
2019-01-28T12:27:00.403 [44890.06805] controller-0 mtcAgent --- nodeClass.cpp (4515) manage_heartbeat_failure: Warn : compute-0 restarting graceful recovery
2019-01-28T12:27:00.403 [44890.06806] controller-0 mtcAgent |-| mtcNodeHdlrs.cpp (1557) recovery_handler : Info : compute-0 Graceful Recovery (uptime was 519)
*Heartbeat seems intermittent because the recovery algorithm reached its retry threshold of 3 and force failed the node*
2019-01-28T12:27:01.556 [44890.06828] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1592) recovery_handler :Error : compute-0 Graceful Recovery Failed (retries=3
2019-01-28T12:27:01.556 [44890.06829] controller-0 mtcAgent |-| nodeClass.cpp (7193) force_full_enable : Info : compute-0 Forcing Full Enable Sequence
2019-01-28T12:27:01.556 [44890.06830] controller-0 mtcAgent --- nodeClass.cpp (1644) alarm_enabled_failure :Error : compute-0 critical enable failure
The force failure caused a reboot and recovery retry where the logs ended.
Sequence will likely continue until the heartbeat (potentially multicast) issue is fixed.
Based on you saying that with the switch removed heartbeat passed I’d look at how the switch or your management network is handling multicast messaging and if the QOS settings on the network might be dropping packets.
Eric.
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Monday, January 28, 2019 3:56 AM *To:* MacDonald, Eric *Cc:* Peters, Matt; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) *Importance:* High
Hi Eric,
That compute reboot issue is resolved, I connect controller & compute connection directly without switch in between after unlock.
I don't know why it is not working with switch after unlock, Please check there is only hear-beat issue on mgmt network or something else, currently i'm using only 1 compute so its fine.
Regards,
Himanshu Goyal
On Mon, Jan 28, 2019 at 12:33 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Hi Eric,
Please find attached required mtcAgent.log containing one cycle of compute from PXE boot to unlock.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 10:12 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Sure Eric, currently i have left for the day, i will get back to you first thing on Monday morning.
Many Thanks,
Himanshu Goyal
On Fri, Jan 25, 2019 at 8:17 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
Hi Himanshu,
The mtcAgent.log shows that compute-0 has experienced the following errors several times over the span of almost 3 days.
*Configuration failures*
2019-01-24T12:30:34.764 [44890.04799] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1046) enable_handler :Error : compute-0 configuration incomplete or failed (oob:2:1)
2019-01-24T12:30:34.764 [44890.04800] controller-0 mtcAgent --- nodeClass.cpp (1606) alarm_config_failure :Error : compute-0 critical config failure
2019-01-24T12:30:34.764 [44890.04801] controller-0 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : compute-0 setting critical 'Configuration' failure alarm (200.011
*Out-Of-Service GoEnable Test Failures*
2019-01-24T12:00:56.186 [44890.04700] controller-0 mtcAgent hdl mtcNodeHdlrs.cpp (1163) enable_handler :Error : compute-0 got GOENABLED
*Experiencing Management network heartbeat failures.*
2019-01-23T15:03:44.535 [44890.01949] controller-0 mtcAgent --- nodeClass.cpp (4506) manage_heartbeat_failure:Error : compute-0 Mgmnt network heartbeat failure
2019-01-23T15:03:45.688 [44890.01967] controller-0 mtcAgent --- nodeClass.cpp (4495) manage_heartbeat_failure:Error : compute-0 Mgmnt *** Heartbeat Loss **
Since compute-0 is locked the in-service alarms against it will be suppressed.
Since there are many errors over time I would recommend tailing the mtcAgent.log file to a new small file you can send to me. This file would contain all the logs that are produced for one cycle of an unlock of compute-0
This way we can focus on one case at a time.
Eric.
*From:* Himanshu Goyal [mailto:himanshugoyal500@gmail.com] *Sent:* Friday, January 25, 2019 9:31 AM *To:* MacDonald, Eric *Cc:* Peters, Matt; starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop) *Importance:* High
Hi Eric,
Below are required outputs of commands:
*fm alarm-list *
######################################################
[root@controller-0 wrsroot(keystone_admin)]# fm alarm-list
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
| 200. | compute-0 was administratively locked to take it out-of-service. | host=compute-0 | warning | 2019-01-25T15: |
| 001 | | | | 00:12.159130 |
| | | | | |
| 300. | No enabled compute host with connectivity to provider network. | service=networking.providernet= | major | 2019-01-25T14: |
| 004 | | 4dbbc02e-c93c-494e-bd8d-1706eb09a3a6 | | 58:56.711665 |
| | | | | |
| 400. | Service group cloud-services loss of redundancy; expected 1 standby member but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | no standby members available | service_group=cloud-services | | 12:07.245142 |
| | | | | |
| 400. | Service group controller-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=controller-services | | 11:40.174144 |
| | | | | |
| 400. | Service group vim-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=vim-services | | 11:39.052152 |
| | | | | |
| 400. | Service group oam-services loss of redundancy; expected 1 standby member but no | service_domain=controller. | major | 2019-01-22T19: |
| 002 | standby members available | service_group=oam-services | | 11:38.728175 |
| | | | | |
| 400. | Service group patching-services loss of redundancy; expected 1 standby member | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but no standby members available | service_group=patching-services | | 11:37.756169 |
| | | | | |
| 400. | Service group directory-services loss of redundancy; expected 2 active members | service_domain=controller. | major | 2019-01-22T19: |
| 002 | but only 1 active member available | service_group=directory-services | | 11:37.513178 |
| | | | | |
| 400. | Service group web-services loss of redundancy; expected 2 active members but | service_domain=controller. | major | 2019-01-22T19: |
| 002 | only 1 active member available | service_group=web-services | | 11:37.189196 |
| | | | | |
| 400. | Communication failure detected with peer over port bond0 on host controller-0 | host=controller-0.network=mgmt | major | 2019-01-22T19: |
| 005 | | | | 11:36.690138 |
| | | | | |
| 400. | Communication failure detected with peer over port bond1 on host controller-0 | host=controller-0.network=oam | major | 2019-01-22T19: |
| 005 | | | | 11:36.427073 |
| | | | | |
+-------+---------------------------------------------------------------------------------+--------------------------------------+----------+----------------+
[root@controller-0 wrsroot(keystone_admin)]#
#########################################################################################################
Please find attached file for fgrep Error /var/log/mtcAgent.log output
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 7:43 PM MacDonald, Eric < Eric.MacDonald@windriver.com> wrote:
fgrep Error /var/log/mtcAgent.log
fm alarm-list
*From:* Peters, Matt [mailto:Matt.Peters@windriver.com] *Sent:* Friday, January 25, 2019 8:05 AM *To:* Himanshu Goyal *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
The DPDK and OVS status look fine now.
I would check the /var/log/mtcAgent.log on the active controller host to determine what is being reported for the reboot.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Friday, January 25, 2019 at 3:56 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Matt,
Please find below output of dpdk-devbind.py --status also:
compute-0:/var/log/puppet/latest# python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
##########################################################################################################
python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:04:00.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci *Active*
Other Network devices
=====================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' unused=vfio-pci
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/var/log/puppet/latest#
################################################################################
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 2:14 PM Himanshu Goyal <himanshugoyal500@gmail.com> wrote:
Hi Matt,
Sorry, interrupt remapping was disabled in my BIOS, I tried again after enable that, Now in dmesg & ovs-vsct there is no errors.* But compute still rebooting in loop*.
Please find attached dmesg log output for reference.
#######################################################################
compute-0:/home/wrsroot#* sudo ovs-vsctl show*
8a1f7179-2014-4b4e-ad23-2065b8024964
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "lldp64416a6d-7d"
Interface "lldp64416a6d-7d"
type: internal
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "eth0"
Interface "eth0"
type: dpdk
options: {dpdk-devargs="0000:04:00.0", n_rxq="2"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
ovs_version: "2.9.0"
compute-0:/home/wrsroot#
#############################################################################
Please suggest me any other logs that i can check.
Regards,
Himanshu Goyal
On Fri, Jan 25, 2019 at 12:21 PM Himanshu Goyal < himanshugoyal500@gmail.com> wrote:
Thanks Matt,
Below are the details of my server BIOS, Currently in my Server Bios Version: SE5C610.86B.01.01.0014.121820151719,
Please suggest which version of Bios we can use & this version is up-gradable or not with required version.
#################################################################
compute-0:/home/wrsroot# dmidecode -t1
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Intel Corporation
Product Name: S2600WT2
Version: ....................
Serial Number: ............
UUID: 803BBEFB-BFC1-E511-906E-0012795D96DD
Wake-up Type: Power Switch
SKU Number: SKU Number
Family: Family
#############################################################################
################################################################################
compute-0:/home/wrsroot# sudo dmidecode --type bios
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Intel Corporation
Version: SE5C610.86B.01.01.0014.121820151719
Release Date: 12/18/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16384 kB
Characteristics:
PCI is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
ACPI is supported
USB legacy is supported
LS-120 boot is supported
ATAPI Zip drive boot is supported
BIOS boot specification is supported
Function key-initiated network boot is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 0.0
Firmware Revision: 0.0
Handle 0x000B, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Abbreviated
Installable Languages: 1
enUS
Currently Installed Language: enUS
########################################################################################################
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 10:55 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Based on the logs, it looks like you have an older system that doesn’t fully support VT-d IOMMU remapping. You can try updating your BIOS if there is a newer version available. We don’t currently support setting arbitrary module parameters, so we don’t have a way to implement the workaround module param specified in the logs (i.e. disabling interrupt remapping).
Maybe there are others on the distribution list that could offer additional suggestions.
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 11:32 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Please find output of required command in attached file.
Regards,
Himanshu Goyal
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:56 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
Can you confirm that VT-d is enabled properly by running the following?
dmesg | grep -i -e DMAR -e IOMMU
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 8:19 AM *To: *"Peters, Matt" <Matt.Peters@windriver.com> *Cc: *"Hu, Yong" <yong.hu@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Peters,
Error logs ovs-vswitched.log:
#######################################
2019-01-24T18:42:04.213Z|00830|dpdk|INFO|EAL: PCI device 0000:02:00.0 on NUMA socket 0
2019-01-24T18:42:04.213Z|00831|dpdk|INFO|EAL: probe driver: 8086:10fb net_ixgbe
2019-01-24T18:42:04.215Z|00832|dpdk|ERR|EAL: 0000:02:00.0 failed to select IOMMU type
2019-01-24T18:42:04.215Z|00833|dpdk|ERR|EAL: Driver cannot attach the device (0000:02:00.0)
2019-01-24T18:42:04.215Z|00834|netdev_dpdk|WARN|Error attaching device '0000:02:00.0' to DPDK
2019-01-24T18:42:04.215Z|00835|netdev|WARN|eth0: could not set configuration (Invalid argument)
########################################
Please find attached log file also.
Regards,
Himanshu Goyal
On Thu, Jan 24, 2019 at 6:32 PM Peters, Matt <Matt.Peters@windriver.com> wrote:
The name of eth0 is correct. It is just an assigned name by configuration management.
What err logs are present in /var/log/openvswitch/ovs-vswitchd.log?
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, January 24, 2019 at 6:52 AM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Thanks a lot Yong,
Checked OVS/DPDK status, It is giving me the error as output shown below:
In br-phy0 is attaching a port name as eth0, but my compute port name is ens513f0, i think that may be the issue.
#################################################################################
*ovs-vsctl show output:*
compute-0:/usr/share/openvswitch/scripts$ sudo ovs-vsctl show
543d08f4-ff1e-4a8d-8e48-11f01356750d
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge "br-phy0"
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "phy-br-phy0"
Interface "phy-br-phy0"
type: patch
options: {peer="int-br-phy0"}
Port "br-phy0"
Interface "br-phy0"
type: internal
Port "lldpabeb30a6-6c"
Interface "lldpabeb30a6-6c"
type: internal
Port "eth0"
* Interface "eth0"*
type: dpdk
options: {dpdk-devargs="0000:02:00.0", n_rxq="2"}
*error: "Error attaching device '0000:02:00.0' to DPDK"*
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "int-br-phy0"
Interface "int-br-phy0"
type: patch
options: {peer="phy-br-phy0"}
Port br-int
Interface br-int
type: internal
ovs_version: "2.9.0"
#####################################################################################################
##############################################################################
*Output of dpdk-devbind.py:*
compute-0:/usr/share/openvswitch/scripts$ python /usr/share/openvswitch/scripts/dpdk-devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:02:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=ens513f1 drv=ixgbe unused=vfio-pci *Active*
0000:04:00.0 'I350 Gigabit Network Connection 1521' if=*enp4s0f0* drv=igb unused=vfio-pci
0000:04:00.3 'I350 Gigabit Network Connection 1521' if=enp4s0f3 drv=igb unused=vfio-pci
Other Network devices
=====================
<none>
Crypto devices using DPDK-compatible driver
===========================================
<none>
Crypto devices using kernel driver
==================================
<none>
Other Crypto devices
====================
<none>
Eventdev devices using DPDK-compatible driver
=============================================
<none>
Eventdev devices using kernel driver
====================================
<none>
Other Eventdev devices
======================
<none>
Mempool devices using DPDK-compatible driver
============================================
<none>
Mempool devices using kernel driver
===================================
<none>
Other Mempool devices
=====================
<none>
compute-0:/usr/share/openvswitch/scripts$
#############################################################
Both management and OAM ports support DPDK.
*lspci output:*
*##################################################*
compute-0:/usr/share/openvswitch/scripts$ lspci -nn | grep Eth
02:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
02:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
04:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
04:00.3 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
###############################################################
compute-0:/usr/share/openvswitch/scripts$
Many Thanks,
Himanshu Goyal
On Thu, Jan 24, 2019 at 2:24 PM Hu, Yong <yong.hu@intel.com> wrote:
I saw you enabled VT-D, how about VT-x in BIOS?
Another possibility is that your ethernet NIC is not compatible with OVS/DPDK.
Please follow these steps, to dig out more info
1. lock your compute-node
2. ssh to your compute-node and check OVS/DPDK status by “sudo ovs-vsctl show” and “python /usr/share/openvswitch/scripts/dpdk-devbind.py --status”
*From: *Himanshu Goyal <himanshugoyal500@gmail.com> *Date: *Thursday, 24 January 2019 at 4:46 PM *To: *"Hu, Yong" <yong.hu@intel.com> *Cc: *"Alonso, Juan Carlos" <juan.carlos.alonso@intel.com>, " starlingx-discuss@lists.starlingx.io" < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Deployment Option (error: compute boot in loop)
Hi Yong,
I checked server BIOS setting Intel VT-D is enabled in my compute machine
compute machine is rebooting only after unlocking, Before reboot it came in unlocked, enabled & online.
Please suggest how can i debug this.
Many Thanks,
Himanshu Goyal
On Wed, Jan 23, 2019 at 9:40 PM Hu, Yong <yong.hu@intel.com> wrote:
Intel_iommu=on and immmu are for PCI PT and SR-IOV, as far as I know.
There are other 2, like VT-x and VT-d flags, to be enabled. Please have a look in BIOS.
If you lock the compute it should stop the reboot loop. Then you can examine the logs. When it loops like that often it is reported in /var/log/puppet/latest/ M On Wed, 2019-01-23 at 16:03 +0530, Himanshu Goyal wrote:
Thanks a lot Juan and Yong,
Able to see compute host in "system-host-list" after connecting with a hub. starlingX Compute installation has been done. But after unlocking the compute nodes the compute node come into an endless boot loop.
participants (6)
-
Alonso, Juan Carlos
-
Himanshu Goyal
-
Hu, Yong
-
MacDonald, Eric
-
Michel Thebeau
-
Peters, Matt