Hi John, thanks for your help! At the moment the controller-0 is stable and spontaneous rebooting. The box is not reachable thru our lab net with the ip 10.62.150.211. The box responds on ICMP at this address but the web server is not reachable. The network config looks like this: Any ideas what is wrong with that? -0oot@controller-0 ~(keystone_admin)]# system host-if-list -a controller-0- +--------------------------------------+-----------+----------+----------+---------+----------------+------------------------+-------------+---------------------------------+--------------------+ | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks | +--------------------------------------+-----------+----------+----------+---------+----------------+------------------------+-------------+---------------------------------+--------------------+ | 04665f98-8318-4855-a591-1307231a40b3 | bond0 | platform | ae | None | [] | [u'ens1f3', u'ens1f1'] | [] | MTU=1500,AE_MODE=active_standby | [] | | 2a6b70cb-53ad-43c3-adc7-b92c4c44470d | enp9s20f6 | None | ethernet | None | [u'enp9s20f6'] | [] | [] | MTU=1500 | [] | | 2dcbe277-530e-400c-b156-43545b4507c8 | ens1f1 | None | ethernet | None | [u'ens1f1'] | [] | [u'bond0'] | MTU=1500 | [] | | 3acd41aa-aeb0-400c-86ba-79f0fdb24829 | ens1f3 | None | ethernet | None | [u'ens1f3'] | [] | [u'bond0'] | MTU=1500 | [] | | 86b8867a-2aee-4cb9-a04a-7587ff8d833d | enp3s0f0 | None | ethernet | None | [u'enp3s0f0'] | [] | [] | MTU=1500 | [] | | ce78e71f-4300-4a6b-ae0a-43e3139a5515 | enp3s0f1 | None | ethernet | None | [u'enp3s0f1'] | [] | [] | MTU=1500 | [] | | d3e101a0-a873-4c76-b7b5-83368626acd4 | enp9s21f0 | None | ethernet | None | [u'enp9s21f0'] | [] | [] | MTU=1500 | [] | | e93f1d26-e6ee-4c7f-84ef-8b68e312ef16 | enp9s21f1 | data | ethernet | None | [u'enp9s21f1'] | [] | [] | MTU=1500,accelerated=True | [u'providernet-a'] | | ed1b47c8-0311-4e39-8840-9931c69c147e | ens1f0 | None | ethernet | None | [u'ens1f0'] | [] | [] | MTU=1500 | [] | | ee4ce037-1408-4695-b05d-e5491f30e274 | enp9s20f7 | platform | ethernet | None | [u'enp9s20f7'] | [] | [] | MTU=1500 | [] | +--------------------------------------+-----------+----------+----------+---------+----------------+------------------------+-------------+---------------------------------+--------------------+ -0oot@controller-0 ~(keystone_admin)]# system host-ethernet-port-list controller- +--------------------------------------+-----------+-------------------+--------------+-----------+----------+--------------------------------------------+----------+ | uuid | name | mac address | pci address | processor | auto neg | device type | boot i/f | +--------------------------------------+-----------+-------------------+--------------+-----------+----------+--------------------------------------------+----------+ | b93a396b-0fcb-4f51-b121-9e85ded4b49d | enp3s0f0 | ec:9e:cd:1f:7e:b0 | 0000:03:00.0 | 0 | Yes | Ethernet Connection X552 10 GbE Backplane | False | | e24e7cca-9e78-48a5-8d84-dd563fea2626 | enp3s0f1 | ec:9e:cd:1f:7e:b1 | 0000:03:00.1 | 0 | Yes | Ethernet Connection X552 10 GbE Backplane | False | | 41a17a4d-f5eb-4dea-9bcb-0decb10137a4 | enp9s20f6 | 02:01:00:10:01:15 | 0000:09:14.6 | 0 | Yes | 82599 Ethernet Controller Virtual Function | False | | 9d159107-2199-40bb-9e6b-189d6285ca4d | enp9s20f7 | 02:01:00:10:02:15 | 0000:09:14.7 | 0 | Yes | 82599 Ethernet Controller Virtual Function | True | | c0bab514-5c77-48a7-97e5-8a9790df1584 | enp9s21f0 | 02:01:00:10:01:16 | 0000:09:15.0 | 0 | Yes | 82599 Ethernet Controller Virtual Function | False | | 55d31ed0-3370-4b5b-822f-d2766d0b769b | enp9s21f1 | 02:01:00:10:02:16 | 0000:09:15.1 | 0 | Yes | 82599 Ethernet Controller Virtual Function | False | | f4105b07-041e-4a75-90c5-f9f097c33616 | ens1f0 | b4:96:91:1a:01:60 | 0000:07:00.0 | 0 | Yes | I350 Gigabit Network Connection | False | | 000f730d-626e-4087-9a3b-33ac673d688c | ens1f1 | b4:96:91:1a:01:61 | 0000:07:00.1 | 0 | Yes | I350 Gigabit Network Connection | False | | cd202069-f5a0-43bb-a445-d3b9b58ce631 | ens1f3 | b4:96:91:1a:01:63 | 0000:07:00.3 | 0 | Yes | I350 Gigabit Network Connection | False | +--------------------------------------+-----------+-------------------+--------------+-----------+----------+--------------------------------------------+----------+ [root@controller-0 ~(keystone_admin)]# [root@controller-0 ~(keystone_admin)]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether ec:9e:cd:1f:7e:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::ee9e:cdff:fe1f:7eb0/64 scope link valid_lft forever preferred_lft forever 3: ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether b4:96:91:1a:01:60 brd ff:ff:ff:ff:ff:ff 4: ens1f1: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether b4:96:91:1a:01:61 brd ff:ff:ff:ff:ff:ff 5: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ec:9e:cd:1f:7e:b1 brd ff:ff:ff:ff:ff:ff 6: ens1f3: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000 link/ether b4:96:91:1a:01:61 brd ff:ff:ff:ff:ff:ff 7: enp9s20f6: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 02:01:00:10:01:15 brd ff:ff:ff:ff:ff:ff 8: enp9s20f7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000 link/ether 02:01:00:10:02:15 brd ff:ff:ff:ff:ff:ff inet 172.27.1.3/24 brd 172.27.1.255 scope global enp9s20f7 valid_lft forever preferred_lft forever inet 169.254.202.2/24 scope global enp9s20f7 valid_lft forever preferred_lft forever inet 172.27.1.2/24 scope global secondary enp9s20f7 valid_lft forever preferred_lft forever inet 172.27.1.5/24 brd 172.27.1.255 scope global secondary enp9s20f7 valid_lft forever preferred_lft forever inet 172.27.1.6/24 brd 172.27.1.255 scope global secondary enp9s20f7 valid_lft forever preferred_lft forever inet6 fe80::1:ff:fe10:215/64 scope link valid_lft forever preferred_lft forever 9: enp9s21f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 02:01:00:10:01:16 brd ff:ff:ff:ff:ff:ff 11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b4:96:91:1a:01:61 brd ff:ff:ff:ff:ff:ff inet 10.62.150.211/24 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::b696:91ff:fe1a:161/64 scope link valid_lft forever preferred_lft forever 12: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether a2:a0:71:9e:02:81 brd ff:ff:ff:ff:ff:ff 13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether 06:dc:56:31:96:42 brd ff:ff:ff:ff:ff:ff 16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether 56:e3:6b:8a:ff:45 brd ff:ff:ff:ff:ff:ff inet6 fe80::54e3:6bff:fe8a:ff45/64 scope link valid_lft forever preferred_lft forever 17: lldp55d31ed0-33: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether e6:e6:7b:46:1a:b2 brd ff:ff:ff:ff:ff:ff inet6 fe80::e4e6:7bff:fe46:1ab2/64 scope link valid_lft forever preferred_lft forever [root@controller-0 ~(keystone_admin)]# Thanks Marcel
"Kung, John" <John.Kung@windriver.com> hat am 13. Februar 2019 um 19:36 geschrieben:
Marcel,
controller-0 should be unlocked successfully for the first time before attempting to provision/configure controller-1
As noted, the presence of 'Error' logs on controller-0 (under directory /var/log/puppet ) would indicate the reason for the configuration failure (200.011).
In regards to the OAM interface configured, please check 'system host-if-list -a controller-0' and 'system host-ethernet-port-list controller-0' to determine the current interface configuration. Furthermore, after the 'host-unlock controller-0', the configured OAM interface IP should be reachable from your OAM network 'system host-addr-list controller-0' to see the addresses allocated for that host.
John ----------------------------------------------------------------------
Message: 1 Date: Wed, 13 Feb 2019 17:31:21 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: Marcel Schaible <marcel@schaible-consulting.de>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA43D2C9@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You'd need to look at the puppet log from the first reboot to see if some failure occurred there, or whether the reboot was triggered by a config change.
I don't know if the configuration failure alarm is related to puppet or not. Given the OAM port alarm, you should double-check your OAM interface and configuration.
Maybe John Kung has better advice on what to check... John?
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Wednesday, February 13, 2019 12:27 PM To: Penney, Don; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
Ok. The latest puppet logs from controller-0 do not contain any errors.
Should I try to install a newer image?
"Penney, Don" <Don.Penney@windriver.com> hat am 13. Februar 2019 um 18:20 geschrieben:
I would suggest checking the puppet logs for a failure.... grep for Error. The redundancy alarms are because you don't have controller-1 yet, and can be ignored. The port failure alarm is concerning, I think... maybe that's related to the configuration error.
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Wednesday, February 13, 2019 12:18 PM To: Penney, Don; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
At the moment the system seems to stablize after 2 reboots. I'll keep an eye on that.
Is the alarm list looking reasonable after the controller-0 is unlocked and rebooted? If yes, I would start to configure the controller-1.
root@controller-0 ~(keystone_admin)]# fm alarm-list +----------+-------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+ | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | +----------+-------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+ | 200.011 | controller-0 experienced a configuration failure. | host=controller-0 | critical | 2019-02-13T18:57:13.772108 | | 400.002 | Service group controller-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=controller-services | major | 2019-02-13T18:56:15.215083 | | 400.002 | Service group vim-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=vim-services | major | 2019-02-13T18:56:15.174083 | | 400.002 | Service group cloud-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=cloud-services | major | 2019-02-13T18:56:12.213065 | | 400.002 | Service group oam-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=oam-services | major | 2019-02-13T18:56:05.330099 | | 400.002 | Service group patching-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=patching-services | major | 2019-02-13T18:56:04.276424 | | 400.002 | Service group directory-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=directory-services | major | 2019-02-13T18:56:04.195081 | | 400.002 | Service group web-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=web-services | major | 2019-02-13T18:56:04.114064 | | 400.005 | Communication failure detected with peer over port enp9s20f7 on host controller-0 | host=controller-0.network=mgmt | major | 2019-02-13T18:56:03.790070 | | 100.106 | 'OAM' Port failed. | host=controller-0.port=cd202069-f5a0-43bb-a445-d3b9b58ce631 | major | 2019-02-13T18:56:00.563067 | | 100.107 | 'OAM' Interface degraded. | host=controller-0.interface=oam | major | 2019-02-13T18:56:00.515836 | | 300.004 | No enabled compute host with connectivity to provider network. | service=networking.providernet=e9eef26b-d266-4b3f-b1b7-6d9c2cd3a86b | major | 2019-02-13T17:20:21.110405 | +----------+-------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+ [root@controller-0 ~(keystone_admin)]#
"Penney, Don" <Don.Penney@windriver.com> hat am 13. Februar 2019 um 18:02 geschrieben:
Ok, my updates related to the DRBD state change failures merged Feb 8th.
I wouldn't expect to see an additional reboot. So what you're seeing, just to confirm, is: * install node, run config_controller, do config steps * system host-unlock controller-0 * system reboots * system reboots a second time?
Yes, exactly.
At what point is it rebooting a second time? Maybe a reboot is being triggered due to config changes made when the puppet manifests apply, which I think we could confirm from the puppet logs.
The second reboot starts in about 1 minute after the login prompt is shown.
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Wednesday, February 13, 2019 11:48 AM To: Penney, Don; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
I am using the build centos-20190206T060000Z.
The main problem, beside of the noise is, that the system comes up and reboots immediately again.
"Penney, Don" <Don.Penney@windriver.com> hat am 13. Februar 2019 um 17:41 geschrieben:
Yes, this is the correct behavior.
As for the drbd warnings, there are some state change failures that occur as the system shuts down, but aren't a problem (aside from the noise). I've managed to clean up most of these in a recent update. What build are you using?
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Wednesday, February 13, 2019 11:28 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
Hi,
when I unlock the controller-0 after installation following this tutorial
https://docs.starlingx.io/installation_guide/duplex.html#duplex
the system is rebooting automatically with a lot of errors from drdb.
Controller-1 is not configured at this point in time.
Must I configure the secondary controller before unlocking controller-0?
Thanks
Marcel
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss