[Starlingx-discuss] Starlingx-discuss Digest, Vol 9, Issue 54
Marcel Schaible
marcel at schaible-consulting.de
Thu Feb 14 12:39:14 UTC 2019
Hi John,
thanks for your help!
At the moment the controller-0 is stable and spontaneous rebooting.
The box is not reachable thru our lab net with the ip 10.62.150.211.
The box responds on ICMP at this address but the web server is not reachable.
The network config looks like this:
Any ideas what is wrong with that?
-0oot at controller-0 ~(keystone_admin)]# system host-if-list -a controller-0-
+--------------------------------------+-----------+----------+----------+---------+----------------+------------------------+-------------+---------------------------------+--------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+-----------+----------+----------+---------+----------------+------------------------+-------------+---------------------------------+--------------------+
| 04665f98-8318-4855-a591-1307231a40b3 | bond0 | platform | ae | None | [] | [u'ens1f3', u'ens1f1'] | [] | MTU=1500,AE_MODE=active_standby | [] |
| 2a6b70cb-53ad-43c3-adc7-b92c4c44470d | enp9s20f6 | None | ethernet | None | [u'enp9s20f6'] | [] | [] | MTU=1500 | [] |
| 2dcbe277-530e-400c-b156-43545b4507c8 | ens1f1 | None | ethernet | None | [u'ens1f1'] | [] | [u'bond0'] | MTU=1500 | [] |
| 3acd41aa-aeb0-400c-86ba-79f0fdb24829 | ens1f3 | None | ethernet | None | [u'ens1f3'] | [] | [u'bond0'] | MTU=1500 | [] |
| 86b8867a-2aee-4cb9-a04a-7587ff8d833d | enp3s0f0 | None | ethernet | None | [u'enp3s0f0'] | [] | [] | MTU=1500 | [] |
| ce78e71f-4300-4a6b-ae0a-43e3139a5515 | enp3s0f1 | None | ethernet | None | [u'enp3s0f1'] | [] | [] | MTU=1500 | [] |
| d3e101a0-a873-4c76-b7b5-83368626acd4 | enp9s21f0 | None | ethernet | None | [u'enp9s21f0'] | [] | [] | MTU=1500 | [] |
| e93f1d26-e6ee-4c7f-84ef-8b68e312ef16 | enp9s21f1 | data | ethernet | None | [u'enp9s21f1'] | [] | [] | MTU=1500,accelerated=True | [u'providernet-a'] |
| ed1b47c8-0311-4e39-8840-9931c69c147e | ens1f0 | None | ethernet | None | [u'ens1f0'] | [] | [] | MTU=1500 | [] |
| ee4ce037-1408-4695-b05d-e5491f30e274 | enp9s20f7 | platform | ethernet | None | [u'enp9s20f7'] | [] | [] | MTU=1500 | [] |
+--------------------------------------+-----------+----------+----------+---------+----------------+------------------------+-------------+---------------------------------+--------------------+
-0oot at controller-0 ~(keystone_admin)]# system host-ethernet-port-list controller-
+--------------------------------------+-----------+-------------------+--------------+-----------+----------+--------------------------------------------+----------+
| uuid | name | mac address | pci address | processor | auto neg | device type | boot i/f |
+--------------------------------------+-----------+-------------------+--------------+-----------+----------+--------------------------------------------+----------+
| b93a396b-0fcb-4f51-b121-9e85ded4b49d | enp3s0f0 | ec:9e:cd:1f:7e:b0 | 0000:03:00.0 | 0 | Yes | Ethernet Connection X552 10 GbE Backplane | False |
| e24e7cca-9e78-48a5-8d84-dd563fea2626 | enp3s0f1 | ec:9e:cd:1f:7e:b1 | 0000:03:00.1 | 0 | Yes | Ethernet Connection X552 10 GbE Backplane | False |
| 41a17a4d-f5eb-4dea-9bcb-0decb10137a4 | enp9s20f6 | 02:01:00:10:01:15 | 0000:09:14.6 | 0 | Yes | 82599 Ethernet Controller Virtual Function | False |
| 9d159107-2199-40bb-9e6b-189d6285ca4d | enp9s20f7 | 02:01:00:10:02:15 | 0000:09:14.7 | 0 | Yes | 82599 Ethernet Controller Virtual Function | True |
| c0bab514-5c77-48a7-97e5-8a9790df1584 | enp9s21f0 | 02:01:00:10:01:16 | 0000:09:15.0 | 0 | Yes | 82599 Ethernet Controller Virtual Function | False |
| 55d31ed0-3370-4b5b-822f-d2766d0b769b | enp9s21f1 | 02:01:00:10:02:16 | 0000:09:15.1 | 0 | Yes | 82599 Ethernet Controller Virtual Function | False |
| f4105b07-041e-4a75-90c5-f9f097c33616 | ens1f0 | b4:96:91:1a:01:60 | 0000:07:00.0 | 0 | Yes | I350 Gigabit Network Connection | False |
| 000f730d-626e-4087-9a3b-33ac673d688c | ens1f1 | b4:96:91:1a:01:61 | 0000:07:00.1 | 0 | Yes | I350 Gigabit Network Connection | False |
| cd202069-f5a0-43bb-a445-d3b9b58ce631 | ens1f3 | b4:96:91:1a:01:63 | 0000:07:00.3 | 0 | Yes | I350 Gigabit Network Connection | False |
+--------------------------------------+-----------+-------------------+--------------+-----------+----------+--------------------------------------------+----------+
[root at controller-0 ~(keystone_admin)]#
[root at controller-0 ~(keystone_admin)]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ec:9e:cd:1f:7e:b0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::ee9e:cdff:fe1f:7eb0/64 scope link
valid_lft forever preferred_lft forever
3: ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:96:91:1a:01:60 brd ff:ff:ff:ff:ff:ff
4: ens1f1: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
link/ether b4:96:91:1a:01:61 brd ff:ff:ff:ff:ff:ff
5: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether ec:9e:cd:1f:7e:b1 brd ff:ff:ff:ff:ff:ff
6: ens1f3: <NO-CARRIER,BROADCAST,MULTICAST,PROMISC,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
link/ether b4:96:91:1a:01:61 brd ff:ff:ff:ff:ff:ff
7: enp9s20f6: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 02:01:00:10:01:15 brd ff:ff:ff:ff:ff:ff
8: enp9s20f7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
link/ether 02:01:00:10:02:15 brd ff:ff:ff:ff:ff:ff
inet 172.27.1.3/24 brd 172.27.1.255 scope global enp9s20f7
valid_lft forever preferred_lft forever
inet 169.254.202.2/24 scope global enp9s20f7
valid_lft forever preferred_lft forever
inet 172.27.1.2/24 scope global secondary enp9s20f7
valid_lft forever preferred_lft forever
inet 172.27.1.5/24 brd 172.27.1.255 scope global secondary enp9s20f7
valid_lft forever preferred_lft forever
inet 172.27.1.6/24 brd 172.27.1.255 scope global secondary enp9s20f7
valid_lft forever preferred_lft forever
inet6 fe80::1:ff:fe10:215/64 scope link
valid_lft forever preferred_lft forever
9: enp9s21f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 02:01:00:10:01:16 brd ff:ff:ff:ff:ff:ff
11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b4:96:91:1a:01:61 brd ff:ff:ff:ff:ff:ff
inet 10.62.150.211/24 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::b696:91ff:fe1a:161/64 scope link
valid_lft forever preferred_lft forever
12: ovs-netdev: <BROADCAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether a2:a0:71:9e:02:81 brd ff:ff:ff:ff:ff:ff
13: br-int: <BROADCAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 06:dc:56:31:96:42 brd ff:ff:ff:ff:ff:ff
16: br-phy0: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether 56:e3:6b:8a:ff:45 brd ff:ff:ff:ff:ff:ff
inet6 fe80::54e3:6bff:fe8a:ff45/64 scope link
valid_lft forever preferred_lft forever
17: lldp55d31ed0-33: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether e6:e6:7b:46:1a:b2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e4e6:7bff:fe46:1ab2/64 scope link
valid_lft forever preferred_lft forever
[root at controller-0 ~(keystone_admin)]#
Thanks
Marcel
> "Kung, John" <John.Kung at windriver.com> hat am 13. Februar 2019 um 19:36 geschrieben:
>
>
> Marcel,
>
> controller-0 should be unlocked successfully for the first time before attempting to provision/configure controller-1
>
> As noted, the presence of 'Error' logs on controller-0 (under directory /var/log/puppet ) would indicate the reason for the configuration failure (200.011).
>
> In regards to the OAM interface configured, please check 'system host-if-list -a controller-0' and 'system host-ethernet-port-list controller-0' to determine the current interface configuration. Furthermore, after the 'host-unlock controller-0', the configured OAM interface IP should be reachable from your OAM network 'system host-addr-list controller-0' to see the addresses allocated for that host.
>
>
> John
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 13 Feb 2019 17:31:21 +0000
> From: "Penney, Don" <Don.Penney at windriver.com>
> To: Marcel Schaible <marcel at schaible-consulting.de>,
> "starlingx-discuss at lists.starlingx.io"
> <starlingx-discuss at lists.starlingx.io>
> Subject: Re: [Starlingx-discuss] Duplex Configuration: Unlock of
> controller-0 causes reboot
> Message-ID:
> <6703202FD9FDFF4A8DA9ACF104AE129FBA43D2C9 at ALA-MBD.corp.ad.wrs.com>
> Content-Type: text/plain; charset="utf-8"
>
> You'd need to look at the puppet log from the first reboot to see if some failure occurred there, or whether the reboot was triggered by a config change.
>
> I don't know if the configuration failure alarm is related to puppet or not. Given the OAM port alarm, you should double-check your OAM interface and configuration.
>
> Maybe John Kung has better advice on what to check... John?
>
> -----Original Message-----
> From: Marcel Schaible [mailto:marcel at schaible-consulting.de]
> Sent: Wednesday, February 13, 2019 12:27 PM
> To: Penney, Don; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
>
> Ok. The latest puppet logs from controller-0 do not contain any errors.
>
> Should I try to install a newer image?
>
>
> > "Penney, Don" <Don.Penney at windriver.com> hat am 13. Februar 2019 um 18:20 geschrieben:
> >
> >
> > I would suggest checking the puppet logs for a failure.... grep for Error. The redundancy alarms are because you don't have controller-1 yet, and can be ignored. The port failure alarm is concerning, I think... maybe that's related to the configuration error.
> >
> > -----Original Message-----
> > From: Marcel Schaible [mailto:marcel at schaible-consulting.de]
> > Sent: Wednesday, February 13, 2019 12:18 PM
> > To: Penney, Don; starlingx-discuss at lists.starlingx.io
> > Subject: RE: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
> >
> > At the moment the system seems to stablize after 2 reboots. I'll keep an eye on that.
> >
> > Is the alarm list looking reasonable after the controller-0 is unlocked and rebooted?
> > If yes, I would start to configure the controller-1.
> >
> >
> > root at controller-0 ~(keystone_admin)]# fm alarm-list
> > +----------+-------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+
> > | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
> > +----------+-------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+
> > | 200.011 | controller-0 experienced a configuration failure. | host=controller-0 | critical | 2019-02-13T18:57:13.772108 |
> > | 400.002 | Service group controller-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=controller-services | major | 2019-02-13T18:56:15.215083 |
> > | 400.002 | Service group vim-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=vim-services | major | 2019-02-13T18:56:15.174083 |
> > | 400.002 | Service group cloud-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=cloud-services | major | 2019-02-13T18:56:12.213065 |
> > | 400.002 | Service group oam-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=oam-services | major | 2019-02-13T18:56:05.330099 |
> > | 400.002 | Service group patching-services loss of redundancy; expected 1 standby member but no standby members available | service_domain=controller.service_group=patching-services | major | 2019-02-13T18:56:04.276424 |
> > | 400.002 | Service group directory-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=directory-services | major | 2019-02-13T18:56:04.195081 |
> > | 400.002 | Service group web-services loss of redundancy; expected 2 active members but only 1 active member available | service_domain=controller.service_group=web-services | major | 2019-02-13T18:56:04.114064 |
> > | 400.005 | Communication failure detected with peer over port enp9s20f7 on host controller-0 | host=controller-0.network=mgmt | major | 2019-02-13T18:56:03.790070 |
> > | 100.106 | 'OAM' Port failed. | host=controller-0.port=cd202069-f5a0-43bb-a445-d3b9b58ce631 | major | 2019-02-13T18:56:00.563067 |
> > | 100.107 | 'OAM' Interface degraded. | host=controller-0.interface=oam | major | 2019-02-13T18:56:00.515836 |
> > | 300.004 | No enabled compute host with connectivity to provider network. | service=networking.providernet=e9eef26b-d266-4b3f-b1b7-6d9c2cd3a86b | major | 2019-02-13T17:20:21.110405 |
> > +----------+-------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------+----------+----------------------------+
> > [root at controller-0 ~(keystone_admin)]#
> >
> >
> >
> > > "Penney, Don" <Don.Penney at windriver.com> hat am 13. Februar 2019 um 18:02 geschrieben:
> > >
> > >
> > > Ok, my updates related to the DRBD state change failures merged Feb 8th.
> > >
> > > I wouldn't expect to see an additional reboot. So what you're seeing, just to confirm, is:
> > > * install node, run config_controller, do config steps
> > > * system host-unlock controller-0
> > > * system reboots
> > > * system reboots a second time?
> >
> > Yes, exactly.
> >
> > >
> > > At what point is it rebooting a second time? Maybe a reboot is being triggered due to config changes made when the puppet manifests apply, which I think we could confirm from the puppet logs.
> >
> > The second reboot starts in about 1 minute after the login prompt is shown.
> >
> >
> > >
> > > -----Original Message-----
> > > From: Marcel Schaible [mailto:marcel at schaible-consulting.de]
> > > Sent: Wednesday, February 13, 2019 11:48 AM
> > > To: Penney, Don; starlingx-discuss at lists.starlingx.io
> > > Subject: Re: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
> > >
> > > I am using the build centos-20190206T060000Z.
> > >
> > > The main problem, beside of the noise is, that the system comes up and reboots immediately again.
> > >
> > >
> > >
> > > > "Penney, Don" <Don.Penney at windriver.com> hat am 13. Februar 2019 um 17:41 geschrieben:
> > > >
> > > >
> > > > Yes, this is the correct behavior.
> > > >
> > > > As for the drbd warnings, there are some state change failures that occur as the system shuts down, but aren't a problem (aside from the noise). I've managed to clean up most of these in a recent update. What build are you using?
> > > >
> > > > -----Original Message-----
> > > > From: Marcel Schaible [mailto:marcel at schaible-consulting.de]
> > > > Sent: Wednesday, February 13, 2019 11:28 AM
> > > > To: starlingx-discuss at lists.starlingx.io
> > > > Subject: [Starlingx-discuss] Duplex Configuration: Unlock of controller-0 causes reboot
> > > >
> > > > Hi,
> > > >
> > > > when I unlock the controller-0 after installation following this tutorial
> > > >
> > > > https://docs.starlingx.io/installation_guide/duplex.html#duplex
> > > >
> > > > the system is rebooting automatically with a lot of errors from drdb.
> > > >
> > > > Controller-1 is not configured at this point in time.
> > > >
> > > > Must I configure the secondary controller before unlocking controller-0?
> > > >
> > > > Thanks
> > > >
> > > > Marcel
> > > >
> > > > _______________________________________________
> > > > Starlingx-discuss mailing list
> > > > Starlingx-discuss at lists.starlingx.io
> > > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> > >
> > > _______________________________________________
> > > Starlingx-discuss mailing list
> > > Starlingx-discuss at lists.starlingx.io
> > > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
More information about the Starlingx-discuss
mailing list