[Starlingx-discuss] 2nd node stuck at pxe installation
Yang, Bin
bin.yang at intel.com
Fri May 3 01:50:12 UTC 2019
On Thu, 2019-05-02 at 13:31 +0000, Penney, Don wrote:
> Running a "collect" and providing the tarball might help us to understand
> what's happening. Can you provide a "system host-if-list" for your controllers,
> at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config
I had reproduced it on my machine.
Here is the collect tarball: https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4z
fKmu-IOqddLrH/view?usp=sharing
8<-------------------------------------------------------------------------------
-
[wrsroot at controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | None | None | locked | disabled | offline |
| 3 | None | None | locked | disabled | offline |
| 4 | controller-1 | controller | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+--------------+
[wrsroot at controller-0 ~(keystone_admin)]$ system host-if-list 1
+--------------------------------------+------+----------+----------+---------+
-----------+----------+-------------+------------+---------------+
| uuid | name | class | type | vlan id |
ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+------+----------+----------+---------+
-----------+----------+-------------+------------+---------------+
| 53173371-999c-4724-8df0-550b9ed283e0 | ens3 | platform | ethernet | None |
[u'ens3'] | [] | [] | MTU=1500 | [] |
| 8bc280a4-c7fe-41f1-8771-8f951f62a053 | ens4 | platform | ethernet | None |
[u'ens4'] | [] | [] | MTU=1500 | [] |
+--------------------------------------+------+----------+----------+---------+
-----------+----------+-------------+------------+---------------+
cat /opt/platform/config/19.01/cgcs_co
[cSYSTEM]
# System Configuration
SYSTEM_MODE=duplex
TIMEZONE=UTC
[cPXEBOOT]
# PXEBoot Network Support Configuration
PXECONTROLLER_FLOATING_HOSTNAME=pxecontroller
[cMGMT]
# Management Network Configuration
MANAGEMENT_INTERFACE_NAME=ens4
MANAGEMENT_INTERFACE=ens4
MANAGEMENT_MTU=1500
MANAGEMENT_SUBNET=192.168.204.0/24
LAG_MANAGEMENT_INTERFACE=no
CONTROLLER_FLOATING_ADDRESS=192.168.204.2
CONTROLLER_0_ADDRESS=192.168.204.3
CONTROLLER_1_ADDRESS=192.168.204.4
NFS_MANAGEMENT_ADDRESS_1=192.168.204.5
NFS_MANAGEMENT_ADDRESS_2=192.168.204.6
CONTROLLER_FLOATING_HOSTNAME=controller
CONTROLLER_HOSTNAME_PREFIX=controller-
OAMCONTROLLER_FLOATING_HOSTNAME=oamcontroller
DYNAMIC_ADDRESS_ALLOCATION=yes
MANAGEMENT_MULTICAST_SUBNET=239.1.1.0/28
[cINFRA]
# Infrastructure Network Configuration
INFRASTRUCTURE_INTERFACE_NAME=NC
INFRASTRUCTURE_INTERFACE=NC
INFRASTRUCTURE_VLAN=NC
INFRASTRUCTURE_MTU=NC
INFRASTRUCTURE_SUBNET=NC
LAG_INFRASTRUCTURE_INTERFACE=no
INFRASTRUCTURE_BOND_MEMBER_0=NC
INFRASTRUCTURE_BOND_MEMBER_1=NC
INFRASTRUCTURE_BOND_POLICY=NC
CONTROLLER_0_INFRASTRUCTURE_ADDRESS=NC
CONTROLLER_1_INFRASTRUCTURE_ADDRESS=NC
NFS_INFRASTRUCTURE_ADDRESS_1=NC
STORAGE_0_INFRASTRUCTURE_ADDRESS=NC
STORAGE_1_INFRASTRUCTURE_ADDRESS=NC
CONTROLLER_INFRASTRUCTURE_HOSTNAME_SUFFIX=NC
INFRASTRUCTURE_START_ADDRESS=NC
INFRASTRUCTURE_END_ADDRESS=NC
[cCLUSTER]
# Cluster Host Network Configuration
CLUSTER_INTERFACE_NAME=ens4
CLUSTER_INTERFACE=ens4
CLUSTER_VLAN=NC
CLUSTER_MTU=1500
CLUSTER_SUBNET=192.168.206.0/24
LAG_CLUSTER_INTERFACE=no
[cEXT_OAM]
# External OAM Network Configuration
EXTERNAL_OAM_INTERFACE_NAME=ens3
EXTERNAL_OAM_INTERFACE=ens3
EXTERNAL_OAM_VLAN=NC
EXTERNAL_OAM_MTU=1500
LAG_EXTERNAL_OAM_INTERFACE=no
EXTERNAL_OAM_SUBNET=10.10.10.0/24
EXTERNAL_OAM_GATEWAY_ADDRESS=10.10.10.1
EXTERNAL_OAM_FLOATING_ADDRESS=10.10.10.2
EXTERNAL_OAM_0_ADDRESS=10.10.10.3
EXTERNAL_OAM_1_ADDRESS=10.10.10.4
[cDNS]
# DNS Configuration
NAMESERVER_1=10.248.2.1
NAMESERVER_2=NC
NAMESERVER_3=NC
[cDOCKER_REGISTRY]
# Docker Registry Configuration
DOCKER_K8S_REGISTRY=edgehost01.sh.intel.com:5000
DOCKER_GCR_REGISTRY=edgehost01.sh.intel.com:5000
DOCKER_QUAY_REGISTRY=edgehost01.sh.intel.com:5000
DOCKER_DOCKER_REGISTRY=edgehost01.sh.intel.com:5000
IS_SECURE_REGISTRY=False
[cSECURITY]
[cREGION]
# Region Configuration
REGION_CONFIG=False
[wrsroot at controller-0 ~(keystone_admin)]$
>
> -----Original Message-----
> From: Sun, Austin [mailto:austin.sun at intel.com]
> Sent: Wednesday, May 01, 2019 10:08 PM
> To: Penney, Don; Marcel Schaible; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
>
> Hi Penney:
> Yes. pxecontrooler IP is active on controller-0 management interface
> . The environment is virtual environment .
>
> The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it
> is already failed. From my understanding , the ip in controller-1 when
> install.
> The ip should be in range pxe subnet.
>
> Thanks.
> BR
> Austin Sun.
>
> -----Original Message-----
> From: Penney, Don [mailto:Don.Penney at windriver.com]
> Sent: Wednesday, May 1, 2019 10:11 PM
> To: Sun, Austin <austin.sun at intel.com>; Marcel Schaible <marcel at schaible-consul
> ting.de>; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
>
> Is the pxecontroller IP active on your management interface? For example (if
> your management isn't on a vlan):
>
> controller-0:/home/wrsroot# grep management_interface
> /etc/platform/platform.conf
> management_interface=eno2
> controller-0:/home/wrsroot# grep pxecontroller /etc/hosts
> 169.254.202.2 pxecontroller
> controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2
> inet 169.254.202.2/24 scope global eno2
>
>
> If not, is the pxeboot-ip service in SM active?
> controller-0:/home/wrsroot# sm-dump |grep pxe
> pxeboot-ip enabled-active enabled-active
>
>
> Cheers,
> Don.
>
>
> -----Original Message-----
> From: Penney, Don
> Sent: Wednesday, May 01, 2019 9:54 AM
> To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
>
> The pxecontroller address is purposefully on a separate subnet. Our networking
> team could provide a better answer, but I believe it was due to issues with
> pxeboot and VLANs. This address should absolutely be available from the
> installer.
>
>
> -----Original Message-----
> From: Sun, Austin [mailto:austin.sun at intel.com]
> Sent: Tuesday, April 30, 2019 10:59 PM
> To: Penney, Don; Marcel Schaible; starlingx-discuss at lists.starlingx.io
> Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
>
> Hi Penny :
> Thanks a lot. some update from my side.
> From log Dracut [1]; squashfs.img was not successfully download on controller-
> 1.
> And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl
> command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o
> mytest.img" can successfully download squashfs.img .
> But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/s
> quashfs.img" , it will failed.
>
> From controller "nslookup pxecontroller" , the pxecontroller ip addr is
> 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense
> pxecontroller can not be accessed by controller-1(pxe boot stage).
>
> If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip
> (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be
> installed successfully.
>
> BTW: This issue is 100% reproducible on certain servers, but not for all kinds
> of servers.
>
> [1] http://paste.openstack.org/show/750134/
> [2] http://paste.openstack.org/show/750135/
>
>
> Thanks.
> BR
> Austin Sun.
>
> -----Original Message-----
> From: Penney, Don [mailto:Don.Penney at windriver.com]
> Sent: Tuesday, April 30, 2019 11:04 PM
> To: Marcel Schaible <marcel at schaible-consulting.de>; starlingx-discuss at lists.st
> arlingx.io
> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
>
> You can also look in /var/log/daemon.log for TFTP logs showing the requests and
> transfers of files, and /www/var/log/lighttpd-access.log for transfer of the
> squashfs.img.
>
> Since the console showed "Loading rel-19.01/installer-initrd...........ready",
> it seems like the TFTP transfer was fine, which is why I suggested modifying
> the console setting to ensure you're seeing what's happening. This is what I'd
> expect to see if the console is set to serial, but there's no serial console
> attached (such as using virtual box without a serial console).
>
> For example, you can see the bzImage and initrd here being transferred via TFTP
> in daemon.log:
> 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent
> /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24
> 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent
> /pxeboot/rel-19.01/installer-initrd to 192.168.202.24
>
> followed by the squashfs.img in /www/var/log/lighttpd-access.log:
> 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET
> /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0"
>
> If only the bzImage and initrd is transferred, without seeing the logs on
> console, my guess would be an unsupported NIC preventing the initrd from
> accessing the management network in order to request the squashfs.img (which
> provides the installer rootfs)
>
>
> -----Original Message-----
> From: Marcel Schaible [mailto:marcel at schaible-consulting.de]
> Sent: Tuesday, April 30, 2019 10:46 AM
> To: starlingx-discuss at lists.starlingx.io
> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
>
> Hi,
>
> we have the same problem with the image from 20190411. It Looks like that
> controller-1 is not finding the pxe boot image.
>
> Kind regards
>
> Marcel
>
> >
> > Message: 1
> > Date: Tue, 30 Apr 2019 13:54:17 +0000
> > From: "Penney, Don" <Don.Penney at windriver.com>
> > To: "Lin, Shuicheng" <shuicheng.lin at intel.com>,
> > "starlingx-discuss at lists.starlingx.io"
> > <starlingx-discuss at lists.starlingx.io>
> > Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
> > Message-ID:
> > <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86 at ALA-MBD.corp.ad.wrs.com>
> > Content-Type: text/plain; charset="utf-8"
> >
> > You can set the console either via the web browser in the installation
> > parameters, or at command-line, such as:
> > system host-update 2 personality=controller console=
> >
> > The default is the serial console, "ttyS0,115200". If you're using a
> > graphical console, you can set it blank, or console=tty0, for example. This
> > should allow you to see if there are some errors occurring trying to load the
> > installer.
> >
> >
> > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> > Sent: Tuesday, April 30, 2019 1:03 AM
> > To: starlingx-discuss at lists.starlingx.io
> > Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
> >
> > Hi all,
> > I try to do duplex deploy, and find controller-1 stuck at pxe installation
> > step.
> > The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But
> > package installation is not started as host-show command shown in controller-
> > 0.
> > I meet this issue with latest code. Do you know what cause the issue, or how
> > to debug it?
> > Thanks.
> >
> > [wrsroot at controller-0 ~(keystone_admin)]$ system host-list
> > +----+--------------+-------------+----------------+-------------+-----------
> > ---+
> > >
> > > id | hostname | personality | administrative | operational |
> > > availability |
> > +----+--------------+-------------+----------------+-------------+-----------
> > ---+
> > >
> > > 1 | controller-0 | controller | unlocked | enabled |
> > > available |
> > > 2 | controller-1 | controller | locked | disabled |
> > > offline |
> > +----+--------------+-------------+----------------+-------------+-----------
> > ---+
> > [wrsroot at controller-0 ~(keystone_admin)]$ system host-show
> > controller-1 | grep install
> > >
> > > install_output | text |
> > > install_state | None |
> > > install_state_info | None |
> > > subfunction_avail | not-installed
> > Best Regards
> > Shuicheng
> >
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
More information about the Starlingx-discuss
mailing list