Re: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan): controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2 If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active Cheers, Don. -----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment . The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet. Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan): controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2 If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active Cheers, Don. -----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment . The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet. Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan): controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2 If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active Cheers, Don. -----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
We solved our problem by setting the management network start and end address during Installation of controller-0. The default ip address 172.27.0.2 collides on our server with a dhcp server running on non-starlingx managament cpu.
"Penney, Don" <Don.Penney@windriver.com> hat am 2. Mai 2019 um 15:31 geschrieben:
Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment .
The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet.
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan):
controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2
If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active
Cheers, Don.
-----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer.
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed.
From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage).
If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully.
BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers.
[1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img.
Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console).
For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24
followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0"
If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs)
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi,
we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image.
Kind regards
Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
The management network is supposed to be a separate network and should be isolated from other nodes/systems, so there shouldn't be any collisions. Having another DHCP server on the management network could especially be problematic. -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Thursday, May 02, 2019 9:40 AM To: Penney, Don; Sun, Austin; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation We solved our problem by setting the management network start and end address during Installation of controller-0. The default ip address 172.27.0.2 collides on our server with a dhcp server running on non-starlingx managament cpu.
"Penney, Don" <Don.Penney@windriver.com> hat am 2. Mai 2019 um 15:31 geschrieben:
Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment .
The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet.
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan):
controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2
If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active
Cheers, Don.
-----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer.
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed.
From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage).
If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully.
BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers.
[1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img.
Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console).
For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24
followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0"
If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs)
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi,
we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image.
Kind regards
Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Penney: Thanks . The collect tar is attached to [1] The controller-0 is ok. controller-1 is stuck on pxe installation . [1] https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4zfKmu-IOqddLrH/view?usp=s... Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Thursday, May 2, 2019 9:32 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment . The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet. Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan): controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2 If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active Cheers, Don. -----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Austin, I'm hoping to hear from some folks more familiar with this, but I noticed a couple of things from your logs. * controller-0 received DHCP request from controller-1 at 09:43 for MAC 52:54:00:3b:0f:84 and sent back an IP address on the pxeboot subnet * sysinv should add an entry to the hosts table at this point, as the node has been "discovered". At 09:53:25, you did a "system host-add", defining the host associated with MAC 52:54:00:3b:0f:84 as a controller, at which point the controller-1 (with its IP from DHCP on pxeboot network) TFTPed the bzImage and initrd. Controller-1 would have then loaded the kernel (bzImage) and initrd and reinitialized the network interface, sending a new DHCP request, which happened at 09:53:50. At this point, however, the DHCP response provided 192.168.204.4 as the IP address, which was then unable to reach the pxecontroller IP. My guess is that this is due to the host-add command, which I believe is intended for provisioning hosts BEFORE they're powered on and start network booting. If the node boots before this and is "discovered" by sysinv, you should instead be using the "system host-update" command to modify/configure the discovered host, which you'd see in the "system host-list" output after it is discovered. My suggestion would be to try this instead. You could shutdown controller-1 and reinstall controller-0 to start fresh, or maybe you can delete the host from sysinv (system host-delete) and try again. Let the host network boot to get discovered, then do the system host-update to configure its personality as "controller". -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Thursday, May 02, 2019 9:44 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penney: Thanks . The collect tar is attached to [1] The controller-0 is ok. controller-1 is stuck on pxe installation . [1] https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4zfKmu-IOqddLrH/view?usp=s... Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Thursday, May 2, 2019 9:32 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment . The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet. Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan): controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2 If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active Cheers, Don. -----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Penney: Thanks your advice. My environment was broken, I will try and update once my setup get back normal. Thanks BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Friday, May 3, 2019 11:02 AM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Austin, I'm hoping to hear from some folks more familiar with this, but I noticed a couple of things from your logs. * controller-0 received DHCP request from controller-1 at 09:43 for MAC 52:54:00:3b:0f:84 and sent back an IP address on the pxeboot subnet * sysinv should add an entry to the hosts table at this point, as the node has been "discovered". At 09:53:25, you did a "system host-add", defining the host associated with MAC 52:54:00:3b:0f:84 as a controller, at which point the controller-1 (with its IP from DHCP on pxeboot network) TFTPed the bzImage and initrd. Controller-1 would have then loaded the kernel (bzImage) and initrd and reinitialized the network interface, sending a new DHCP request, which happened at 09:53:50. At this point, however, the DHCP response provided 192.168.204.4 as the IP address, which was then unable to reach the pxecontroller IP. My guess is that this is due to the host-add command, which I believe is intended for provisioning hosts BEFORE they're powered on and start network booting. If the node boots before this and is "discovered" by sysinv, you should instead be using the "system host-update" command to modify/configure the discovered host, which you'd see in the "system host-list" output after it is discovered. My suggestion would be to try this instead. You could shutdown controller-1 and reinstall controller-0 to start fresh, or maybe you can delete the host from sysinv (system host-delete) and try again. Let the host network boot to get discovered, then do the system host-update to configure its personality as "controller". -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Thursday, May 02, 2019 9:44 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penney: Thanks . The collect tar is attached to [1] The controller-0 is ok. controller-1 is stuck on pxe installation . [1] https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4zfKmu-IOqddLrH/view?usp=s... Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Thursday, May 2, 2019 9:32 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment . The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet. Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan): controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2 If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active Cheers, Don. -----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer. -----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
This may be a different problem, but I recently had second node launches hard hang at the pxeboot. This is using qemu on a Ubuntu 16.04.6 system, not virtualbox. And using an old iso that *used to work* fine. It looked like the broadcast packets weren't being forwarded to controller-0's tap interface by the bridge. There was a recent change to the Ubuntu kernel's bridging function to now drop packets instead of forwarding them (with truncation I presume) if the mtu of the connected receiver device (tap in our case) is smaller than the packet size. Given that those packets are driven out of the bios, I simply dropped the "-bios /usr/share/seabios/bios-256k.bin" option that I was using on qemu, and it started working again. -Jim On 2019-05-03 7:09 a.m., Sun, Austin wrote:
Hi Penney: Thanks your advice. My environment was broken, I will try and update once my setup get back normal.
Thanks BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Friday, May 3, 2019 11:02 AM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Austin,
I'm hoping to hear from some folks more familiar with this, but I noticed a couple of things from your logs. * controller-0 received DHCP request from controller-1 at 09:43 for MAC 52:54:00:3b:0f:84 and sent back an IP address on the pxeboot subnet * sysinv should add an entry to the hosts table at this point, as the node has been "discovered".
At 09:53:25, you did a "system host-add", defining the host associated with MAC 52:54:00:3b:0f:84 as a controller, at which point the controller-1 (with its IP from DHCP on pxeboot network) TFTPed the bzImage and initrd.
Controller-1 would have then loaded the kernel (bzImage) and initrd and reinitialized the network interface, sending a new DHCP request, which happened at 09:53:50. At this point, however, the DHCP response provided 192.168.204.4 as the IP address, which was then unable to reach the pxecontroller IP.
My guess is that this is due to the host-add command, which I believe is intended for provisioning hosts BEFORE they're powered on and start network booting. If the node boots before this and is "discovered" by sysinv, you should instead be using the "system host-update" command to modify/configure the discovered host, which you'd see in the "system host-list" output after it is discovered.
My suggestion would be to try this instead. You could shutdown controller-1 and reinstall controller-0 to start fresh, or maybe you can delete the host from sysinv (system host-delete) and try again. Let the host network boot to get discovered, then do the system host-update to configure its personality as "controller".
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Thursday, May 02, 2019 9:44 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Thanks . The collect tar is attached to [1]
The controller-0 is ok. controller-1 is stuck on pxe installation .
[1] https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4zfKmu-IOqddLrH/view?usp=s...
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Thursday, May 2, 2019 9:32 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment .
The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet.
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan):
controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2
If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active
Cheers, Don.
-----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer.
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed.
From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage).
If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully.
BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers.
[1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img.
Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console).
For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24
followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0"
If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs)
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi,
we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image.
Kind regards
Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Jim and Penny: the issue we observed is related host environment because we are using virtual (libvirt+qemu) env. once we changed mgmt ip subnet to 192.178.204.0/24, this issue is gone. Pxeboot and some broadcast packets were forwarded to other machine, not controller-0. Thanks. BR Austin Sun. -----Original Message----- From: Jim Somerville [mailto:jim.somerville@windriver.com] Sent: Thursday, May 9, 2019 6:15 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation This may be a different problem, but I recently had second node launches hard hang at the pxeboot. This is using qemu on a Ubuntu 16.04.6 system, not virtualbox. And using an old iso that *used to work* fine. It looked like the broadcast packets weren't being forwarded to controller-0's tap interface by the bridge. There was a recent change to the Ubuntu kernel's bridging function to now drop packets instead of forwarding them (with truncation I presume) if the mtu of the connected receiver device (tap in our case) is smaller than the packet size. Given that those packets are driven out of the bios, I simply dropped the "-bios /usr/share/seabios/bios-256k.bin" option that I was using on qemu, and it started working again. -Jim On 2019-05-03 7:09 a.m., Sun, Austin wrote:
Hi Penney: Thanks your advice. My environment was broken, I will try and update once my setup get back normal.
Thanks BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Friday, May 3, 2019 11:02 AM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Austin,
I'm hoping to hear from some folks more familiar with this, but I noticed a couple of things from your logs. * controller-0 received DHCP request from controller-1 at 09:43 for MAC 52:54:00:3b:0f:84 and sent back an IP address on the pxeboot subnet * sysinv should add an entry to the hosts table at this point, as the node has been "discovered".
At 09:53:25, you did a "system host-add", defining the host associated with MAC 52:54:00:3b:0f:84 as a controller, at which point the controller-1 (with its IP from DHCP on pxeboot network) TFTPed the bzImage and initrd.
Controller-1 would have then loaded the kernel (bzImage) and initrd and reinitialized the network interface, sending a new DHCP request, which happened at 09:53:50. At this point, however, the DHCP response provided 192.168.204.4 as the IP address, which was then unable to reach the pxecontroller IP.
My guess is that this is due to the host-add command, which I believe is intended for provisioning hosts BEFORE they're powered on and start network booting. If the node boots before this and is "discovered" by sysinv, you should instead be using the "system host-update" command to modify/configure the discovered host, which you'd see in the "system host-list" output after it is discovered.
My suggestion would be to try this instead. You could shutdown controller-1 and reinstall controller-0 to start fresh, or maybe you can delete the host from sysinv (system host-delete) and try again. Let the host network boot to get discovered, then do the system host-update to configure its personality as "controller".
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Thursday, May 02, 2019 9:44 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Thanks . The collect tar is attached to [1]
The controller-0 is ok. controller-1 is stuck on pxe installation .
[1] https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4zfKmu-IOqddLrH/view ?usp=sharing
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Thursday, May 2, 2019 9:32 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment .
The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet.
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan):
controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2
If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active
Cheers, Don.
-----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer.
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed.
From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage).
If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully.
BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers.
[1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img.
Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console).
For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24
followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0"
If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs)
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi,
we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image.
Kind regards
Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install | install_output | text | | install_state | None | | install_state_info | None | | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Thu, 2019-05-02 at 13:31 +0000, Penney, Don wrote:
Running a "collect" and providing the tarball might help us to understand what's happening. Can you provide a "system host-if-list" for your controllers, at a minimum? And maybe the content of /opt/platform/config/19.01/cgcs_config
I had reproduced it on my machine. Here is the collect tarball: https://drive.google.com/file/d/1-2vavbkjYouvmhp4p4z fKmu-IOqddLrH/view?usp=sharing 8<------------------------------------------------------------------------------- - [wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+--------------+ | id | hostname | personality | administrative | operational | availability | +----+--------------+-------------+----------------+-------------+--------------+ | 1 | controller-0 | controller | unlocked | enabled | available | | 2 | None | None | locked | disabled | offline | | 3 | None | None | locked | disabled | offline | | 4 | controller-1 | controller | locked | disabled | offline | +----+--------------+-------------+----------------+-------------+--------------+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-if-list 1 +--------------------------------------+------+----------+----------+---------+ -----------+----------+-------------+------------+---------------+ | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks | +--------------------------------------+------+----------+----------+---------+ -----------+----------+-------------+------------+---------------+ | 53173371-999c-4724-8df0-550b9ed283e0 | ens3 | platform | ethernet | None | [u'ens3'] | [] | [] | MTU=1500 | [] | | 8bc280a4-c7fe-41f1-8771-8f951f62a053 | ens4 | platform | ethernet | None | [u'ens4'] | [] | [] | MTU=1500 | [] | +--------------------------------------+------+----------+----------+---------+ -----------+----------+-------------+------------+---------------+ cat /opt/platform/config/19.01/cgcs_co [cSYSTEM] # System Configuration SYSTEM_MODE=duplex TIMEZONE=UTC [cPXEBOOT] # PXEBoot Network Support Configuration PXECONTROLLER_FLOATING_HOSTNAME=pxecontroller [cMGMT] # Management Network Configuration MANAGEMENT_INTERFACE_NAME=ens4 MANAGEMENT_INTERFACE=ens4 MANAGEMENT_MTU=1500 MANAGEMENT_SUBNET=192.168.204.0/24 LAG_MANAGEMENT_INTERFACE=no CONTROLLER_FLOATING_ADDRESS=192.168.204.2 CONTROLLER_0_ADDRESS=192.168.204.3 CONTROLLER_1_ADDRESS=192.168.204.4 NFS_MANAGEMENT_ADDRESS_1=192.168.204.5 NFS_MANAGEMENT_ADDRESS_2=192.168.204.6 CONTROLLER_FLOATING_HOSTNAME=controller CONTROLLER_HOSTNAME_PREFIX=controller- OAMCONTROLLER_FLOATING_HOSTNAME=oamcontroller DYNAMIC_ADDRESS_ALLOCATION=yes MANAGEMENT_MULTICAST_SUBNET=239.1.1.0/28 [cINFRA] # Infrastructure Network Configuration INFRASTRUCTURE_INTERFACE_NAME=NC INFRASTRUCTURE_INTERFACE=NC INFRASTRUCTURE_VLAN=NC INFRASTRUCTURE_MTU=NC INFRASTRUCTURE_SUBNET=NC LAG_INFRASTRUCTURE_INTERFACE=no INFRASTRUCTURE_BOND_MEMBER_0=NC INFRASTRUCTURE_BOND_MEMBER_1=NC INFRASTRUCTURE_BOND_POLICY=NC CONTROLLER_0_INFRASTRUCTURE_ADDRESS=NC CONTROLLER_1_INFRASTRUCTURE_ADDRESS=NC NFS_INFRASTRUCTURE_ADDRESS_1=NC STORAGE_0_INFRASTRUCTURE_ADDRESS=NC STORAGE_1_INFRASTRUCTURE_ADDRESS=NC CONTROLLER_INFRASTRUCTURE_HOSTNAME_SUFFIX=NC INFRASTRUCTURE_START_ADDRESS=NC INFRASTRUCTURE_END_ADDRESS=NC [cCLUSTER] # Cluster Host Network Configuration CLUSTER_INTERFACE_NAME=ens4 CLUSTER_INTERFACE=ens4 CLUSTER_VLAN=NC CLUSTER_MTU=1500 CLUSTER_SUBNET=192.168.206.0/24 LAG_CLUSTER_INTERFACE=no [cEXT_OAM] # External OAM Network Configuration EXTERNAL_OAM_INTERFACE_NAME=ens3 EXTERNAL_OAM_INTERFACE=ens3 EXTERNAL_OAM_VLAN=NC EXTERNAL_OAM_MTU=1500 LAG_EXTERNAL_OAM_INTERFACE=no EXTERNAL_OAM_SUBNET=10.10.10.0/24 EXTERNAL_OAM_GATEWAY_ADDRESS=10.10.10.1 EXTERNAL_OAM_FLOATING_ADDRESS=10.10.10.2 EXTERNAL_OAM_0_ADDRESS=10.10.10.3 EXTERNAL_OAM_1_ADDRESS=10.10.10.4 [cDNS] # DNS Configuration NAMESERVER_1=10.248.2.1 NAMESERVER_2=NC NAMESERVER_3=NC [cDOCKER_REGISTRY] # Docker Registry Configuration DOCKER_K8S_REGISTRY=edgehost01.sh.intel.com:5000 DOCKER_GCR_REGISTRY=edgehost01.sh.intel.com:5000 DOCKER_QUAY_REGISTRY=edgehost01.sh.intel.com:5000 DOCKER_DOCKER_REGISTRY=edgehost01.sh.intel.com:5000 IS_SECURE_REGISTRY=False [cSECURITY] [cREGION] # Region Configuration REGION_CONFIG=False [wrsroot@controller-0 ~(keystone_admin)]$
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Wednesday, May 01, 2019 10:08 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penney: Yes. pxecontrooler IP is active on controller-0 management interface . The environment is virtual environment .
The issue is that in controller-1 , the ip has been assigned to mgmt_ip if it is already failed. From my understanding , the ip in controller-1 when install. The ip should be in range pxe subnet.
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Wednesday, May 1, 2019 10:11 PM To: Sun, Austin <austin.sun@intel.com>; Marcel Schaible <marcel@schaible-consul ting.de>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Is the pxecontroller IP active on your management interface? For example (if your management isn't on a vlan):
controller-0:/home/wrsroot# grep management_interface /etc/platform/platform.conf management_interface=eno2 controller-0:/home/wrsroot# grep pxecontroller /etc/hosts 169.254.202.2 pxecontroller controller-0:/home/wrsroot# ip addr show eno2 | grep 169.254.202.2 inet 169.254.202.2/24 scope global eno2
If not, is the pxeboot-ip service in SM active? controller-0:/home/wrsroot# sm-dump |grep pxe pxeboot-ip enabled-active enabled-active
Cheers, Don.
-----Original Message----- From: Penney, Don Sent: Wednesday, May 01, 2019 9:54 AM To: 'Sun, Austin'; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
The pxecontroller address is purposefully on a separate subnet. Our networking team could provide a better answer, but I believe it was due to issues with pxeboot and VLANs. This address should absolutely be available from the installer.
-----Original Message----- From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Tuesday, April 30, 2019 10:59 PM To: Penney, Don; Marcel Schaible; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller- 1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/s quashfs.img" , it will failed.
From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage).
If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully.
BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers.
[1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/
Thanks. BR Austin Sun.
-----Original Message----- From: Penney, Don [mailto:Don.Penney@windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible <marcel@schaible-consulting.de>; starlingx-discuss@lists.st arlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img.
Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console).
For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24
followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0"
If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs)
-----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi,
we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image.
Kind regards
Marcel
Message: 1 Date: Tue, 30 Apr 2019 13:54:17 +0000 From: "Penney, Don" <Don.Penney@windriver.com> To: "Lin, Shuicheng" <shuicheng.lin@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Message-ID: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86@ALA-MBD.corp.ad.wrs.com> Content-Type: text/plain; charset="utf-8"
You can set the console either via the web browser in the installation parameters, or at command-line, such as: system host-update 2 personality=controller console=
The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer.
From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com] Sent: Tuesday, April 30, 2019 1:03 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] 2nd node stuck at pxe installation
Hi all, I try to do duplex deploy, and find controller-1 stuck at pxe installation step. The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller- 0. I meet this issue with latest code. Do you know what cause the issue, or how to debug it? Thanks.
[wrsroot@controller-0 ~(keystone_admin)]$ system host-list +----+--------------+-------------+----------------+-------------+----------- ---+
id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+----------- ---+
1 | controller-0 | controller | unlocked | enabled | available | 2 | controller-1 | controller | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+----------- ---+ [wrsroot@controller-0 ~(keystone_admin)]$ system host-show controller-1 | grep install
install_output | text | install_state | None | install_state_info | None | subfunction_avail | not-installed
Best Regards Shuicheng
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (5)
-
Jim Somerville
-
Marcel Schaible
-
Penney, Don
-
Sun, Austin
-
Yang, Bin