[Starlingx-discuss] Compute boot-loop
Hello, there, i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. Error messages are not really noticeable, except: compute-X 'ceilometer-polling' process has failed. Auto recovery in progress any suggestions? Greez & thx, volker...
Maybe it helps: https://ibb.co/n2OgoL ---------------------------------------------------------------- Hello, there, i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. Error messages are not really noticeable, except: compute-X 'ceilometer-polling' process has failed. Auto recovery in progress any suggestions? Greez & thx, volker...
Hi Volker, Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes? How many cpus do your computes have? Which build are you using? Did you build your own? When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/ M On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote:
Hello, there, i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. Error messages are not really noticeable, except: compute-X 'ceilometer-polling' process has failed. Auto recovery in progress
any suggestions? Greez & thx, volker...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi, I think you set up a Duplex system (two controllers), right? I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso. Please set up again a Duplex system with today's ISO to see if issue is still presented. Regards. Juan Carlos Alonso -----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop Hi Volker, Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes? How many cpus do your computes have? Which build are you using? Did you build your own? When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/ M On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote:
Hello, there, i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. Error messages are not really noticeable, except: compute-X 'ceilometer-polling' process has failed. Auto recovery in progress
any suggestions? Greez & thx, volker...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Another head-up is that we need to assure VT is turned on in BIOS for the compute node. On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote: Hi, I think you set up a Duplex system (two controllers), right? I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso. Please set up again a Duplex system with today's ISO to see if issue is still presented. Regards. Juan Carlos Alonso -----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop Hi Volker, Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes? How many cpus do your computes have? Which build are you using? Did you build your own? When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/ M On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Ack Yong. I received this initial internal response: "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts." Volker, please let us know when you've confirmed that. M On 2018-11-01 9:19 p.m., Hu, Yong wrote:
Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote:
Hi,
I think you set up a Duplex system (two controllers), right?
I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
Please set up again a Duplex system with today's ISO to see if issue is still presented.
Regards. Juan Carlos Alonso
-----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Ok, shame on me! I have to read all the hints from you and realy check all the details... I have an AMD EPYC 7601 (32 cores), and I didn’t realized that the point "IOMMU" in my BIOS the generic term for VT-d is, so it is all the time leave disabled -.- After enabled this option, my VM seems to run, currently booted but still in "In-Test" mode... we will see... Thx for help! Volker... -----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Montag, 5. November 2018 16:34 An: Hu, Yong; Alonso, Juan Carlos; von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ack Yong. I received this initial internal response: "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts." Volker, please let us know when you've confirmed that. M On 2018-11-01 9:19 p.m., Hu, Yong wrote:
Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote:
Hi,
I think you set up a Duplex system (two controllers), right?
I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
Please set up again a Duplex system with today's ISO to see if issue is still presented.
Regards. Juan Carlos Alonso
-----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
build instructions: https://docs.starlingx.io/developer_guide/ (2018-11-02) install instruction: https://docs.starlingx.io/installation_guide/dedicated_storage.html#dedicate... the installation runs without problems, only at the very end the unlocking of the compute-nodes ends in said endless loop. here are the last ~500kb of puppet.log from the compute-node: https://pastebin.com/wsSUY741 it seems that there are problems with the network interface eno1. the output of "ip a" only shows eno2, eno1 is not started. did i understand correctly that the "data" interface at the compute-nodes is the interface via which the "provider-networks" can be reached (via VLAN) ? i refer to this line from the doku: system host-if-modify -p providernet-a -c data compute-0 eno1 here is an overview of my network, basically very primitive and i think also in the sense of starlingX/openstack quite classic. https://docdro.id/0CRp2OA i am grateful for any help! Volker... -----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Donnerstag, 1. November 2018 17:29 An: von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Hi Volker, Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes? How many cpus do your computes have? Which build are you using? Did you build your own? When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/ M On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote:
Hello, there, i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. Error messages are not really noticeable, except: compute-X 'ceilometer-polling' process has failed. Auto recovery in progress
any suggestions? Greez & thx, volker...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Hi Volker, The puppet log is pointing at vswitch. I'll ask one of the networking guys to look at this thread. 2018-11-05T11:16:26.170 [mNotice: 2018-11-05 11:16:26 +0000 /Stage[main]/Platform::Vswitch::Ovs/Platform::Vswitch::Ovs::Device[0000:01:00.0]/Exec[ovs-bind-device: 0000:01:00.0]/returns: Error: bind failed for 0000:01:00.0 - Cannot bind to driver vfio-pci[0m 2018-11-05T11:16:26.173 [mNotice: 2018-11-05 11:16:26 +0000 /Stage[main]/Platform::Vswitch::Ovs/Platform::Vswitch::Ovs::Device[0000:01:00.0]/Exec[ovs-bind-device: 0000:01:00.0]/returns: Error: unbind failed for 0000:01:00.0 - Cannot open /sys/bus/pci/drivers//unbind[0m 2018-11-05T11:16:26.175 [1;31mError: 2018-11-05 11:16:26 +0000 dpdk-devbind.py --bind=vfio-pci 0000:01:00.0 returned 1 instead of one of [0] === PS. It is normal for ifconfig and ip commands to not show the interfaces configured for platform data. M On 2018-11-05 8:56 a.m., von Hoesslin, Volker wrote:
build instructions: https://docs.starlingx.io/developer_guide/ (2018-11-02) install instruction: https://docs.starlingx.io/installation_guide/dedicated_storage.html#dedicate...
the installation runs without problems, only at the very end the unlocking of the compute-nodes ends in said endless loop. here are the last ~500kb of puppet.log from the compute-node:
it seems that there are problems with the network interface eno1. the output of "ip a" only shows eno2, eno1 is not started. did i understand correctly that the "data" interface at the compute-nodes is the interface via which the "provider-networks" can be reached (via VLAN) ? i refer to this line from the doku:
system host-if-modify -p providernet-a -c data compute-0 eno1
here is an overview of my network, basically very primitive and i think also in the sense of starlingX/openstack quite classic.
i am grateful for any help!
Volker...
-----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Donnerstag, 1. November 2018 17:29 An: von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote:
Hello, there, i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. Error messages are not really noticeable, except: compute-X 'ceilometer-polling' process has failed. Auto recovery in progress
any suggestions? Greez & thx, volker...
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (4)
-
Alonso, Juan Carlos
-
Hu, Yong
-
Michel Thebeau
-
von Hoesslin, Volker