Re: [Starlingx-discuss] Compute boot-loop
...make long story short: - activate IOMMU for an already installed compute-node will end in some other crashes - try to reinstall a compute-node ends in a realy desaster -> server briks completely -> have to flash BIOS - now my 9341-4i MegaRaid controller is not regonized by anyway :( nevermind, for testing just use the on-board sata controller - install compute-node run like a charm - unlock the compute-node -> auto-restart -> testing -> CRASH! Now I got this nova error log: https://pastebin.com/fvA7yKmT (loops for some times -> reboot with auto-revover try ...) So I'm so short befor getting a running starlingX cluster, but someone/something don’t want this :( PLZ HELP :) greez, volker... -----Ursprüngliche Nachricht----- Von: von Hoesslin, Volker [mailto:Volker.Hoesslin@swsn.de] Gesendet: Montag, 5. November 2018 17:06 An: 'Michel Thebeau'; Hu, Yong; Alonso, Juan Carlos; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ok, shame on me! I have to read all the hints from you and realy check all the details... I have an AMD EPYC 7601 (32 cores), and I didn’t realized that the point "IOMMU" in my BIOS the generic term for VT-d is, so it is all the time leave disabled -.- After enabled this option, my VM seems to run, currently booted but still in "In-Test" mode... we will see... Thx for help! Volker... -----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Montag, 5. November 2018 16:34 An: Hu, Yong; Alonso, Juan Carlos; von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ack Yong. I received this initial internal response: "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts." Volker, please let us know when you've confirmed that. M On 2018-11-01 9:19 p.m., Hu, Yong wrote:
Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote:
Hi,
I think you set up a Duplex system (two controllers), right?
I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
Please set up again a Duplex system with today's ISO to see if issue is still presented.
Regards. Juan Carlos Alonso
-----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
That's a lot to take in Volker ;) *If* this if the first time you've worked with StarlingX I recommend trying it out under the virtualization usecase. That would be a good way to become familiar with the install process. The actual error listed in pastebin is interesting, referring to "vm_4K = vm_4K_nodes[cell.id]" in nova/virt/libvirt/host.py. I'll ask internally if that error looks familiar. M ________________________________________ From: von Hoesslin, Volker [Volker.Hoesslin@swsn.de] Sent: Friday, November 09, 2018 10:00 AM To: 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] Compute boot-loop ...make long story short: - activate IOMMU for an already installed compute-node will end in some other crashes - try to reinstall a compute-node ends in a realy desaster -> server briks completely -> have to flash BIOS - now my 9341-4i MegaRaid controller is not regonized by anyway :( nevermind, for testing just use the on-board sata controller - install compute-node run like a charm - unlock the compute-node -> auto-restart -> testing -> CRASH! Now I got this nova error log: https://pastebin.com/fvA7yKmT (loops for some times -> reboot with auto-revover try ...) So I'm so short befor getting a running starlingX cluster, but someone/something don’t want this :( PLZ HELP :) greez, volker... -----Ursprüngliche Nachricht----- Von: von Hoesslin, Volker [mailto:Volker.Hoesslin@swsn.de] Gesendet: Montag, 5. November 2018 17:06 An: 'Michel Thebeau'; Hu, Yong; Alonso, Juan Carlos; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ok, shame on me! I have to read all the hints from you and realy check all the details... I have an AMD EPYC 7601 (32 cores), and I didn’t realized that the point "IOMMU" in my BIOS the generic term for VT-d is, so it is all the time leave disabled -.- After enabled this option, my VM seems to run, currently booted but still in "In-Test" mode... we will see... Thx for help! Volker... -----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Montag, 5. November 2018 16:34 An: Hu, Yong; Alonso, Juan Carlos; von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ack Yong. I received this initial internal response: "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts." Volker, please let us know when you've confirmed that. M On 2018-11-01 9:19 p.m., Hu, Yong wrote:
Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote:
Hi,
I think you set up a Duplex system (two controllers), right?
I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
Please set up again a Duplex system with today's ISO to see if issue is still presented.
Regards. Juan Carlos Alonso
-----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
I took a brief look at the pastebin nova-compute.log and noticed you have 8 numa nodes and 2 sockets. See section : 2018-11-09 14:42:10.928 69097 INFO nova.virt.libvirt.host [-] Libvirt host capabilities <capabilities> i.e., see Libvirt host <capabilities><host><topology> -- there are 8 NUMA Cells, but only 2 socket_ids. Our sysinv and platform code does not support numa nodes per socket, we require a 1-to-1 mapping of socket to NUMA node. The specific error in nova/virt/libvirt/host.py", line 743, in get_capabilities() occurs since platform did not provision all 8 numa nodes (Cells). i.e., caps = self._host.get_capabilities() 2018-11-09 14:42:10.936 69097 ERROR oslo_service.service vm_4K = vm_4K_nodes[cell.id] 2018-11-09 14:42:10.936 69097 ERROR oslo_service.service IndexError: list index out of range The underlying routine _get_configured_pages() replies on platform configuration knowledge of sockets/numa cells. The platform likely provided 2 sockets of configuration data, but this code required 8 numa Cell indices. We have seen cases like this where a BIOS for setting like 'enable Sub NUMA Cluster configuration' is enabled. If that is the case, disable that setting, then reinstall the compute. -jim -----Original Message----- From: Thebeau, Michel Sent: November-09-18 11:26 AM To: von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] Compute boot-loop That's a lot to take in Volker ;) *If* this if the first time you've worked with StarlingX I recommend trying it out under the virtualization usecase. That would be a good way to become familiar with the install process. The actual error listed in pastebin is interesting, referring to "vm_4K = vm_4K_nodes[cell.id]" in nova/virt/libvirt/host.py. I'll ask internally if that error looks familiar. M ________________________________________ From: von Hoesslin, Volker [Volker.Hoesslin@swsn.de] Sent: Friday, November 09, 2018 10:00 AM To: 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] Compute boot-loop ...make long story short: - activate IOMMU for an already installed compute-node will end in some other crashes - try to reinstall a compute-node ends in a realy desaster -> server briks completely -> have to flash BIOS - now my 9341-4i MegaRaid controller is not regonized by anyway :( nevermind, for testing just use the on-board sata controller - install compute-node run like a charm - unlock the compute-node -> auto-restart -> testing -> CRASH! Now I got this nova error log: https://pastebin.com/fvA7yKmT (loops for some times -> reboot with auto-revover try ...) So I'm so short befor getting a running starlingX cluster, but someone/something don’t want this :( PLZ HELP :) greez, volker... -----Ursprüngliche Nachricht----- Von: von Hoesslin, Volker [mailto:Volker.Hoesslin@swsn.de] Gesendet: Montag, 5. November 2018 17:06 An: 'Michel Thebeau'; Hu, Yong; Alonso, Juan Carlos; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ok, shame on me! I have to read all the hints from you and realy check all the details... I have an AMD EPYC 7601 (32 cores), and I didn’t realized that the point "IOMMU" in my BIOS the generic term for VT-d is, so it is all the time leave disabled -.- After enabled this option, my VM seems to run, currently booted but still in "In-Test" mode... we will see... Thx for help! Volker... -----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Montag, 5. November 2018 16:34 An: Hu, Yong; Alonso, Juan Carlos; von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ack Yong. I received this initial internal response: "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts." Volker, please let us know when you've confirmed that. M On 2018-11-01 9:19 p.m., Hu, Yong wrote:
Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote:
Hi,
I think you set up a Duplex system (two controllers), right?
I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
Please set up again a Duplex system with today's ISO to see if issue is still presented.
Regards. Juan Carlos Alonso
-----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
unbelievable! after I tortured the server with several restarts it seems to run: Used hardware: - Mainboard: H11DSi-NT - CPU: 2x AMD Epyc 7601 - RAM: 256GB (ECC 2666MHz) Important BIOS settings: - IOMMU: Enabeld // for Neutron - SR-IOV: Enabled - Memory interleaving: Die // This equals NUMA-Count and sockets (in my case to 2/2) Many thanks for the really good and very target-oriented support! I've spent a lot of time setting up an OpenStack, but this project trumps everything so far! Go on! I will certainly get back to you one or two times but so far it's a lot of fun! greetings, volker -----Ursprüngliche Nachricht----- Von: Gauld, James [mailto:James.Gauld@windriver.com] Gesendet: Freitag, 9. November 2018 20:10 An: Thebeau, Michel; von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: RE: [Starlingx-discuss] Compute boot-loop I took a brief look at the pastebin nova-compute.log and noticed you have 8 numa nodes and 2 sockets. See section : 2018-11-09 14:42:10.928 69097 INFO nova.virt.libvirt.host [-] Libvirt host capabilities <capabilities> i.e., see Libvirt host <capabilities><host><topology> -- there are 8 NUMA Cells, but only 2 socket_ids. Our sysinv and platform code does not support numa nodes per socket, we require a 1-to-1 mapping of socket to NUMA node. The specific error in nova/virt/libvirt/host.py", line 743, in get_capabilities() occurs since platform did not provision all 8 numa nodes (Cells). i.e., caps = self._host.get_capabilities() 2018-11-09 14:42:10.936 69097 ERROR oslo_service.service vm_4K = vm_4K_nodes[cell.id] 2018-11-09 14:42:10.936 69097 ERROR oslo_service.service IndexError: list index out of range The underlying routine _get_configured_pages() replies on platform configuration knowledge of sockets/numa cells. The platform likely provided 2 sockets of configuration data, but this code required 8 numa Cell indices. We have seen cases like this where a BIOS for setting like 'enable Sub NUMA Cluster configuration' is enabled. If that is the case, disable that setting, then reinstall the compute. -jim -----Original Message----- From: Thebeau, Michel Sent: November-09-18 11:26 AM To: von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Subject: RE: [Starlingx-discuss] Compute boot-loop That's a lot to take in Volker ;) *If* this if the first time you've worked with StarlingX I recommend trying it out under the virtualization usecase. That would be a good way to become familiar with the install process. The actual error listed in pastebin is interesting, referring to "vm_4K = vm_4K_nodes[cell.id]" in nova/virt/libvirt/host.py. I'll ask internally if that error looks familiar. M ________________________________________ From: von Hoesslin, Volker [Volker.Hoesslin@swsn.de] Sent: Friday, November 09, 2018 10:00 AM To: 'starlingx-discuss@lists.starlingx.io' Subject: Re: [Starlingx-discuss] Compute boot-loop ...make long story short: - activate IOMMU for an already installed compute-node will end in some other crashes - try to reinstall a compute-node ends in a realy desaster -> server briks completely -> have to flash BIOS - now my 9341-4i MegaRaid controller is not regonized by anyway :( nevermind, for testing just use the on-board sata controller - install compute-node run like a charm - unlock the compute-node -> auto-restart -> testing -> CRASH! Now I got this nova error log: https://pastebin.com/fvA7yKmT (loops for some times -> reboot with auto-revover try ...) So I'm so short befor getting a running starlingX cluster, but someone/something don’t want this :( PLZ HELP :) greez, volker... -----Ursprüngliche Nachricht----- Von: von Hoesslin, Volker [mailto:Volker.Hoesslin@swsn.de] Gesendet: Montag, 5. November 2018 17:06 An: 'Michel Thebeau'; Hu, Yong; Alonso, Juan Carlos; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ok, shame on me! I have to read all the hints from you and realy check all the details... I have an AMD EPYC 7601 (32 cores), and I didn’t realized that the point "IOMMU" in my BIOS the generic term for VT-d is, so it is all the time leave disabled -.- After enabled this option, my VM seems to run, currently booted but still in "In-Test" mode... we will see... Thx for help! Volker... -----Ursprüngliche Nachricht----- Von: Michel Thebeau [mailto:michel.thebeau@windriver.com] Gesendet: Montag, 5. November 2018 16:34 An: Hu, Yong; Alonso, Juan Carlos; von Hoesslin, Volker; 'starlingx-discuss@lists.starlingx.io' Betreff: Re: [Starlingx-discuss] Compute boot-loop Ack Yong. I received this initial internal response: "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts." Volker, please let us know when you've confirmed that. M On 2018-11-01 9:19 p.m., Hu, Yong wrote:
Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso@intel.com> wrote:
Hi,
I think you set up a Duplex system (two controllers), right?
I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
Please set up again a Duplex system with today's ISO to see if issue is still presented.
Regards. Juan Carlos Alonso
-----Original Message----- From: Michel Thebeau [mailto:michel.thebeau@windriver.com] Sent: Thursday, November 1, 2018 10:29 AM To: von Hoesslin, Volker <Volker.Hoesslin@swsn.de>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Compute boot-loop
Hi Volker,
Could you describe the instructions you were using to setup startlingx (were you following a wiki online)? Did you recheck the instructions versus what is actually configured on the computes?
How many cpus do your computes have?
Which build are you using? Did you build your own?
When reboot-loop happens on a node often the error will be presented on the console during boot or could be inferred from console, sometimes referring to puppet logs of the node /var/log/puppet/
M
On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote: > Hello, there, > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop. > Error messages are not really noticeable, except: > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress > > any suggestions? Greez & thx, volker... > > > _______________________________________________ > Starlingx-discuss mailing list > Starlingx-discuss@lists.starlingx.io > http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
Gauld, James
-
Thebeau, Michel
-
von Hoesslin, Volker