[Starlingx-discuss] Compute boot-loop

von Hoesslin, Volker Volker.Hoesslin at swsn.de
Mon Nov 12 10:55:40 UTC 2018


unbelievable! after I tortured the server with several restarts it seems to run:

Used hardware:
- Mainboard: H11DSi-NT
- CPU: 2x AMD Epyc 7601
- RAM: 256GB (ECC 2666MHz)

Important BIOS settings:
- IOMMU: Enabeld // for Neutron
- SR-IOV: Enabled
- Memory interleaving: Die // This equals NUMA-Count and sockets (in my case to 2/2)

Many thanks for the really good and very target-oriented support! I've spent a lot of time setting up an OpenStack, but this project trumps everything so far! Go on! I will certainly get back to you one or two times but so far it's a lot of fun!

greetings, volker

-----Ursprüngliche Nachricht-----
Von: Gauld, James [mailto:James.Gauld at windriver.com] 
Gesendet: Freitag, 9. November 2018 20:10
An: Thebeau, Michel; von Hoesslin, Volker; 'starlingx-discuss at lists.starlingx.io'
Betreff: RE: [Starlingx-discuss] Compute boot-loop

I took a brief look at the pastebin nova-compute.log and noticed you have 8 numa nodes and 2 sockets.

See section :
2018-11-09 14:42:10.928 69097 INFO nova.virt.libvirt.host [-] Libvirt host capabilities <capabilities>
i.e., see Libvirt host <capabilities><host><topology> -- there are 8 NUMA Cells, but only 2 socket_ids.

Our sysinv and platform code does not support numa nodes per socket, we require a 1-to-1 mapping of socket to NUMA node.

The specific error in nova/virt/libvirt/host.py", line 743, in get_capabilities() occurs since platform did not provision all 8 numa nodes (Cells).

i.e.,
caps = self._host.get_capabilities()
2018-11-09 14:42:10.936 69097 ERROR oslo_service.service     vm_4K = vm_4K_nodes[cell.id]
2018-11-09 14:42:10.936 69097 ERROR oslo_service.service IndexError: list index out of range

The underlying routine _get_configured_pages() replies on platform configuration knowledge of sockets/numa cells.  The platform likely provided 2 sockets of configuration data, but this code required 8 numa Cell indices.

We have seen cases like this where a BIOS for setting like 'enable Sub NUMA Cluster configuration' is enabled. If that is the case, disable that setting, then reinstall the compute.
-jim

-----Original Message-----
From: Thebeau, Michel 
Sent: November-09-18 11:26 AM
To: von Hoesslin, Volker; 'starlingx-discuss at lists.starlingx.io'
Subject: RE: [Starlingx-discuss] Compute boot-loop

That's a lot to take in Volker ;)

*If* this if the first time you've worked with StarlingX I recommend trying it out under the virtualization usecase. That would be a good way to become familiar with the install process.

The actual error listed in pastebin is interesting, referring to "vm_4K = vm_4K_nodes[cell.id]" in nova/virt/libvirt/host.py.  I'll ask internally if that error looks familiar.

M


________________________________________
From: von Hoesslin, Volker [Volker.Hoesslin at swsn.de]
Sent: Friday, November 09, 2018 10:00 AM
To: 'starlingx-discuss at lists.starlingx.io'
Subject: Re: [Starlingx-discuss] Compute boot-loop

...make long story short:
- activate IOMMU for an already installed compute-node will end in some other crashes
- try to reinstall a compute-node ends in a realy desaster -> server briks completely -> have to flash BIOS
- now my 9341-4i MegaRaid controller is not regonized by anyway :( nevermind, for testing just use the on-board sata controller
- install compute-node run like a charm
- unlock the compute-node -> auto-restart -> testing -> CRASH!

Now I got this nova error log: https://pastebin.com/fvA7yKmT (loops for some times -> reboot with auto-revover try ...)

So I'm so short befor getting a running starlingX cluster, but someone/something don’t want this :(

PLZ HELP :) greez, volker...

-----Ursprüngliche Nachricht-----
Von: von Hoesslin, Volker [mailto:Volker.Hoesslin at swsn.de]
Gesendet: Montag, 5. November 2018 17:06
An: 'Michel Thebeau'; Hu, Yong; Alonso, Juan Carlos; 'starlingx-discuss at lists.starlingx.io'
Betreff: Re: [Starlingx-discuss] Compute boot-loop

Ok, shame on me!
I have to read all the hints from you and realy check all the details... I have an AMD EPYC 7601 (32 cores), and I didn’t realized that the point "IOMMU" in my BIOS the generic term for VT-d is, so it is all the time leave disabled -.-

After enabled this option, my VM seems to run, currently booted but still in "In-Test" mode... we will see...

Thx for help! Volker...

-----Ursprüngliche Nachricht-----
Von: Michel Thebeau [mailto:michel.thebeau at windriver.com]
Gesendet: Montag, 5. November 2018 16:34
An: Hu, Yong; Alonso, Juan Carlos; von Hoesslin, Volker; 'starlingx-discuss at lists.starlingx.io'
Betreff: Re: [Starlingx-discuss] Compute boot-loop

Ack Yong.

I received this initial internal response:  "The usual cause of this error is when they don't enable VT-d (Intel Virtualization Technology for Directed I/O) in the BIOS for the compute hosts."

Volker, please let us know when you've confirmed that.

M



On 2018-11-01 9:19 p.m., Hu, Yong wrote:
> Another head-up is that we need to assure VT is turned on in BIOS for the compute node.
>
>
> On 02/11/2018, 6:39 AM, "Alonso, Juan Carlos" <juan.carlos.alonso at intel.com> wrote:
>
>      Hi,
>
>      I think you set up a Duplex system (two controllers), right?
>
>      I set up a Duplex system with two ISOs: stx-2018-10-30-171-master.iso and stx-2018-11-01-8-master.iso
>      I faced the same issue you mention with first ISO, but did not with today's ISO: stx-2018-11-01-8-master.iso.
>
>      Please set up again a Duplex system with today's ISO to see if issue is still presented.
>
>      Regards.
>      Juan Carlos Alonso
>
>      -----Original Message-----
>      From: Michel Thebeau [mailto:michel.thebeau at windriver.com]
>      Sent: Thursday, November 1, 2018 10:29 AM
>      To: von Hoesslin, Volker <Volker.Hoesslin at swsn.de>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
>      Subject: Re: [Starlingx-discuss] Compute boot-loop
>
>      Hi Volker,
>
>      Could you describe the instructions you were using to setup startlingx (were you following a wiki online)?  Did you recheck the instructions versus what is actually configured on the computes?
>
>      How many cpus do your computes have?
>
>      Which build are you using?  Did you build your own?
>
>      When reboot-loop happens on a node often the error will be 
> presented on the console during boot or could be inferred from 
> console, sometimes referring to puppet logs of the node 
> /var/log/puppet/
>
>      M
>
>
>      On 2018-11-01 6:45 a.m., von Hoesslin, Volker wrote:
>      > Hello, there,
>      > i have now built a complete stack and believe i did everything right. after unlocking the compute nodes (two pieces), the nodes come into an endless boot loop.
>      > Error messages are not really noticeable, except:
>      > compute-X 'ceilometer-polling' process has failed. Auto recovery in progress
>      >
>      > any suggestions? Greez & thx, volker...
>      >
>      >
>      > _______________________________________________
>      > Starlingx-discuss mailing list
>      > Starlingx-discuss at lists.starlingx.io
>      > 
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
>
>      _______________________________________________
>      Starlingx-discuss mailing list
>      Starlingx-discuss at lists.starlingx.io
>      http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>      _______________________________________________
>      Starlingx-discuss mailing list
>      Starlingx-discuss at lists.starlingx.io
>      
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
>

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss


More information about the Starlingx-discuss mailing list