[Starlingx-discuss] bnx2x driver panic. Is source same as CentOS 7?
Hi, I see a bnx2x driver panic during install of controller-1. Where do I find the driver source for the same to confirm whether this is stock from CentOS 7 or are there any changes? If it is the same as CentOS, then I presume there is no point raising a bug in StarlingX LP? I have installed CentOS7 on the same hardware (albeit not via pxe from the same broadcom "custer mgmt" NIC). I see a lot of these: bnx2x: [bnx2x_write_dmae:549(eno51)]DMAE returned failure -1 NIC is HP 536FLB which is QLogic 57840S Thanks in advance Pratik
On 8/3/20 3:03 AM, Pratik M. wrote:
Hi, I see a bnx2x driver panic during install of controller-1. Where do I find the driver source for the same to confirm whether this is stock from CentOS 7 or are there any changes? If it is the same as CentOS, then I presume there is no point raising a bug in StarlingX LP?
Since you did not say if you are using the StarlingX 3.0 or master ISO, I wil try to answer for both! For 3.0, StarlingX uses the CentOS7 3.10 kernel with some additional patches and configuration changes. For 4.0 (soon to be released) and master, StarlingX uses the CentOS8 4.18 kernel with a smaller number of patches and configuration changes. Can you try installing from the master ISO to see if the 4.18 kernel addresses these panics? Does this only happen on the machine when installed via PXE? Does it happen if you install the ISO directly? This could be related to the initrd setup possibly then.
I have installed CentOS7 on the same hardware (albeit not via pxe from the same broadcom "custer mgmt" NIC).
Just to confirm, you have installed a generic CentOS7 on this hardware and it works, but not via PXE boot, can you try to install the generic CentOS7 via PXE boot, this might rule out some of the StarlingX configuration changes.
I see a lot of these: bnx2x: [bnx2x_write_dmae:549(eno51)]DMAE returned failure -1
In what setup are you seeing these message? StarlingX and Generic CentOS7 or just StarlingX?
NIC is HP 536FLB which is QLogic 57840S
Thanks in advance Pratik
Thanks Sau!
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
On Wed, Aug 5, 2020 at 8:56 PM Saul Wold <sgw@linux.intel.com> wrote: [...]
Since you did not say if you are using the StarlingX 3.0 or master ISO, I wil try to answer for both!
Thank you for the help. I was using R3.0.
Can you try installing from the master ISO to see if the 4.18 kernel addresses these panics?
Sure, I will try and update.
Does this only happen on the machine when installed via PXE? Does it happen if you install the ISO directly? This could be related to the initrd setup possibly then.
I see this on C1 (controller-1) when it is booting from C0. If I try a direct iso install on the same server, as C0, I do not see it. Meanwhile, I found this: https://gist.github.com/michalskalski/c4bde2b14d1247daa1f8f8c333d00de3, where a BIOS workaround was suggested https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04565693 When I applied that, the problem disappeared. And I was able to proceed w/ the install. Absurd thing is that I have installed CentOS 6/7 and Ubuntu on this host w/ intel_iommu=on. Maybe something to do w/ PXE boot or possibly their installer possibly not enabling intel_iommu=on. Confused.
Just to confirm, you have installed a generic CentOS7 on this hardware and it works, but not via PXE boot, can you try to install the generic CentOS7 via PXE boot, this might rule out some of the StarlingX configuration changes.
Difficult, since my PXE setup is on the network that I use for OAM, but for C1 needs to PXE boot from MGMT. I will try to see if I am able to tweak the C0 PXE settings to serve CentOS7.
I see a lot of these: bnx2x: [bnx2x_write_dmae:549(eno51)]DMAE returned failure -1
In what setup are you seeing these message? StarlingX and Generic CentOS7 or just StarlingX?
Only during the StarlingX C1 install, which then aborts. Hit this 3 times before the BIOS workaround. Thanks Pratik
On Fri, Aug 7, 2020 at 3:46 PM Pratik M. <pvmpublic@gmail.com> wrote:
On Wed, Aug 5, 2020 at 8:56 PM Saul Wold <sgw@linux.intel.com> wrote: [...]
Can you try installing from the master ISO to see if the 4.18 kernel addresses these panics?
Sure, I will try and update.
Hi, As an update, I tried using R4.0, but that didn't work. Is it because the initrd does not have the necessary firmware? https://bugs.launchpad.net/starlingx/+bug/1891876 Thanks
Firmware isn’t my area, but we don’t have bnx2x-bnxZ2x-e2-7.13.11.0.fw in our load at all. We are currently including linux-firmware-20190429-72.gitddde598.el7.noarch.rpm in our system: https://opendev.org/starlingx/tools/src/commit/e390e85bd5f079d827bfabb69208c... The closest it has to what you’re looking for is: /usr/lib/firmware/bnx2x/bnx2x-e1-7.13.11.0.fw /usr/lib/firmware/bnx2x/bnx2x-e1h-7.13.11.0.fw /usr/lib/firmware/bnx2x/bnx2x-e2-7.13.11.0.fw I see nothing with bnxZ2x. We don’t currently update the initrd or squashfs (installer rootfs) with the linux-firmware RPM, so we just have what’s in the stock installer images. The initrd only has a handful of firmware images, while the squashfs.img appears to have a slightly older version of the linux-firmware RPM content (missing the 7.13.11.0 images listed above). If the firmware you need is in a newer version of the linux-firmware RPM, we’d need to update the rpms_centos.lst file to move to the newer version, and also update the update-pxe-network-installer utility to update the linux-firmware RPM content in the initrd.img and squashfs.img images: https://opendev.org/starlingx/root/src/branch/master/build-tools/update-pxe-... Cheers, Don. From: Pratik M. <pvmpublic@gmail.com> Sent: Monday, August 17, 2020 6:44 AM To: Saul Wold <sgw@linux.intel.com> Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] bnx2x driver panic. Is source same as CentOS 7? On Fri, Aug 7, 2020 at 3:46 PM Pratik M. <pvmpublic@gmail.com<mailto:pvmpublic@gmail.com>> wrote:
On Wed, Aug 5, 2020 at 8:56 PM Saul Wold <sgw@linux.intel.com<mailto:sgw@linux.intel.com>> wrote: [...]
Can you try installing from the master ISO to see if the 4.18 kernel addresses these panics?
Sure, I will try and update.
Hi, As an update, I tried using R4.0, but that didn't work. Is it because the initrd does not have the necessary firmware? https://bugs.launchpad.net/starlingx/+bug/1891876 Thanks
Don: Thanks! I had a similar email queued up! Pratik: I have added some additional information. On 8/17/20 2:42 PM, Penney, Don wrote:
Firmware isn’t my area, but we don’t have bnx2x-bnxZ2x-e2-7.13.11.0.fw in our load at all. We are currently including linux-firmware-20190429-72.gitddde598.el7.noarch.rpm in our system:
https://opendev.org/starlingx/tools/src/commit/e390e85bd5f079d827bfabb69208c...
The closest it has to what you’re looking for is:
/usr/lib/firmware/bnx2x/bnx2x-e1-7.13.11.0.fw
/usr/lib/firmware/bnx2x/bnx2x-e1h-7.13.11.0.fw
/usr/lib/firmware/bnx2x/bnx2x-e2-7.13.11.0.fw
I think that the e2-7.13.11.0 will work.
I see nothing with bnxZ2x.
We don’t currently update the initrd or squashfs (installer rootfs) with the linux-firmware RPM, so we just have what’s in the stock installer images. The initrd only has a handful of firmware images, while the squashfs.img appears to have a slightly older version of the linux-firmware RPM content (missing the 7.13.11.0 images listed above).
Yeah, I was looking into this also, there is the 7.13.1.0 firmware
If the firmware you need is in a newer version of the linux-firmware RPM, we’d need to update the rpms_centos.lst file to move to the newer version, and also update the update-pxe-network-installer utility to update the linux-firmware RPM content in the initrd.img and squashfs.img images:
https://opendev.org/starlingx/root/src/branch/master/build-tools/update-pxe-...
A quick starting point might be to try adding the newer firmware load into the /pxeboot/rel-20.06/installer-intel-x86-64-initrd_1.0 image on Controller-0. If I remember your trying to use this for Controller-1. We would welcome a general change that could add specified modules/firmware to the initrd.img based on what Don suggested above. Sau!
Cheers,
Don.
*From:* Pratik M. <pvmpublic@gmail.com> *Sent:* Monday, August 17, 2020 6:44 AM *To:* Saul Wold <sgw@linux.intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] bnx2x driver panic. Is source same as CentOS 7?
On Fri, Aug 7, 2020 at 3:46 PM Pratik M. <pvmpublic@gmail.com <mailto:pvmpublic@gmail.com>> wrote:
On Wed, Aug 5, 2020 at 8:56 PM Saul Wold <sgw@linux.intel.com
<mailto:sgw@linux.intel.com>> wrote:
[...]
Can you try installing from the master ISO to see if the 4.18 kernel addresses these panics?
Sure, I will try and update.
Hi,
As an update, I tried using R4.0, but that didn't work. Is it because the initrd does not have the necessary firmware?
https://bugs.launchpad.net/starlingx/+bug/1891876
Thanks
Hi, Thank you Don and Saul for looking into this. Seems http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/linux-firmware-2... is the rpm to use. Rather than the linux-firmware-20190429-72.gitddde598.el7.noarch.rpm, used currently. I will try to submit a diff, but not sure if I understood the update-pxe-network-installer script entirely. BR Pratik On Tue, Aug 18, 2020 at 3:49 AM Saul Wold <sgw@linux.intel.com> wrote:
Don: Thanks! I had a similar email queued up!
Pratik: I have added some additional information.
On 8/17/20 2:42 PM, Penney, Don wrote:
Firmware isn’t my area, but we don’t have bnx2x-bnxZ2x-e2-7.13.11.0.fw in our load at all. We are currently including linux-firmware-20190429-72.gitddde598.el7.noarch.rpm in our system:
https://opendev.org/starlingx/tools/src/commit/e390e85bd5f079d827bfabb69208c...
The closest it has to what you’re looking for is:
/usr/lib/firmware/bnx2x/bnx2x-e1-7.13.11.0.fw
/usr/lib/firmware/bnx2x/bnx2x-e1h-7.13.11.0.fw
/usr/lib/firmware/bnx2x/bnx2x-e2-7.13.11.0.fw
I think that the e2-7.13.11.0 will work.
I see nothing with bnxZ2x.
We don’t currently update the initrd or squashfs (installer rootfs) with the linux-firmware RPM, so we just have what’s in the stock installer images. The initrd only has a handful of firmware images, while the squashfs.img appears to have a slightly older version of the linux-firmware RPM content (missing the 7.13.11.0 images listed above).
Yeah, I was looking into this also, there is the 7.13.1.0 firmware
If the firmware you need is in a newer version of the linux-firmware RPM, we’d need to update the rpms_centos.lst file to move to the newer version, and also update the update-pxe-network-installer utility to update the linux-firmware RPM content in the initrd.img and squashfs.img images:
https://opendev.org/starlingx/root/src/branch/master/build-tools/update-pxe-...
A quick starting point might be to try adding the newer firmware load into the /pxeboot/rel-20.06/installer-intel-x86-64-initrd_1.0 image on Controller-0. If I remember your trying to use this for Controller-1.
We would welcome a general change that could add specified modules/firmware to the initrd.img based on what Don suggested above.
Sau!
Cheers,
Don.
*From:* Pratik M. <pvmpublic@gmail.com> *Sent:* Monday, August 17, 2020 6:44 AM *To:* Saul Wold <sgw@linux.intel.com> *Cc:* starlingx-discuss@lists.starlingx.io *Subject:* Re: [Starlingx-discuss] bnx2x driver panic. Is source same as CentOS 7?
On Fri, Aug 7, 2020 at 3:46 PM Pratik M. <pvmpublic@gmail.com <mailto:pvmpublic@gmail.com>> wrote:
On Wed, Aug 5, 2020 at 8:56 PM Saul Wold <sgw@linux.intel.com
<mailto:sgw@linux.intel.com>> wrote:
[...]
Can you try installing from the master ISO to see if the 4.18 kernel addresses these panics?
Sure, I will try and update.
Hi,
As an update, I tried using R4.0, but that didn't work. Is it because the initrd does not have the necessary firmware?
https://bugs.launchpad.net/starlingx/+bug/1891876
Thanks
participants (3)
-
Penney, Don
-
Pratik M.
-
Saul Wold