[Starlingx-discuss] Duplex: Controller-1 Kernel Panic
[Hardware: ArteSyn MaxCore, 64GB RAM, 2 CPU Boards] Hi, after setting the personality of our controller-1 we get the follwoing kernel panic: … [ 19.876920] iTCO_vendor_support: vendor-support=0 [ 19.878838] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 19.878895] iTCO_wdt: I/O address 0x0460 already in use, device disabled [ 19.878902] iTCO_wdt: probe of iTCO_wdt.0.auto failed with error -16 [ 19.928431] nvme nvme0: missing or invalid SUBNQN field. [ 20.003872] nvme nvme1: missing or invalid SUBNQN field. [ 20.132408] ata1: SATA link down (SStatus 0 SControl 300) [ 20.138460] ata3: SATA link down (SStatus 0 SControl 300) [ 20.144507] ata5: SATA link down (SStatus 0 SControl 300) [ 20.150552] ata4: SATA link down (SStatus 0 SControl 300) [ 20.156587] ata6: SATA link down (SStatus 0 SControl 300) [ 20.162632] ata2: SATA link down (SStatus 0 SControl 300) [ 20.808356] scsi 0:0:0:0: Direct-Access Generic Ultra HS-COMBO 1.98 PQ: 0 ANSI: 0 [ 20.831406] sd 0:0:0:0: [sda] Attached SCSI removable disk [ 20.839066] random: fast init done [ 80.503675] nvme nvme1: I/O 22 QID 0 timeout, disable controller [ 80.511636] nvme nvme0: I/O 0 QID 0 timeout, disable controller [ 80.610660] ------------[ cut here ]------------ [ 80.615824] WARNING: CPU: 3 PID: 4793 at kernel/irq/manage.c:1355 __free_irq+0xb3/0x250 [ 80.618638] ------------[ cut here ]------------ [ 80.618642] WARNING: CPU: 1 PID: 4792 at kernel/irq/manage.c:1355 __free_irq+0xb3/0x250 [ 80.618643] Trying to free already-free IRQ 143 [ 80.618673] Modules linked in: sd_mod crc_t10dif crct10dif_generic intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel uas kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel usb_storage aesni_intel glue_helper ablk_helper cryptd wdat_wdt ahci lpc_ich i2c_i801 libahci nvme nvme_core acpi_power_meter 8021q garp mrp stp llc sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs tpm_tis(O) tpm_tis_core(O) tpm(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_en(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) ixgbe(O) dca i40e(O) e1000e(O) ip_tables [ 80.618683] CPU: 1 PID: 4792 Comm: kworker/1:1H Tainted: G W OE ------------ 3.10.0-957.1.3.el7.1.tis.x86_64 #1 [ 80.618684] Hardware name: ARTESYN PCIE-7410/PCIE-7410, BIOS 1.7.3 5-Jun-2018 10:49 [ 80.618689] Workqueue: kblockd blk_mq_timeout_work [ 80.618690] Call Trace: [ 80.618695] [<ffffffff9a207861>] dump_stack+0x19/0x1b [ 80.618700] [<ffffffff99a7b4e8>] __warn+0xd8/0x100 [ 80.618702] [<ffffffff99a7b56f>] warn_slowpath_fmt+0x5f/0x80 [ 80.618703] [<ffffffff99b21e43>] __free_irq+0xb3/0x250 [ 80.618704] [<ffffffff99b22069>] free_irq+0x39/0x90 [ 80.618708] [<ffffffffc11db3e3>] nvme_dev_disable+0x113/0x4a0 [nvme] [ 80.618712] [<ffffffff99e49acc>] ? dev_warn+0x6c/0x90 [ 80.618714] [<ffffffffc11dba94>] nvme_timeout+0x204/0x2d0 [nvme] [ 80.618719] [<ffffffff99ad5ca1>] ? cpuacct_charge+0x61/0x70 [ 80.618720] [<ffffffff99ac444c>] ? update_curr+0x14c/0x210 [ 80.618722] [<ffffffff99d18d12>] blk_mq_rq_timed_out+0x32/0x80 [ 80.618723] [<ffffffff99d18dbc>] blk_mq_check_expired+0x5c/0x60 [ 80.618725] [<ffffffff99d1c2c4>] bt_iter+0x54/0x60 [ 80.618726] [<ffffffff99d1cc6b>] blk_mq_queue_tag_busy_iter+0x11b/0x290 [ 80.618727] [<ffffffff99d18d60>] ? blk_mq_rq_timed_out+0x80/0x80 [ 80.618728] [<ffffffff99d18d60>] ? blk_mq_rq_timed_out+0x80/0x80 [ 80.618732] [<ffffffff99ab3500>] ? __sched_fork+0x250/0x260 [ 80.618733] [<ffffffff99d1760b>] blk_mq_timeout_work+0xbb/0x1c0 [ 80.618738] [<ffffffff99a9e686>] process_one_work+0x176/0x4a0 [ 80.618739] [<ffffffff99a9f3d6>] worker_thread+0x126/0x3b0 [ 80.618741] [<ffffffff99a9f2b0>] ? manage_workers.isra.28+0x2a0/0x2a0 [ 80.618743] [<ffffffff99aa5ee1>] kthread+0xd1/0xe0 [ 80.618744] [<ffffffff99aa5e10>] ? kthread_create_on_node+0x140/0x140 [ 80.618748] [<ffffffff9a216e1d>] ret_from_fork_nospec_begin+0x7/0x21 [ 80.618750] [<ffffffff99aa5e10>] ? kthread_create_on_node+0x140/0x140 [ 80.618750] ---[ end trace 29827fc8c242fa85 ]--- [ 80.618756] BUG: unable to handle kernel NULL pointer dereference at 0000000000000048 [ 80.618757] IP: [<ffffffff99b22069>] free_irq+0x39/0x90 [ 80.618758] PGD 0 [ 80.618759] Oops: 0000 [#1] PREEMPT SMP [ 80.618774] Modules linked in: sd_mod crc_t10dif crct10dif_generic intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel uas kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel usb_storage aesni_intel glue_helper ablk_helper cryptd wdat_wdt ahci lpc_ich i2c_i801 libahci nvme nvme_core acpi_power_meter 8021q garp mrp stp llc sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs tpm_tis(O) tpm_tis_core(O) tpm(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_en(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) ixgbe(O) dca i40e(O) e1000e(O) ip_tables [ 80.618780] CPU: 1 PID: 4792 Comm: kworker/1:1H Tainted: G W OE ------------ 3.10.0-957.1.3.el7.1.tis.x86_64 #1 [ 80.618780] Hardware name: ARTESYN PCIE-7410/PCIE-7410, BIOS 1.7.3 5-Jun-2018 10:49 [ 80.618782] Workqueue: kblockd blk_mq_timeout_work … I suspect that the root cause is the error "nvme nvme0: missing or invalid SUBNQN field." nvme0n1 is our boot_device and rootfs_device. Asking Google I'll found something similar (https://unix.stackexchange.com/questions/470778/nvme-missing-or-invalid-subn...). What is the official way to change the kernel boot parameters for pxe boot? Thanks Marcel
Hi Marcel, We don't currently have a mechanism to allow a user to add arbitrary kernel parameters to the installation boot cmdline. In order to test to see if adding this parameter resolves your issue, you could modify the /pxeboot/pxelinux.cfg/ file that corresponds to your controller-1. If it works, we can look at either adding it to the default cmdline, or add support for specifying either this value or something more flexible. Cheers, Don. -----Original Message----- From: Marcel Schaible [mailto:marcel@schaible-consulting.de] Sent: Monday, April 29, 2019 1:49 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Duplex: Controller-1 Kernel Panic [Hardware: ArteSyn MaxCore, 64GB RAM, 2 CPU Boards] Hi, after setting the personality of our controller-1 we get the follwoing kernel panic: … [ 19.876920] iTCO_vendor_support: vendor-support=0 [ 19.878838] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 19.878895] iTCO_wdt: I/O address 0x0460 already in use, device disabled [ 19.878902] iTCO_wdt: probe of iTCO_wdt.0.auto failed with error -16 [ 19.928431] nvme nvme0: missing or invalid SUBNQN field. [ 20.003872] nvme nvme1: missing or invalid SUBNQN field. [ 20.132408] ata1: SATA link down (SStatus 0 SControl 300) [ 20.138460] ata3: SATA link down (SStatus 0 SControl 300) [ 20.144507] ata5: SATA link down (SStatus 0 SControl 300) [ 20.150552] ata4: SATA link down (SStatus 0 SControl 300) [ 20.156587] ata6: SATA link down (SStatus 0 SControl 300) [ 20.162632] ata2: SATA link down (SStatus 0 SControl 300) [ 20.808356] scsi 0:0:0:0: Direct-Access Generic Ultra HS-COMBO 1.98 PQ: 0 ANSI: 0 [ 20.831406] sd 0:0:0:0: [sda] Attached SCSI removable disk [ 20.839066] random: fast init done [ 80.503675] nvme nvme1: I/O 22 QID 0 timeout, disable controller [ 80.511636] nvme nvme0: I/O 0 QID 0 timeout, disable controller [ 80.610660] ------------[ cut here ]------------ [ 80.615824] WARNING: CPU: 3 PID: 4793 at kernel/irq/manage.c:1355 __free_irq+0xb3/0x250 [ 80.618638] ------------[ cut here ]------------ [ 80.618642] WARNING: CPU: 1 PID: 4792 at kernel/irq/manage.c:1355 __free_irq+0xb3/0x250 [ 80.618643] Trying to free already-free IRQ 143 [ 80.618673] Modules linked in: sd_mod crc_t10dif crct10dif_generic intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel uas kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel usb_storage aesni_intel glue_helper ablk_helper cryptd wdat_wdt ahci lpc_ich i2c_i801 libahci nvme nvme_core acpi_power_meter 8021q garp mrp stp llc sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs tpm_tis(O) tpm_tis_core(O) tpm(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_en(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) ixgbe(O) dca i40e(O) e1000e(O) ip_tables [ 80.618683] CPU: 1 PID: 4792 Comm: kworker/1:1H Tainted: G W OE ------------ 3.10.0-957.1.3.el7.1.tis.x86_64 #1 [ 80.618684] Hardware name: ARTESYN PCIE-7410/PCIE-7410, BIOS 1.7.3 5-Jun-2018 10:49 [ 80.618689] Workqueue: kblockd blk_mq_timeout_work [ 80.618690] Call Trace: [ 80.618695] [<ffffffff9a207861>] dump_stack+0x19/0x1b [ 80.618700] [<ffffffff99a7b4e8>] __warn+0xd8/0x100 [ 80.618702] [<ffffffff99a7b56f>] warn_slowpath_fmt+0x5f/0x80 [ 80.618703] [<ffffffff99b21e43>] __free_irq+0xb3/0x250 [ 80.618704] [<ffffffff99b22069>] free_irq+0x39/0x90 [ 80.618708] [<ffffffffc11db3e3>] nvme_dev_disable+0x113/0x4a0 [nvme] [ 80.618712] [<ffffffff99e49acc>] ? dev_warn+0x6c/0x90 [ 80.618714] [<ffffffffc11dba94>] nvme_timeout+0x204/0x2d0 [nvme] [ 80.618719] [<ffffffff99ad5ca1>] ? cpuacct_charge+0x61/0x70 [ 80.618720] [<ffffffff99ac444c>] ? update_curr+0x14c/0x210 [ 80.618722] [<ffffffff99d18d12>] blk_mq_rq_timed_out+0x32/0x80 [ 80.618723] [<ffffffff99d18dbc>] blk_mq_check_expired+0x5c/0x60 [ 80.618725] [<ffffffff99d1c2c4>] bt_iter+0x54/0x60 [ 80.618726] [<ffffffff99d1cc6b>] blk_mq_queue_tag_busy_iter+0x11b/0x290 [ 80.618727] [<ffffffff99d18d60>] ? blk_mq_rq_timed_out+0x80/0x80 [ 80.618728] [<ffffffff99d18d60>] ? blk_mq_rq_timed_out+0x80/0x80 [ 80.618732] [<ffffffff99ab3500>] ? __sched_fork+0x250/0x260 [ 80.618733] [<ffffffff99d1760b>] blk_mq_timeout_work+0xbb/0x1c0 [ 80.618738] [<ffffffff99a9e686>] process_one_work+0x176/0x4a0 [ 80.618739] [<ffffffff99a9f3d6>] worker_thread+0x126/0x3b0 [ 80.618741] [<ffffffff99a9f2b0>] ? manage_workers.isra.28+0x2a0/0x2a0 [ 80.618743] [<ffffffff99aa5ee1>] kthread+0xd1/0xe0 [ 80.618744] [<ffffffff99aa5e10>] ? kthread_create_on_node+0x140/0x140 [ 80.618748] [<ffffffff9a216e1d>] ret_from_fork_nospec_begin+0x7/0x21 [ 80.618750] [<ffffffff99aa5e10>] ? kthread_create_on_node+0x140/0x140 [ 80.618750] ---[ end trace 29827fc8c242fa85 ]--- [ 80.618756] BUG: unable to handle kernel NULL pointer dereference at 0000000000000048 [ 80.618757] IP: [<ffffffff99b22069>] free_irq+0x39/0x90 [ 80.618758] PGD 0 [ 80.618759] Oops: 0000 [#1] PREEMPT SMP [ 80.618774] Modules linked in: sd_mod crc_t10dif crct10dif_generic intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel uas kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel usb_storage aesni_intel glue_helper ablk_helper cryptd wdat_wdt ahci lpc_ich i2c_i801 libahci nvme nvme_core acpi_power_meter 8021q garp mrp stp llc sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs tpm_tis(O) tpm_tis_core(O) tpm(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_en(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) ixgbe(O) dca i40e(O) e1000e(O) ip_tables [ 80.618780] CPU: 1 PID: 4792 Comm: kworker/1:1H Tainted: G W OE ------------ 3.10.0-957.1.3.el7.1.tis.x86_64 #1 [ 80.618780] Hardware name: ARTESYN PCIE-7410/PCIE-7410, BIOS 1.7.3 5-Jun-2018 10:49 [ 80.618782] Workqueue: kblockd blk_mq_timeout_work … I suspect that the root cause is the error "nvme nvme0: missing or invalid SUBNQN field." nvme0n1 is our boot_device and rootfs_device. Asking Google I'll found something similar (https://unix.stackexchange.com/questions/470778/nvme-missing-or-invalid-subn...). What is the official way to change the kernel boot parameters for pxe boot? Thanks Marcel _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (2)
-
Marcel Schaible
-
Penney, Don