[Starlingx-discuss] Duplex: Controller-1 Kernel Panic

Mon Apr 29 17:49:11 UTC 2019

[Hardware: ArteSyn MaxCore, 64GB RAM, 2 CPU Boards]

Hi,

after setting the personality of our controller-1 we get the follwoing kernel panic:

…
[   19.876920] iTCO_vendor_support: vendor-support=0
[   19.878838] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   19.878895] iTCO_wdt: I/O address 0x0460 already in use, device disabled
[   19.878902] iTCO_wdt: probe of iTCO_wdt.0.auto failed with error -16
[   19.928431] nvme nvme0: missing or invalid SUBNQN field.
[   20.003872] nvme nvme1: missing or invalid SUBNQN field.
[   20.132408] ata1: SATA link down (SStatus 0 SControl 300)
[   20.138460] ata3: SATA link down (SStatus 0 SControl 300)
[   20.144507] ata5: SATA link down (SStatus 0 SControl 300)
[   20.150552] ata4: SATA link down (SStatus 0 SControl 300)
[   20.156587] ata6: SATA link down (SStatus 0 SControl 300)
[   20.162632] ata2: SATA link down (SStatus 0 SControl 300)
[   20.808356] scsi 0:0:0:0: Direct-Access     Generic  Ultra HS-COMBO   1.98 PQ: 0 ANSI: 0
[   20.831406] sd 0:0:0:0: [sda] Attached SCSI removable disk
[   20.839066] random: fast init done

[   80.503675] nvme nvme1: I/O 22 QID 0 timeout, disable controller
[   80.511636] nvme nvme0: I/O 0 QID 0 timeout, disable controller
[   80.610660] ------------[ cut here ]------------
[   80.615824] WARNING: CPU: 3 PID: 4793 at kernel/irq/manage.c:1355 __free_irq+0xb3/0x250
[   80.618638] ------------[ cut here ]------------
[   80.618642] WARNING: CPU: 1 PID: 4792 at kernel/irq/manage.c:1355 __free_irq+0xb3/0x250
[   80.618643] Trying to free already-free IRQ 143
[   80.618673] Modules linked in: sd_mod crc_t10dif crct10dif_generic intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel uas kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel usb_storage aesni_intel glue_helper ablk_helper cryptd wdat_wdt ahci lpc_ich i2c_i801 libahci nvme nvme_core acpi_power_meter 8021q garp mrp stp llc sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs tpm_tis(O) tpm_tis_core(O) tpm(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_en(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) ixgbe(O) dca i40e(O) e1000e(O) ip_tables
[   80.618683] CPU: 1 PID: 4792 Comm: kworker/1:1H Tainted: G        W  OE  ------------   3.10.0-957.1.3.el7.1.tis.x86_64 #1
[   80.618684] Hardware name: ARTESYN PCIE-7410/PCIE-7410, BIOS 1.7.3 5-Jun-2018 10:49
[   80.618689] Workqueue: kblockd blk_mq_timeout_work
[   80.618690] Call Trace:
[   80.618695]  [<ffffffff9a207861>] dump_stack+0x19/0x1b
[   80.618700]  [<ffffffff99a7b4e8>] __warn+0xd8/0x100
[   80.618702]  [<ffffffff99a7b56f>] warn_slowpath_fmt+0x5f/0x80
[   80.618703]  [<ffffffff99b21e43>] __free_irq+0xb3/0x250
[   80.618704]  [<ffffffff99b22069>] free_irq+0x39/0x90
[   80.618708]  [<ffffffffc11db3e3>] nvme_dev_disable+0x113/0x4a0 [nvme]
[   80.618712]  [<ffffffff99e49acc>] ? dev_warn+0x6c/0x90
[   80.618714]  [<ffffffffc11dba94>] nvme_timeout+0x204/0x2d0 [nvme]
[   80.618719]  [<ffffffff99ad5ca1>] ? cpuacct_charge+0x61/0x70
[   80.618720]  [<ffffffff99ac444c>] ? update_curr+0x14c/0x210
[   80.618722]  [<ffffffff99d18d12>] blk_mq_rq_timed_out+0x32/0x80
[   80.618723]  [<ffffffff99d18dbc>] blk_mq_check_expired+0x5c/0x60
[   80.618725]  [<ffffffff99d1c2c4>] bt_iter+0x54/0x60
[   80.618726]  [<ffffffff99d1cc6b>] blk_mq_queue_tag_busy_iter+0x11b/0x290
[   80.618727]  [<ffffffff99d18d60>] ? blk_mq_rq_timed_out+0x80/0x80
[   80.618728]  [<ffffffff99d18d60>] ? blk_mq_rq_timed_out+0x80/0x80
[   80.618732]  [<ffffffff99ab3500>] ? __sched_fork+0x250/0x260
[   80.618733]  [<ffffffff99d1760b>] blk_mq_timeout_work+0xbb/0x1c0
[   80.618738]  [<ffffffff99a9e686>] process_one_work+0x176/0x4a0
[   80.618739]  [<ffffffff99a9f3d6>] worker_thread+0x126/0x3b0
[   80.618741]  [<ffffffff99a9f2b0>] ? manage_workers.isra.28+0x2a0/0x2a0
[   80.618743]  [<ffffffff99aa5ee1>] kthread+0xd1/0xe0
[   80.618744]  [<ffffffff99aa5e10>] ? kthread_create_on_node+0x140/0x140
[   80.618748]  [<ffffffff9a216e1d>] ret_from_fork_nospec_begin+0x7/0x21
[   80.618750]  [<ffffffff99aa5e10>] ? kthread_create_on_node+0x140/0x140
[   80.618750] ---[ end trace 29827fc8c242fa85 ]---
[   80.618756] BUG: unable to handle kernel NULL pointer dereference at 0000000000000048
[   80.618757] IP: [<ffffffff99b22069>] free_irq+0x39/0x90
[   80.618758] PGD 0
[   80.618759] Oops: 0000 [#1] PREEMPT SMP
[   80.618774] Modules linked in: sd_mod crc_t10dif crct10dif_generic intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel uas kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel usb_storage aesni_intel glue_helper ablk_helper cryptd wdat_wdt ahci lpc_ich i2c_i801 libahci nvme nvme_core acpi_power_meter 8021q garp mrp stp llc sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx raid1 raid0 iscsi_ibft iscsi_boot_sysfs tpm_tis(O) tpm_tis_core(O) tpm(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_en(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) ixgbe(O) dca i40e(O) e1000e(O) ip_tables
[   80.618780] CPU: 1 PID: 4792 Comm: kworker/1:1H Tainted: G        W  OE  ------------   3.10.0-957.1.3.el7.1.tis.x86_64 #1
[   80.618780] Hardware name: ARTESYN PCIE-7410/PCIE-7410, BIOS 1.7.3 5-Jun-2018 10:49
[   80.618782] Workqueue: kblockd blk_mq_timeout_work
…

I suspect that the root cause is the error "nvme nvme0: missing or invalid SUBNQN field."

nvme0n1 is our boot_device and rootfs_device.

Asking Google I'll found something similar (https://unix.stackexchange.com/questions/470778/nvme-missing-or-invalid-subnqn-field). 

What is the official way to change the kernel boot parameters for pxe boot?

Thanks

Marcel