CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Giuseppe,
The community briefly touched on the challenge you brought up in this thread on the StarlingX TSC & Community call last week.
Based on the conversation, the StarlingX community is still ahead of focusing more specifically on GPU support in the platform. The challenge you're facing seems like it's addressed in a later OpenStack version, compared to what's integrated in the platform
version you're using. There might be a way to port that back.
Were you able to make progress with troubleshooting on your end in the meantime?
Plans for upcoming StarlingX versions for GPUs include a few additions from the CNCF ecosystem, including:
- Nvidia DRA driver for GPUs in Kubernetes:
https://github.com/NVIDIA/k8s-dra-driver-gpu
-
https://docs.google.com/document/d/1BNWqgx_SmZDi-va_V31v3DnuVwYnF2EmN7D-O_fB6Oo/edit?tab=t.0#heading=h.bxuci8gx6hna
@Brad, @Chris: Can you please add and correct anything I missed?
Thanks and Best Regards,
Ildikó
———
Ildikó Váncsa
Director of Community
OpenInfra Foundation
> On Nov 7, 2025, at 01:31, giuseppe.delgaudio@nttdata.com wrote:
>
> Chris Friesen wrote:
>> We are exploring the use of NVIDIA GPUs, but from the context of making
>> the GPU resources available within Kubernetes. Currently we're trying
>> to get the NVIDIA GPU Operator for Kubernetes working, and we haven't
>> done much testing of PCI passthrough to VMs.
>>
>> It's interesting that the NVIDIA documentation you linked doesn't seem
>> to mention the "Hopper" architecture at all.
>>
>> Can you provide a source for the nvidia-vgpu-ubuntu-aie-580_580.82.0.deb
>> package that you said you installed?
>>
>> After installing the NVIDIA vGPU software were the NVIDIA-specific vfio
>> drivers installed as per Virtual GPU Software User Guide - NVIDIA Docs
>> <
https://docs.nvidia.com/vgpu/latest/grid-vgpu-user-guide/index.html#verify-install-update-vgpu-ubuntu> ?
>>
>> # lsmod | grep vfio
>> nvidia_vgpu_vfio 27099 0
>> nvidia 12316924 1 nvidia_vgpu_vfio
>> vfio_mdev 12841 0
>> mdev 20414 2 vfio_mdev,nvidia_vgpu_vfio
>> vfio_iommu_type1 22342 0
>> vfio 32331 3 vfio_mdev,nvidia_vgpu_vfio,vfio_iommu_type1
>> #
>>
>> Which version of the guest drivers were you using?
>>
>> Regards,
>> Chris
>>
> Hi,
>
> Thank you for the reply, sorry fo the delayed response but i don't received any notification.
>
> At the moment, we are completely focused on sharing vGPUs (specifically MIG) via Openstack on the STX 10 platform.
> As you mentioned, the Nvidia documentation does not include any reference to the Hopper architecture, but the procedure we followed to create VFs, etc., works correctly on Hopper.
>
> We cannot provide the VGPU manager software because it is bound by a license. This software installs the manager and the GPU driver. We have also installed the latest driver nvidia-vgpu-ubuntu-aie-580_580.95 provided with the nvidia 580.95.02 driver. As mentioned
above, this type of GPU on STX Kernel 6.6 uses Vendor Specific VFIO, so we cannot use MDEV to share cards between Openstack and STX bare metal, only VF.
>
>
> On the system after the package installation we have the following kernel module installed:
>
> sysadmin@controller-0:~$ lsmod | grep vfio
> nvidia_vgpu_vfio 126976 12
> nvidia 14381056 4 nvidia_vgpu_vfio
> vfio_pci_core 90112 1 nvidia_vgpu_vfio
> mdev 20480 1 nvidia_vgpu_vfio
> vfio_iommu_type1 45056 0
> vfio 61440 3 vfio_pci_core,nvidia_vgpu_vfio,vfio_iommu_type1
> kvm 1347584 2 kvm_amd,nvidia_vgpu_vfio
> irqbypass 12288 3 vfio_pci_core,nvidia_vgpu_vfio,kvm
>
> all the devices are attached with nvidia kernel module
>
> The guest drivers on the VMs are associated with the VGPU manager and are provided in the same zip file as the NVIDIA software, but the VM fails to even start up. When OpenStack schedules the VM and starts the machine, the physical STX host crashes and reboots
with a kernel panic.
>
> On nova conf we add this configuration to pass the VF:
>
> [pci]
> device_spec = {"address": "0000:03:00.2", "resource_class": "H200-vf", "managed": "no", "remote_managed": "true"}
> device_spec = {"address": "0000:03:00.3", "resource_class": "H200-vf", "managed": "no", "remote_managed": "true"}
> device_spec = {"address": "0000:03:00.4", "resource_class": "H200-vf", "managed": "no", "remote_managed": "true"}
> device_spec = {"address": "0000:03:00.5", "resource_class": "H200-vf", "managed": "no", "remote_managed": "true"}
> alias = {"name": "ah200-vf", "device_type":"type-VF", "resource_class": "H200-vf"}
>
> After Openstack shedule the VM the bare-metal host have the following crash:
>
> ============================================================
>
> [ 168.612066] EXT4-fs (rbd0): mounted filesystem 7fe62dc3-02b1-4fa2-8314-2043ff4c0fa5 r/w with ordered data mode. Quota mode: none.
> [ 168.612113] EXT4-fs (rbd1): mounted filesystem 76d79800-51f9-47db-b8cd-2a7b83c74ee1 r/w with ordered data mode. Quota mode: none.
> [ 430.648672] nvidia 0000:03:01.2: Enabling HDA controller
> [ 430.649641] nvidia 0000:03:01.2: Enabling HDA controller
> [ 430.649676] nvidia 0000:03:01.2: Runtime PM usage count underflow!
> [ 430.649818] ------------[ cut here ]------------
> [ 430.649821] WARNING: CPU: 0 PID: 9 at drivers/vfio/group.c:688 vfio_device_set_group+0xc8/0x1d0 [vfio]
> [ 430.649833] Modules linked in: vfio_pci sctp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag xt_CHECKSUM nbd rbd libceph dns_resolver nfnetlink_cttimeout ip6_tables xt_set ip6t_rpfilter ipt_rpfilter ip_set_hash_net ip_set_hash_ip ip_set veth xt_statistic
wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 curve25519_x86_64 libcurve25519_generic libchacha vxlan gre openvswitch nf_conncount nf_conntrack_netlink xt_recent xt_MASQUERADE xt_mark xt_conntrack bnxt_en(OE) nft_chain_nat xt_comment xt_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_multiport binfmt_misc iscsi_target_mod target_core_mod pci_pf_stub nfsv3 nfs fscache netfs esp4 dm_crypt trusted asn1_encoder xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge nfsd virtio_net net_failover
failover auth_rpcgss nfs_acl lockd grace 8021q garp stp mrp llc xfs cls_u32 sch_sfq sch_htb nvidia_vgpu_vfio(OE) nvidia(OE) vfio_pci_core mdev vfio_iommu_type1 vfio sctp ip6_udp_tunnel udp_tunnel xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O)
> [ 430.649866] nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_cm(O) intel_uncore_frequency_common drbd lru_cache libcrc32c fuse drm sunrpc bpf_preload efivarfs ip_tables overlay ext4 mbcache jbd2 dm_multipath
dm_mod mlx5_ib(O) ib_uverbs(O) wmi_bmof kvm_amd kvm ib_core(O) mlx5_core(O) irqbypass mlxfw(O) crct10dif_pclmul mlxdevm(O) crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 mlx_compat(O) rapl nvme psample uas nvme_core tls t10_pi acpi_cpufreq usb_storage
macsec crc64_rocksoft_generic pci_hyperv_intf ccp crc64_rocksoft mpt3sas crc64 wmi i2c_designware_platform i2c_designware_core ipmi_si ipmi_devintf ipmi_msghandler iavf(O) i40e(O) ice(O) [last unloaded: bnxt_en(OE)]
> [ 430.649899] CPU: 0 PID: 9 Comm: kworker/0:1 Kdump: loaded Tainted: G OE 6.6.0-1-amd64 #1 Debian 6.6.63-1.stx.99
> [ 430.649902] Hardware name: Lenovo ThinkSystem SR675 V3/SB27B87354, BIOS QGE140H-8.21 05/14/2025
> [ 430.649904] Workqueue: events work_for_cpu_fn
> [ 430.649909] RIP: 0010:vfio_device_set_group+0xc8/0x1d0 [vfio]
> [ 430.649914] Code: 8b 60 03 00 00 48 8d 42 d8 48 39 d1 75 0f eb 5f 48 8b 50 28 48 8d 42 d8 48 39 d1 74 52 4c 3b 28 75 ee 4c 89 f7 e8 a8 d4 49 c8 <0f> 0b 48 c7 c7 d8 ca ac c1 48 c7 c3 ea ff ff ff e8 93 d4 49 c8 4c
> [ 430.649916] RSP: 0018:ff4108f84028fd80 EFLAGS: 00010246
> [ 430.649918] RAX: ff18105388a1b080 RBX: ff1810543e9b0000 RCX: ff1810543e9b0360
> [ 430.649919] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff1810543e9b0370
> [ 430.649919] RBP: ff1810571c71e000 R08: ff18105573011b0e R09: 0000000000000000
> [ 430.649920] R10: 0000000000000001 R11: 0000000000000000 R12: ff1810539209c600
> [ 430.649920] R13: ff1810538d9a20c0 R14: ff1810543e9b0370 R15: ff4108f88850f990
> [ 430.649921] FS: 0000000000000000(0000) GS:ff1810b14f200000(0000) knlGS:0000000000000000
> [ 430.649922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 430.649923] CR2: 00007fb69bcb0000 CR3: 0000006091402002 CR4: 0000000000771ef0
> [ 430.649924] PKRU: 55555554
> [ 430.649924] Call Trace:
> [ 430.649927] <TASK>
> [ 430.649930] ? __warn+0x84/0x140
> [ 430.649933] ? vfio_device_set_group+0xc8/0x1d0 [vfio]
> [ 430.649939] ? report_bug+0x198/0x1b0
> [ 430.649943] ? handle_bug+0x53/0x90
> [ 430.649946] ? exc_invalid_op+0x18/0x70
> [ 430.649948] ? asm_exc_invalid_op+0x1a/0x20
> [ 430.649951] ? vfio_device_set_group+0xc8/0x1d0 [vfio]
> [ 430.649955] vfio_register_group_dev+0x4e/0xd0 [vfio]
> [ 430.649961] vfio_pci_core_register_device+0x197/0x410 [vfio_pci_core]
> [ 430.649967] ? device_initialize+0xab/0x110
> [ 430.649973] vfio_pci_probe+0x51/0x120 [vfio_pci]
> [ 430.649976] local_pci_probe+0x47/0xa0
> [ 430.649980] work_for_cpu_fn+0x17/0x30
> [ 430.649982] process_one_work+0x175/0x360
> [ 430.649984] worker_thread+0x280/0x390
> [ 430.649986] ? __pfx_worker_thread+0x10/0x10
> [ 430.649987] kthread+0xdd/0x110
> [ 430.649990] ? __pfx_kthread+0x10/0x10
> [ 430.649991] ret_from_fork+0x31/0x50
> [ 430.649994] ? __pfx_kthread+0x10/0x10
> [ 430.649996] ret_from_fork_asm+0x1b/0x30
> [ 430.650000] </TASK>
> [ 430.650000] ---[ end trace 0000000000000000 ]---
> [ 430.650010] vfio-pci: probe of 0000:03:01.2 failed with error -22
> [ 430.662135] tun: Universal TUN/TAP device driver, 1.6
> [ 430.663054] tap5f9c2335-39: entered promiscuous mode
> [ 431.328345] BUG: kernel NULL pointer dereference, address: 0000000000000010
> [ 431.328586] #PF: supervisor read access in kernel mode
> [ 431.328748] #PF: error_code(0x0000) - not-present page
> [ 431.328912] PGD 488b01067 P4D 0
> [ 431.329016] Oops: 0000 [#1] PREEMPT SMP NOPTI
> [ 431.329153] CPU: 55 PID: 109169 Comm: qemu-system-x86 Kdump: loaded Tainted: G W OE 6.6.0-1-amd64 #1 Debian 6.6.63-1.stx.99
> [ 431.329538] Hardware name: Lenovo ThinkSystem SR675 V3/SB27B87354, BIOS QGE140H-8.21 05/14/2025
> [ 431.329811] RIP: 0010:vfio_df_open+0x37/0x110 [vfio]
> [ 431.329978] Code: 48 83 ec 08 48 8b 1f 8b 83 5c 03 00 00 85 c0 75 5d c7 83 5c 03 00 00 01 00 00 00 48 8b 2f 4c 8b 67 28 48 8b 45 00 48 8b 40 68 <48> 8b 78 10 e8 90 f2 8e c7 84 c0 0f 84 af 00 00 00 4d 85 e4 74 51
> [ 431.330566] RSP: 0018:ff4108f8980efcf0 EFLAGS: 00010246
> [ 431.330730] RAX: 0000000000000000 RBX: ff1810543e9b4800 RCX: 00000000000000d7
> [ 431.330954] RDX: ffffffff893b78f0 RSI: ffffffffc14c3980 RDI: ff1810b2e0c3c000
> [ 431.331178] RBP: ff1810543e9b4800 R08: 000000000000000c R09: ff1810b2e0c3c000
> [ 431.331402] R10: 0000000000000001 R11: fefefefefefefeff R12: 0000000000000000
> [ 431.331626] R13: 000000000000002b R14: ff1810b2e0c3c000 R15: ffffffffe0c3c000
> [ 431.331850] FS: 00007f80e22c3e80(0000) GS:ff181110cb5c0000(0000) knlGS:0000000000000000
> [ 431.332105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 431.332286] CR2: 0000000000000010 CR3: 00000004717fa001 CR4: 0000000000771ee0
> [ 431.332511] PKRU: 55555554
> [ 431.332597] Call Trace:
> [ 431.332677] <TASK>
> [ 431.332744] ? __die+0x24/0x70
> [ 431.332845] ? page_fault_oops+0x15b/0x460
> [ 431.332973] ? terminate_walk+0xee/0x100
> [ 431.341558] ? bsearch+0x57/0x90
> [ 431.350127] ? exc_page_fault+0x69/0x150
> [ 431.358608] ? asm_exc_page_fault+0x26/0x30
> [ 431.366985] ? __symbol_put+0x70/0xa0
> [ 431.375241] ? vfio_df_open+0x37/0x110 [vfio]
> [ 431.383395] vfio_group_fops_unl_ioctl+0x292/0x720 [vfio]
> [ 431.391539] __x64_sys_ioctl+0x8f/0xd0
> [ 431.399556] do_syscall_64+0x58/0xb0
> [ 431.407438] ? sched_clock+0x10/0x30
> [ 431.415189] ? get_vtime_delta+0xf/0xb0
> [ 431.422834] ? ct_kernel_exit.constprop.0+0x81/0xa0
> [ 431.430412] ? __ct_user_enter+0x5e/0xd0
> [ 431.437861] ? syscall_exit_to_user_mode+0x32/0x40
> [ 431.445257] ? do_syscall_64+0x65/0xb0
> [ 431.452534] ? __do_sys_newlstat+0x64/0x90
> [ 431.459705] ? sched_clock+0x10/0x30
> [ 431.466733] ? get_vtime_delta+0xf/0xb0
> [ 431.473674] ? ct_kernel_exit.constprop.0+0x81/0xa0
> [ 431.480554] ? __ct_user_enter+0x5e/0xd0
> [ 431.487304] ? syscall_exit_to_user_mode+0x32/0x40
> [ 431.494015] ? do_syscall_64+0x65/0xb0
> [ 431.500672] ? __ct_user_enter+0x5e/0xd0
> [ 431.507176] entry_SYSCALL_64_after_hwframe+0x78/0xe2
> [ 431.513580] RIP: 0033:0x7f80e2cf5277
> [ 431.519962] Code: 00 00 00 48 8b 05 19 cc 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e9 cb 0d 00 f7 d8 64 89 01 48
> [ 431.533533] RSP: 002b:00007ffd09bc3398 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [ 431.540313] RAX: ffffffffffffffda RBX: 0000559d6081bcb0 RCX: 00007f80e2cf5277
> [ 431.546982] RDX: 0000559d6081d690 RSI: 0000000000003b6a RDI: 0000000000000028
> [ 431.553537] RBP: 0000559d60822800 R08: 0000559d5f19cbd0 R09: 00007ffd09bc12f9
> [ 431.559970] R10: 000000000000006f R11: 0000000000000246 R12: 0000559d6081b280
> [ 431.566256] R13: 0000559d6081d690 R14: 00007ffd09bc45c0 R15: 0000559d6081d690
> [ 431.572378] </TASK>
> [ 431.578322] Modules linked in: vhost_net vhost vhost_iotlb tap tun vfio_pci sctp_diag tcp_diag udp_diag raw_diag inet_diag unix_diag xt_CHECKSUM nbd rbd libceph dns_resolver nfnetlink_cttimeout ip6_tables xt_set ip6t_rpfilter ipt_rpfilter ip_set_hash_net
ip_set_hash_ip ip_set veth xt_statistic wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 curve25519_x86_64 libcurve25519_generic libchacha vxlan gre openvswitch nf_conncount nf_conntrack_netlink xt_recent xt_MASQUERADE xt_mark xt_conntrack bnxt_en(OE)
nft_chain_nat xt_comment xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_multiport binfmt_misc iscsi_target_mod target_core_mod pci_pf_stub nfsv3 nfs fscache netfs esp4 dm_crypt trusted asn1_encoder xt_addrtype nft_compat nf_tables nfnetlink br_netfilter
bridge nfsd virtio_net net_failover failover auth_rpcgss nfs_acl lockd grace 8021q garp stp mrp llc xfs cls_u32 sch_sfq sch_htb nvidia_vgpu_vfio(OE) nvidia(OE) vfio_pci_core mdev vfio_iommu_type1 vfio sctp ip6_udp_tunnel udp_tunnel xprtrdma(O)
> [ 431.578381] svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_cm(O) intel_uncore_frequency_common drbd lru_cache libcrc32c fuse drm sunrpc bpf_preload efivarfs ip_tables
overlay ext4 mbcache jbd2 dm_multipath dm_mod mlx5_ib(O) ib_uverbs(O) wmi_bmof kvm_amd kvm ib_core(O) mlx5_core(O) irqbypass mlxfw(O) crct10dif_pclmul mlxdevm(O) crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 mlx_compat(O) rapl nvme psample uas
nvme_core tls t10_pi acpi_cpufreq usb_storage macsec crc64_rocksoft_generic pci_hyperv_intf ccp crc64_rocksoft mpt3sas crc64 wmi i2c_designware_platform i2c_designware_core ipmi_si ipmi_devintf ipmi_msghandler iavf(O) i40e(O) ice(O) [last unloaded: bnxt_en(OE)]
> [ 431.704871] CR2: 0000000000000010
>
> ===========================================================
>
> Do you have any suggestions on OpenStack or STX to make everything work? Or do you have any ideas about this error?
>
> Notes:
> e also tested this type of passthrough on a standard system (NO STX) with Ubuntu 24, Kernel 6.8, and KVM 10, and everything works correctly.
>
> P.S
> We tried also with STX 11 rc1 and with kernel 6.12 and we have the same error,
>
>
> Thanks for the help
>
> Regards,