r/VFIO • u/Public-Net-8864 • 1d ago
Two Proxmox VMs sharing one NVIDIA GPU for CUDA, with no passthrough and no vGPU license. Free package to try.
reddit.comHey, check this out…
r/VFIO • u/Public-Net-8864 • 1d ago
Hey, check this out…
r/VFIO • u/KstrlWorks • 2d ago
You ever wonder why your AMD GPU boots fine as your main display card, but the moment you move it to a VM you hit black screens, code 43, hangs, lspci telling you your card is revision ff, and only a full host reboot brings it back? Ask about it anywhere and somebody will tell you it's the AMD reset bug. Maybe. But most people do not have a reset bug at all, they have one of several problems that look identical from outside the case.
Reset gap: the card holds state that nothing in the PCIe reset path can see or clear. Every consumer AMD card from Polaris on has this, the RX 9000 included.
Reset bug: the gap, plus AMD's own driver being unable to bring the card back, leaving custom code or a full power cycle as the only ways out.
When the driver binds it loads firmware into two small processors on the card, the PSP and the SMU, and when it unbinds nothing clears them. The only reset Linux gets on an RX 5000 (which I'll use for this example but the same logic applies to the other RDNA cards) is a Secondary Bus Reset, the top diagram. The bridge drops the link, config space gets cleared, the card's PCIe front end comes back up and reports that it worked. It never goes near Region 5, the BAR where the PSP and SMU mailboxes live, so your next guest inherits whatever the last driver left running.
AMD's real reset is mode1, the bottom diagram, a single command left in the PSP's mailbox. Only a driver that has mapped Region 5 can send it, which means amdgpu or vendor-reset. vfio-pci knows nothing about graphics hardware, so it never will.
amdgpu resets the card when it binds, not when it unbinds, so on most cards a dirty GPU gets cleaned up just by binding it back to amdgpu. On the RX 5000 that reset goes through the PSP, and amdgpu waits for the PSP to say it's ready before sending it. If the PSP has gone quiet you get psp is not working correctly before mode1 reset! then asic reset on init failed, and the driver never attaches. No retry, no fallback.
Every generation since moved the reset further from that problem:
| Card | Mode1 goes to | Checked before sending |
|---|---|---|
| RX 5000 | the PSP mailbox | PSP ready |
| RX 6000 | the SMU mailbox | PM firmware 58.26, PSP alive |
| RX 7000 and RX 9000 | the SMU debug mailbox | nothing |
amdgpu refuses. It also blocks the bus reset that knocks these cards off the bus. Its udev rule writes device_specific to the GPU's reset_method so the kernel actually uses it.amdgpu and unbind it again before starting the VM, which gets you the reset for free.AMD is technically fully compliant with everything they're supposed to do. If you've done any compliance work you're quite aware of what I mean, i.e. the spirit of the law vs the word of the law. The spec defines what a reset has to do and AMD's PCIe front end does all of it. What the spec never explicitly writes down is the two processors sat behind that front end still holding the last owner's firmware. Which leaves two ways out. FLR is optional, but AMD left the bit clear. Or the kernel's own quirk table, which is where Intel, Samsung, and Qualcomm put devices that need special handling. There's a working reset on the card, it's just not one other than amdgpu can call.
So I tried to keep this higher level since I wanted people to understand, and kept the full version here. I realized when the post is too long no one reads it, and that kinda negates the entire point of it. Oh the States you'll go to: The AMD Reset Bug and Everything It Isn't
r/VFIO • u/OfficalFAK • 3d ago
Nvidia has actively tried to keep enterprise features locked behind its premium cards for the longest time. However, a few years ago, they gave us a straightforward way to pass through their GPUs to VMs without running into the infamous reset bug.
If Nvidia can do it, why is AMD so stubborn about it? My RX 9070 XT has been an absolute pain to get working reliably in a VM. I've pretty much given up on GPU passthrough, whereas my GTX 1070 Ti ran flawlessly for almost two years with virtually no issues.
This is very disappointing as the Rx 9070 XT is a 3 slot card, making it impossible for me to add the secondary GPU to my computer. So my only options are to either look for a single slot Quadro or change my computer case.
UPDATE:
After hearing all you guys saying that it is possible, i did try again and got everything working soo far, i dont know how stable the setup will be but for now, its working, ill share the full config soon after ive tested it a few weeks.
r/VFIO • u/Opening_Nature_2904 • 4d ago
r/VFIO • u/Public-Net-8864 • 6d ago
My earlier post was done using Qwen2.5 and rightly so I was called out for it. So I reran it with Qwen3.8 27B, Qwen3.5 9B and Qwen3.6 35B-A3B with as much proof as I could think of. The VM has no NVIDIA device, driver or nvidia-smi. llama.cpp's CUDA calls go through VMAT to the card on the host. Qwen3.8 27B: 45.4 tokens/s on the host, 44.6 in the VM. All four models land at 98%. Three VMs at once get 45.7 together, against 45.0 for three programs on the host.
Once I get my hands on Qwen4, I’ll post another video based on it…
Full video with every step: https://youtu.be/r6yhOVWxP5Y?is=reuv6gZptnDNzU1D
r/VFIO • u/Zestyclose_Dig8916 • 6d ago
vmdb.it is my site, so full disclosure right away. It collects compatibility reports for gaming in virtual machines, VMware, VirtualBox, QEMU/KVM, Hyper-V, Parallels, and it also takes regular Proton reports for the same games. The VM part exists because the question "will this game's anti-cheat ban me if I run it in a VM" comes up constantly and the answers are scattered across dead forum threads.
Reports are anonymous, no account needed, takes a few minutes. There's also a My Setup thing where you store your hardware/VM/Proton config in your own browser storage. Reports matching your setup get sorted first and the report form pre-fills from it, so the second report is mostly clicking submit.
Anyway, the actual reason for the post: coverage is thin and it's not growing on its own. Pulled these from the API a few minutes ago:
\- Crimson Desert Enhanced: 0 reports. It's also one of the most searched games on the site right now.
\- Bodycam: 0
\- Marathon: 0
\- CoD Modern Warfare 4: 1 VM report, 0 Proton
\- Forza Horizon 6: 3 VM reports, 0 Proton
If you run any of these in a VM or through Proton, a report takes like five minutes and the next person who searches the game gets an actual answer instead of an empty page. Same goes for anything else you play of course, those are just the gaps people keep hitting.
And if you ran a game in a VM and got banned by anti-cheat, please write that up too. That's data that basically doesn't exist anywhere else.
The API is public and read-only if you'd rather script against it: API docs
If you think the whole idea is dumb or something on the site is broken, I want to hear that too.
r/VFIO • u/BubblyZebra616 • 8d ago
Specs: OS: Proxmox CPU: Ryzen 9 5900x MOBO: Gigabyte X570 Aorus Master GPU1: Gigabyte Radeon RX 9070 XT GAMING OC 16G GPU2: RTX 3070 ti GPU3: RX 6400 (used for host) RAM: 64GB DDR4 (F4-3200C16D-64GVK)
I have yet to assemble this system but I have some of these parts on the way to complete a new build. I want to pass through the 9070 XT to a CachyOS VM. I have seem some people have successfully done this with a Windows VM but not Linux. Has anyone does this?
r/VFIO • u/programmed_insanity • 8d ago
A year ago I posted about my dynamic GPU passthrough setup for which I wrote a overengineered program to manage and streamline all that. At the time I didn't release it because I didn't think it was robust enough. After a year of using it I only had to add some extra logging or fix parsing. At one point I just started to forget it even existed because I never had any problems with it. So now I think it is somewhat robust enough to be used but I make no promises after all this was made only for one machine and never tested on anything else. The code is here.
So instead of moving scripts around you just add a bit of xml to your VM config. For example this bit of xml for the pci device 0000:0a:00.0. Which tells the hook to unbind from kwin, kill all the processes, unload the nvidia driver and load the vfio driver.
<dyn:dynamic_unbind xmlns:dyn="0.0.0.0">
<pci_device domain="0x0000" bus="0x0a" slot="0x00" function="0x0" driver_finder_on_shutdown_method="store"/>
</dyn:dynamic_unbind>r/VFIO • u/Local_Cobbler_8974 • 8d ago
Has anyone been able to get EveNG to work on Apple Silicon M1 Pro? If so would you mind explaining how you got it to work?
r/VFIO • u/BrazilianAnimal • 9d ago
With Nvidia in the news this week, I want to tell an old story about how I believe this company treats its customers.
In 2021 I had a laptop with a GeForce GTX 1650 and wanted to pass the discrete GPU through to a Xen guest to run KataGo. Nothing exotic: my hardware, my hypervisor, Nvidia's own official Linux driver in the guest.
It didn't work. The driver failed with an error about PCI BARs. So I did what anyone would do and treated it as a real BAR problem. I spent between one and two weeks on firmware settings, kernel parameters and Xen configuration. None of it helped, because none of it was the problem.
Eventually I asked myself a different question: if Nvidia wanted to stop this from working on purpose, what is the laziest way they could do it? Answer: compare the hypervisor's CPUID signature against a hardcoded string.
So I ran the strings command on the driver binary, and there it was: XenVMMXenVMM, Xen's CPUID signature. I changed it to something random of the same size.
The GPU worked immediately. Same hardware, same BARs, same everything.
I never saw the source code (it was not publicly available), so this is inference, but the experiment is hard to read any other way: the driver detects Xen and refuses to run, and it reports the refusal as a hardware fault. Not "GeForce is not supported in virtual machines", which would at least have been honest and saved me over a week. A fake error that sends you debugging something that isn't broken.
I learned afterwards that this was the Linux/Xen face of a well-known error that people had been fighting when trying to pass Nvidia GPUs to guests. This points to driver code deliberately made to fail, with a misleading message. In my opinion, the reason was to push virtualisation users to Quadro cards with similar silicon at several times the price.
They dropped the check later, and from then on the GPU ran without patching their driver with a hex editor.
The moral, for me: a company that will lie to you in an error message has zero ethics. They didn't then, I see no reason to believe they do now, and nothing they ship should be trusted further than what you can verify yourself. Which, with a closed driver, is not very far.
Note: I was just a hobbyist Go player trying to take a break at the time; I did not keep any logs with the specific Xen, Linux and Nvidia driver versions.
r/VFIO • u/protienbudspromax • 12d ago
So my system config is 9900X, 5070ti, 32g ddr5.
I have managed to pass through the igpu of the 9900X and get it binded to a windows VM when I start via qemu/virt-manager. But thing is it seems that once it starts if i shut it down i cannot restart the VM with the gpu still being loaded. I need to reboot my host. (Code 43)
My main job in windows is dev related for windows native stuff. So it works out relatively fine. But i was wondering if I am missing something and this issue is resolved now already.
In case it cant be done was wondering what low powered gpu i should get, cgpt and claude gave mixed answers wrt amd vs nvidia, says rdna 1 cards also have similar issues rdna2 may be better but not sure. I was eyeing the RX 6500 as its cheap enough will solve my problem and can be used without changing my psu (corsair 850w) . So should i bite the bullet and get that 2nd gpu or are there options i can use?
r/VFIO • u/LurkingWithoutEyes • 13d ago
Hey, a while back I posted about getting Hyper-V style GPU-PV working on KVM. The VM runs the real Windows driver and the host keeps the card, so my desktop stays on it the whole time.
Since then I've rewritten it all in Rust, made an app that installs Windows from an ISO by itself in about 9 minutes, and got AMD working. Windows guests run AMD's own driver now, and on my Ryzen's iGPU the desktop, D3D11, D3D12, Vulkan and OpenGL all work. Video decode doesn't yet. Linux guests on AMD work too.
The problem is I've only got my 5090 and that iGPU, and one tester has run it on a 4080. So I'm looking for people with:
You need the open NVIDIA kernel modules or amdgpu on the host. If your card isn't supported, it still installs with plain VMs and tells you why.
Obviously this project is experimental. My point in working on it is to prove that Linux can run GPU VMs. Since no one had ever built a tool like this, I always thought it wasn't really possible, so here we are.
My goal is to support as many GPUs as I can so more people can try it, but I won't be able to manage this project solo.
Repo: https://github.com/xnixy/guesthouse
AMD reports go in #1 and NVIDIA ones in #9, or just reply here.
Edit: Thanks everyone for helping! We made a lot of progress with your help.
r/VFIO • u/PomeloIllustrious219 • 13d ago
r/VFIO • u/CompetitiveDraft9381 • 13d ago
It runs on KVM with two kernel patches (Apple's PAC VM-key registers, and one MMIO store XNU makes that KVM can't handle). omvm builds those as a second kernel and keeps the stock one in the boot menu. QEMU is aquarat's KVM port of the vmapple machine. Graphics go from macOS's paravirtual GPU driver through reims-vgpu to Vulkan on Honeykrisp.
Honeykrisp needed VK_EXT_fragment_shader_interlock exposed, because the macOS compositor uses it almost everywhere. That's in my Mesa branch. It still fails 64 MSAA CTS cases, so it isn't ready for an MR yet. aquarat's 16-bit flat varyings fix is in there too.
xcodebuild runs at 95-99% of native CPU. I use it in a 120 Hz window or over Moonlight from my phone. Apple ID and iMessage don't work, because Apple ties them to a macOS host's Secure Enclave.
Repo: https://github.com/sapoepsilon/omvm
Mesa branches: https://github.com/sapoepsilon/mesa
Follow me on twitter: x.com/killerisya
r/VFIO • u/BrineHero • 14d ago
I’ve been dual-booting Windows and Linux for a long time, but Linux has become my primary OS for pretty much everything except work.
I still regularly need Windows for AutoCAD, Revit, and ReCap. I’ve tried BricsCAD on Linux, and while it’s a great application, it doesn’t quite cover all my needs.
Lately, I’ve been considering switching to a Windows VM using QEMU/KVM with GPU passthrough instead of constantly rebooting into Windows.
I’m particularly interested in hearing from people who actually use a similar setup for daily work, especially with CAD or other GPU-intensive professional software.
A few questions:
How does performance compare to running Windows natively?
Have you experienced stability issues during long working sessions?
How seamless is the integration with the Linux desktop (clipboard, file sharing, multiple monitors, keyboard shortcuts)?
Does GPU passthrough cause any issues with Hyprland or Wayland?
Is the setup reliable enough for professional work, or does dual boot remain the more practical solution?
My current setup:
OS: CachyOS / Windows 11 dual boot
Desktop: Hyprland (Wayland)
CPU: Ryzen 7 5700X
GPU: Radeon RX 570
RAM: 16 GB
I’d really appreciate any real-world experiences, especially from people who have used this kind of setup long-term.
r/VFIO • u/nathannguyennng • 14d ago
Hey everyone, I posted here a few days ago about getting SR-IOV vGPU working on consumer Ampere GPUs. I've made quite a bit of progress since then and just released geforce-sriov v0.2.0 beta.
To recap, geforce-sriov is a toolkit that lets multiple VMs share a consumer NVIDIA GPU through SR-IOV, rather than mdev used by many earlier vGPU unlock projects.
The biggest news is Ada support is now working! Thanks to a friend who generously lent me their RTX 4070 SUPER, I was able to experiment with Ada and figure out how to get SR-IOV vGPU working on it.
Unlike Ampere, most consumer Ada cards report only 1 VF in hardware, so getting 4 VFs working required additional patches to NVIDIA's host driver and vGPU plugin. The current version supports up to 4 VFs per card using the proprietary driver with GSP disabled.
What's new in v0.2.0:
geforce-sriov setup command now handles prerequisite checks, installation, driver patching, and VF activation automatically. Run it once, cold power cycle the host, then run it again.It's still beta, and so far I've only tested it on my dual RTX 3080s and an RTX 4070 SUPER but other Ampere and Ada GPUs should work too.
GitHub: https://github.com/noon-at-cgn/geforce-sriov
Release: https://github.com/noon-at-cgn/geforce-sriov/releases/
If anyone has other Ampere or Ada GPUs and wants to try it, compatibility reports would be really helpful.
r/VFIO • u/sexygraphicscard • 16d ago
Hello everyone,
A few months ago, I had a psuedo single GPU passthrough setup working with my 7900xtx (I say psuedo because I am passing both a 7900xtx and 6600 to the vm.) I was using it one day when I accidentally rebooted the VM instead of shutting it down, and since then, I haven't been able to get a display signal. I can VNC into the VM from my laptop and as far as I can tell, the GPU is attached to the VM with no issues (atleast according to Windows). But in the displays, I see the Amd vDisplay instead of my physical monitor, so I'm not sure what do do next. Here's a list of what I've tried so far:
This is also the current virt manager xml that I have atm:
<domain type='kvm'>
<name>gaming-7900xtx</name>
<uuid>daf7ae8c-24f6-4c3f-9830-577b781f0a91</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>33554432</memory>
<currentMemory unit='KiB'>33554432</currentMemory>
<memoryBacking>
<source type='memfd'/>
<access mode='shared'/>
</memoryBacking>
<vcpu placement='static'>24</vcpu>
<os firmware='efi'>
<type arch='x86_64' machine='pc-q35-11.1'>hvm</type>
<firmware>
<feature enabled='no' name='enrolled-keys'/>
<feature enabled='yes' name='secure-boot'/>
</firmware>
<loader readonly='yes' secure='yes' type='pflash' format='raw'>/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
<nvram template='/usr/share/edk2/x64/OVMF_VARS.4m.fd' templateFormat='raw' format='raw'>/var/lib/libvirt/qemu/nvram/gaming-7900xtx_VARS.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode='custom'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vpindex state='on'/>
<runtime state='on'/>
<synic state='on'/>
<stimer state='on'/>
<vendor_id state='on' value='9876543210ba'/>
<frequencies state='on'/>
<tlbflush state='on'/>
<ipi state='on'/>
<avic state='on'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
<vmport state='off'/>
<smm state='on'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='on'>
<topology sockets='1' dies='1' clusters='1' cores='12' threads='2'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/home/colin/VMs/gaming.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
</controller>
<controller type='pci' index='15' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='15' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='16' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='16' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='17' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='17' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='18' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='18' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='pci' index='19' model='pcie-to-pci-bridge'>
<model name='pcie-pci-bridge'/>
<address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs'/>
<source dir='/home/colin/VMSHARE'/>
<target dir='VMSHARE'/>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs'/>
<source dir='/mnt/5a3a870e-a864-4373-b19f-6732830ebbbc/Games'/>
<target dir='GamerSSD'/>
<address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
<driver type='virtiofs'/>
<source dir='/mnt/33eed47a-4105-468b-8f0d-ce491e8e7f21/Games'/>
<target dir='GamerSSD2'/>
<address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
</filesystem>
<interface type='network'>
<mac address='52:54:00:12:58:00'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<tpm model='tpm-crb'>
<backend type='emulator' version='2.0'>
<profile name='default-v1'/>
</backend>
</tpm>
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' passwd='146064'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<audio id='1' type='none'/>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x13' slot='0x01' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
</source>
<rom bar='off'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</hostdev>
<watchdog model='itco' action='reset'/>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>And here are the relavent specs:
I'm gonna keep trying to chip away at this, but I'll do anything if you have any insight or thoughts about this 🙏 Thank you!!
Hey everyone,
We all know the struggle in the Linux community with software support and Anti-Cheat games. Even VM and GPU passthrough solutions are overly complicated, and one wrong move can turn your whole host into a black screen.
That's why I built a tool called Wise Linux Studio.
I combined advanced backend tools like QEMU/KVM, VFIO-PCI (for hardware isolation), libvirt, and Looking Glass, and wrapped them in a simple Python (Tkinter) GUI. It passes through the actual hardware to the Windows VM, but with a strict safety system. It scans your hardware topology, detects if you have a dual-GPU or single-GPU setup, differentiates the iGPU from the dGPU, and dynamically adjusts how the tool operates based on that.
I just wanted to merge these powerful tools with a safety mechanism and a simple GUI that actually guides you.
It’s not a massive world-changing project (: and I can't guarantee every single program will work perfectly since I haven't tested everything yet (:
But I did test it on a laptop with only an integrated GPU. The safety system immediately detected it, locked the GRUB setup buttons, and blocked the VM creation to protect the host from crashing!
Currently, the tool only supports Fedora.
If anyone is interested or wants to mess around with the code, here is
r/VFIO • u/LurkingWithoutEyes • 17d ago
No VFIO, no vGPU, no SR-IOV. The host keeps the card the whole time.
Windows boots under QEMU/KVM, thinks it's running on Hyper-V, and just loads the normal NVIDIA Windows driver. There's a ~600 line QEMU patch for the VMBus devices plus a Python daemon that pretends to be the Hyper-V host and talks to the NVIDIA Linux driver. Frames go straight from VRAM to a GTK viewer as dmabufs.
Right now D3D12, D3D11, Vulkan and OpenGL all run on the real card. Godot Forward+ works up to 4K (windowed or exclusive fullscreen) and sound goes through PipeWire. I also made a little app that does a fully unattended Windows install from an ISO in about 8 minutes, and the GPU already works when it finishes.
I'm also poking at a Linux guest with GPU-PV and CUDA, but that's still early.
Raw numbers on a 9950X3D + RTX 5090 on Fedora:
| Host | Guest |
|---|---|
| Compute | 115-119 TFLOPS |
| VRAM copy | ~1,070 GB/s |
| Submit round trip | 21 µs |
Vulkan and OpenGL pixel tests match the host exactly.
Code isn't public yet. It still depends on a capture from my own machine and I've only tried it on a 5090. Mostly just wanted to show it off. Anyone else been messing with this on KVM? What would you want tested first?
Edit (forgot to mention): GPU isolation between processes, each Windows process gets its own GPU page tables. In a test where one process tried to read and overwrite another's GPU memory, it got 0 of 16,384 words and the data survived.
Edit 2: I started the public repo, where I’m moving the code in pieces. It will take some time, but at least for now you can review the QEMU patch and the general structure. https://github.com/xnixy/guesthouse/ and roadmap at https://github.com/xnixy/guesthouse/issues/2
Edit 3: almost all code is public, please see the roadmap.
So, lately I’ve been building an AI development environment that includes a graphical interface, and I wanted everything to run as "smoothly" as possible. Since my onboard GPU was just sitting there unused, I figured... why not? Unfortunately, I ended up spending an entire night getting it configured. The documentation is sparse, and every system presents its own unique set of quirks. Anyway, I finally finished around 6:00 AM, and since it’s all set up, I thought others might be interested in this ready-to-use solution.
What’s included: A reproducible kit for an Ubuntu 26.04 virtual machine with an XFCE/Xorg desktop environment, an AMD GPU configured via VFIO PCI passthrough, and a "Looking Glass" client running on the host system (using a second GPU). It features bidirectional clipboard support via SPICE, an ICH9 audio system, LAN bridging, IVSHMEM/KVMFR, automatic startup, and a 4K SDR configuration.
It’s available on GitHub here: https://github.com/rav0rmat/looking-glass-b6-linux-kit
I hope you find it useful—and, of course, I’d appreciate any feedback!
r/VFIO • u/GhettoBSD • 17d ago
TL;DR: the fix
**Symptom:** an NVIDIA Blackwell card passed through to a Linux VM works on the first VM boot after the host boots. Any **guest** reboot or shutdown then hangs. After that, the card is dead inside the VM (`nvidia-smi`: "No devices were found"; dmesg: `unexpected WPR2 already up, cannot proceed with booting GSP`) until the **whole host** is rebooted. FLR and PCIe link resets from the host don't bring it back.
**Cause (in our case):** NVIDIA's *display* modules (`nvidia_modeset`, `nvidia_drm`) hang while unloading in a VM that has no display (stuck in `nvRmVTSwitch`). So the driver never gets to shut the GPU's GSP firmware down. Its protected memory region (WPR2) stays up, and the next driver start refuses the card.
**Fix (headless / compute guests):** never load those two modules:
```
# /etc/modprobe.d/nvidia-headless.conf
blacklist nvidia_drm
blacklist nvidia_modeset
install nvidia_drm /bin/false
install nvidia_modeset /bin/false
```
```
sudo update-initramfs -u
```
Then reboot the **host** once, to get a clean card.
After that only `nvidia` and `nvidia_uvm` load (all that CUDA needs). Guest reboot, guest poweroff and an ACPI shutdown from the host all work, and the card comes back every time without a host reboot, including when the VM autostarts after a host reboot.
**What it does NOT fix:** hard-killing the VM while the driver is loaded (`vm poweroff -f`, `virsh destroy`, and so on) still leaves WPR2 up, and only a host reboot clears that. Shut the guest down properly.
---
## Setup
- **Host:** FreeBSD 15.1-RELEASE, **stock bhyve (no patches)**, vm-bhyve, UEFI guest firmware (`BHYVE_UEFI.fd`). Intel i7-7700T (Kaby Lake), B250 chipset, VT-d enabled, 64 GB RAM.
- **GPU:** NVIDIA RTX PRO 4000 Blackwell SFF Edition, 24 GB (`10de:2c33`, GB203GL), plus its audio function (`10de:22e9`), in the CPU's x16 slot.
- **Guest:** Ubuntu Server 24.04.5, kernel 6.8.0-147-generic, 4 vCPU, 32 GB (wired), headless.
- **Guest driver:** 615.71.09, open kernel module (`nvidia-headless-no-dkms-615-server-open`, `linux-modules-nvidia-615-server-open-generic`, `nvidia-utils-615-server`), CUDA 13.4.
## Passthrough config
Host, `/boot/loader.conf`:
```
vmm_load="YES"
pptdevs="1/0/0 1/0/1"
```
vm-bhyve guest config:
```
passthru0="1/0/0"
passthru1="1/0/1"
wired_memory="yes"
```
That gives bhyve `-S` (wired memory) and `-s 0:6:0,passthru,1/0/0 -s 0:6:1,passthru,1/0/1`.
Host after a reboot:
```
$ pciconf -l | grep ppt
ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2c33 subvendor=0x10de subdevice=0x2053
ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x22e9 subvendor=0x10de subdevice=0x0000
```
Guest:
```
$ lspci | grep -i nvidia
00:06.0 VGA compatible controller: NVIDIA Corporation Device 2c33 (rev a1)
00:06.1 Audio device: NVIDIA Corporation Device 22e9 (rev a1)
```
Driver install (Blackwell needs the **open** driver):
```
sudo apt update
ubuntu-drivers list --gpgpu
sudo ubuntu-drivers install --gpgpu nvidia:615-server-open
sudo apt install nvidia-utils-615-server # --gpgpu leaves out nvidia-smi
```
The first boot with the driver works:
```
$ nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX PRO 4000 Blac... Off | 00000000:00:06.0 Off | Off |
| 30% 36C P8 9W / 70W | 1MiB / 24467MiB | 0% Default |
+-----------------------------------------+------------------------+----------------------+
```
## The problem: what we saw
```
$ nvidia-smi
No devices were found
```
```
$ sudo dmesg | grep -i -e nvrm -e wpr -e gsp | head -15
NVRM: GPU0 objClInitPcieChipset: *** Unable to get PCI port handles
NVRM: Xid (PCI:0000:00:06): 62, 324f892d 0002e740 00000000 2067ede2 2067f180 2067f0c2 20680598 20680a06
NVRM: GPU0 _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x1022 (RECOVERY_ACTION) during bootup without API lock
NVRM: GPU0 _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x1029 (PMU_HALTED) during bootup without API lock
NVRM: GPU0 kgspSendInitRpcs_IMPL: GMCAPI_CMD_GSP_INIT failed: 0x65
NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgspSendInitRpcs(pGpu, pKernelGsp) @ kernel_gsp_gh100.c:1091
NVRM: GPU0 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
NVRM: GPU0 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)
NVRM: GPU0 RmInitAdapter: Cannot initialize GSP firmware RM
NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x62:0x40:2130)
NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0
NVRM: GPU 0000:00:06.0 init failed during probe (ret=-5).
```
Only a full host reboot brought the card back.
(`objClInitPcieChipset: *** Unable to get PCI port handles` shows on every boot, including the good ones. bhyve puts the GPU straight on the guest's bus 0, with no PCIe root port above it. It seems harmless.)
## Troubleshooting
**Does the card support FLR?** On the host:
```
$ pciconf -lc pci0:1:0:0
...
cap 10[60] = PCI-Express 2 legacy endpoint max data 256(256) FLR RO
max read 512
link x8(x16) speed 2.5(32.0) ASPM disabled(L1) ClockPM disabled
...
```
Yes, it does.
**FLR from the host, with the VM stopped:** `devctl reset ppt0` and `devctl reset ppt1` (both silent). Then `vm start`: still "No devices were found".
**A PCIe link reset from the host:** `devctl reset pci1`. That's the bus under the root port, and FreeBSD's bridge reset disables the link and retrains it (a hot reset), saving and restoring the devices' config around it. First we checked that bus 1 holds only the GPU's two functions:
```
$ pciconf -l | grep 'pci0:1:'
$ devinfo -p pci1
pci1 pcib1 pci0 pcib0 acpi0 nexus0
```
Result: no change. The guest dmesg showed `unexpected WPR2 already up` again.
**Research:** this matches the known Blackwell "reset bug" reported for the RTX 5090 and RTX PRO 6000 under Linux KVM/VFIO. GSP firmware sets up a protected region (WPR2) that survives FLR, so the next driver start refuses to boot GSP. Links at the end.
**The idea:** if the guest's driver unloaded properly, GSP would tear WPR2 down itself. On a fresh card (after a host reboot), inside the guest:
```
$ lsmod | grep nvidia
nvidia_uvm 2088960 0
nvidia_drm 155648 0
nvidia_modeset 1556480 1 nvidia_drm
nvidia 16875520 2 nvidia_uvm,nvidia_modeset
$ sudo systemctl stop nvidia-persistenced # fine
$ sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia # hangs forever
```
From a second SSH session:
```
$ sudo dmesg | grep -i nvrm
NVRM: Xid (PCI:0000:00:06): 56, pid=..., name=(udev-worker), CMDre 00000000 00000000 00000400 00000001 00000000
...
nvRmApiControl+0x46/0x60 [nvidia_modeset]
? nvRmVTSwitch+0x2d/0x40 [nvidia_modeset]
```
So the unload hangs in **`nvidia_modeset`'s VT switch** (handing the console back), in a VM where nothing is attached to the GPU's outputs. The same thing happens on reboot or shutdown. GSP never shuts down, WPR2 stays up, and the next start fails. (The `Xid 56` is a display-engine error, logged as `udev-worker` loaded the display modules at boot.)
## The fix
```
f=/etc/modprobe.d/nvidia-headless.conf
echo 'blacklist nvidia_drm' | sudo tee $f
echo 'blacklist nvidia_modeset' | sudo tee -a $f
echo 'install nvidia_drm /bin/false' | sudo tee -a $f
echo 'install nvidia_modeset /bin/false' | sudo tee -a $f
sudo update-initramfs -u
sync
```
Then one host reboot for a clean card. After it:
```
$ lsmod | grep nvidia
nvidia_uvm 2088960 0
nvidia 16875520 1 nvidia_uvm
```
| Test | Before the fix | After the fix |
|---|---|---|
| `sudo poweroff` in the guest, then `vm start` | hang | card back, about 45 s from poweroff to `nvidia-smi` |
| `sudo reboot` in the guest | hang | card back, about 40 s |
| `vm stop` (ACPI) from the host, then `vm start` | hang | card back |
| host reboot, VM autostarts (`vm_list`) | n/a | card back |
## Notes and caveats
- **Headless/compute only.** No display from the passed-through GPU in the guest. If you need the display modules, this isn't your fix. We didn't test a guest with a monitor on the GPU, where the VT switch might behave differently.
- **Don't hard-kill the VM** while the driver is loaded. That still leaves WPR2 up.
- **After a guest driver upgrade,** check the modprobe file is still there and `lsmod` shows only `nvidia` and `nvidia_uvm`.
- So far this is tested with `nvidia-smi` and repeated restarts.
- **KVM/Proxmox users with Linux guests:** we haven't tested on KVM, but if your guest is headless and you hit the Blackwell reset bug, this might be the same mechanism, and it's cheap to try.
- **For FreeBSD folks:** on the same host, a Maxwell GTX 960 couldn't be initialized in a bhyve guest at all (`RmInitAdapter failed`), as others report for older NVIDIA cards without the community bhyve patches. This Blackwell card with the open/GSP driver needed **no** bhyve patches on 15.1.
## Bonus: the same card on the FreeBSD host itself
Before passing it through, we ran the card on the FreeBSD host (Vulkan, for whisper.cpp). Two gotchas:
## References
- Tom's Hardware, RTX 5090 / PRO 6000 reset bug: https://www.tomshardware.com/pc-components/gpus/rtx-5090-pro-6000-bug-forces-host-reboot
- NVIDIA forums, RTX PRO 6000 Max-Q FLR failure after KVM guest shutdown: https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-max-q-10de-2bb4-flr-fails-on-vfio-pci-release-after-kvm-guest-shutdown-gpu-inaccessible-until-cold-power-cycle/386790
- open-gpu-kernel-modules issue #1442 (stale WPR2): https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1442
- Level1Techs, RTX 50 reset bug thread: https://forum.level1techs.com/t/do-your-rtx-5090-or-general-rtx-50-series-has-reset-bug-in-vm-passthrough/228549
r/VFIO
•
u/nathannguyennng
•
18d ago
TLDR:
mdev approach, this project uses actual PCIe SR-IOV virtual functions to share a GPU between multiple VMs.Source code and documentation: https://github.com/noon-at-cgn/geforce-sriov
Update (Nov 7, 2026): Just released geforce-sriov v0.2.0 beta, now with Ada support! This release also brings a much simpler setup process, improved documentation, and several other fixes.
Update (Nov 4, 2026): Initial testing with Ada on a 4070 Super borrowed from my friend shows promising results, although the method requires quite a few changes and has more limitations. I'll update the post once I've done more testing.
I've been experimenting with the hidden PCIe capabilities on NVIDIA GeForce cards, particularly SR-IOV, and managed to get NVIDIA's vGPU stack working on my RTX 3080s using actual PCIe virtual functions.
My goal was to share the GPUs between multiple VMs without dedicating an entire card to each guest. GPU passthrough works well, but it's not ideal when several Linux and Windows VMs only need a portion of the GPU's resources.
The result is 8 concurrent VMs sharing 2 memory-modded RTX 3080 20GB cards on Proxmox, with both Linux CUDA and Windows Direct3D workloads running.
I've open-sourced the implementation and wanted to share how it works here.
Projects like DualCoder/vgpu_unlock and mbilker/vgpu_unlock-rs, along with polloloco/vgpu-proxmox, have helped enable NVIDIA vGPU on consumer Maxwell, Pascal, and Turing cards.
These approaches primarily rely on NVIDIA's older mediated device (mdev) architecture, where the host driver manages multiple virtual GPUs behind a single physical PCIe function.
Ampere and newer professional NVIDIA GPUs use a different architecture based on SR-IOV (Single Root I/O Virtualization).
With SR-IOV, the GPU exposes:
Unlike legacy mdev, these VFs are actual PCIe functions enumerated by Linux.
Consumer NVIDIA GPUs still contain SR-IOV-related capabilities in their PCIe configuration space, even though they aren't exposed through the normal capability discovery process.
The implementation has five main parts.
PCIe devices have a configuration space, a region of registers that the operating system uses to discover and configure device capabilities. An offset is a location within that space, measured in bytes from its beginning.
Normally, Linux discovers capabilities by following a linked list of entries in PCIe configuration space.
On Ampere, the SR-IOV extended capability exists at offset 0xBCC, but isn't linked into that list, so Linux doesn't discover it.
So, I wrote a small Linux kernel livepatch module that makes the PCI subsystem discover this hidden capability on consumer GPUs. The module also handles related PCIe features:
0xBCC, allowing Linux to recognize the available VFs.0xC14 and enables ARI forwarding on the upstream bridge when supported.Since Linux discovers PCIe capabilities during device enumeration, geforce-sriov performs a controlled PCI remove/rescan before NVIDIA's driver binds to the GPU. After the rescan, Linux recognizes the GPU's SR-IOV capability.
Great, with SR-IOV exposed, the next step is getting NVIDIA's vGPU stack to use it.
Once Linux recognizes SR-IOV, geforce-sriov uses NVIDIA's own sriov-manage utility to enable the VFs.
On Ampere, the SR-IOV capability reports support for 16 VFs per GPU, but I can currently get only 4 VFs working. The limitation appears to be related to ARI (Alternative Routing-ID Interpretation), which I'll get into later.
For example:
03:00.0 Physical GPU (PF)
03:00.4 Virtual Function 1
03:00.5 Virtual Function 2
03:00.6 Virtual Function 3
03:00.7 Virtual Function 4These are real SR-IOV virtual functions exposed by the GPU and enumerated as separate PCIe functions, rather than software-mediated devices managed through the legacy mdev approach.
Now we need to make NVIDIA's vGPU driver accept the card. By default, NVIDIA's vGPU software doesn't recognize consumer GeForce GPUs like my RTX 3080s as supported devices.
My solution uses a small userspace identity hook loaded into nvidia-vgpud. The hook presents my RTX 3080 as an RTX A6000, its professional counterpart based on the same GA102 die, allowing NVIDIA's daemon to recognize the card and register the corresponding vGPU profiles.
Once the profiles are registered, we can choose which vGPU type to assign to each VM. These profiles are defined by NVIDIA and determine the resources available to the guest, including its framebuffer allocation.
The RTX A6000 has 23 profiles in the tested driver version. A few examples:
RTXA6000-2Q - 2GB VRAMRTXA6000-4Q - 4GB VRAMRTXA6000-8Q - 8GB VRAM (7GB usable framebuffer)RTXA6000-48Q - 48GB VRAM (45GB usable framebuffer)For my setup, I used RTXA6000-4Q, allocating 4GB per VM. With 4 VFs per GPU, that's 16GB allocated across four guests on each 20GB card.
Note: Workstation and datacenter GPUs typically have more VRAM than their consumer counterparts. For example, the RTX A6000 has 48GB, while my modified RTX 3080 has 20GB. The available profiles aren't automatically restricted to the physical GPU's memory capacity. Four 4Q profiles fit within 20GB, but four 8Q profiles would require 32GB. You need to account for the combined allocations and driver overhead when choosing profiles.
This was one of the biggest obstacles in the project and required quite a bit of reverse engineering of NVIDIA's proprietary vGPU host driver.
With SR-IOV exposed, the VFs created, and the vGPU profiles registered, the guest VMs could boot and recognize their vGPUs. However, the first actual CUDA workload would consistently hang, with no useful errors in the host logs.
After several hours of tracing through the driver with the help of AI, I narrowed the issue down to the GPU's graphics runlist scheduler being disabled. A callback handling a PMU (Power Management Unit) interrupt was repeatedly disabling the scheduler, which prevented guest workloads from executing. I still don't know exactly why NVIDIA's driver disables the scheduler in this situation.
The workaround is a two-byte driver patch that replaces a conditional jump with NOP instructions, making the callback skip the scheduler-disabling call.
geforce-sriov-patch can automatically apply the change to a copy of the user's own NVIDIA driver object. The patched object is then used to rebuild nvidia.ko. With the patch, both CUDA and D3D11 workloads ran successfully.
A few things to clarify about geforce-sriov:
You still need to obtain NVIDIA's vGPU host and guest drivers and licensing yourself to use this project.
| Component | Configuration |
|---|---|
| Hypervisor | Proxmox VE |
| Kernel | 7.0.14-16-pve |
| CPU | Intel Xeon E5 v4 |
| GPUs | 2x memory-modded RTX 3080 20GB (GA102) |
| PCI device ID | 10de:2206 |
| NVIDIA host driver | vGPU-KVM 580.159.01 |
| Virtual functions | 4 per GPU, 8 total |
| Guests | 4 Linux + 4 Windows 11 |
| vGPU profile | RTXA6000-4Q (4GB per VM) |
Each physical card runs two Linux guests and two Windows guests.
On Proxmox, a VF can be assigned to a VM using:
hostpci0: 0000:03:00.4,mdev=nvidia-527Note: Proxmox still uses the mdev= parameter to select the vGPU profile, even though the underlying GPU uses SR-IOV virtual functions rather than legacy mdev sharing.
I ran all eight VMs simultaneously with:
The test ran for 30 minutes without any observed failures. This isn't enough to establish long-term stability, but it verifies that multiple independent Linux and Windows GPU workloads can run concurrently.
I ran the same PyTorch benchmarks in two configurations to measure virtualization overhead with a single active vGPU:
RTXA6000-4Q profile on the same card.Both configurations used the same Debian root filesystem, PyTorch environment, four pinned CPU cores, and 8GB of system RAM.
The workloads were FP32 and FP16 matrix multiplication using 4096 × 4096 matrices, plus a 1GB device memory copy.
| Benchmark | Physical GPU (LXC) | vGPU (VM) |
|---|---|---|
| FP32 matmul | 20.47 TFLOPS | 20.54 TFLOPS |
| FP16 matmul (Tensor Cores) | 58.35 TFLOPS | 58.54 TFLOPS |
| GPU memory copy | 681.7 GB/s | 704.4 GB/s |
The FP32 and FP16 results were within 1% of direct physical-GPU access, suggesting very little virtualization overhead for these particular workloads.
Some qualifications:
The full methodology and screenshots are available here:
https://github.com/noon-at-cgn/geforce-sriov/blob/main/docs/showcase.md
The biggest unresolved issue is ARI (Alternative Routing-ID Interpretation), which is currently limiting me to 4 VFs per GPU.
Normally, PCIe uses a 3-bit function number, allowing up to 2^3 = 8 functions (0–7) per device. ARI repurposes the 5-bit device number field to extend the function number to 8 bits, allowing up to 2^8 = 256 functions on a single device.
This matters for SR-IOV because a GPU may need to expose many VFs. On my RTX 3080s, the SR-IOV capability reports support for 16 VFs.
Without ARI, the function layout looks like this:
Function 0 GPU (Physical Function)
Function 1 GPU HDMI/DP Audio
Function 2 Unused
Function 3 Unused
Function 4 VF 1
Function 5 VF 2
Function 6 VF 3
Function 7 VF 4
------------- 3-bit function limit
Function 8 VF 5 (requires ARI)
Function 9 VF 6 (requires ARI)
...
Function 19 VF 16 (requires ARI)So although conventional PCIe allows eight functions, only four are available for VFs because the GPU's SR-IOV configuration places the first VF at function 4. Functions 0–3 aren't part of the VF allocation.
To expose all 16 VFs on the same device, the GPU needs to support function numbers beyond 7, which needs ARI.
So far, I've managed to:
0xC14.However, the GPU doesn't appear to correctly decode extended function numbers (8 and above), even with ARI enabled in software.
For now, the working configuration is limited to 4 VFs per card, despite the hardware reporting support for 16.
This is still an experimental project, not a general-purpose GeForce vGPU solution yet.
Verified so far:
10de:2206, subsystem 10de:146d).7.0.14-16-pve.580.159.01.Not yet verified:
Other practical limitations:
I would like to experiment more with both generations, but I don't currently have access to Ada or Blackwell GPUs for testing.
Disclaimer: geforce-sriov is an experimental, unofficial project and is not affiliated with or endorsed by NVIDIA. Use it at your own risk, preferably on a non-production system.
The full source code, installation instructions, technical documentation, and benchmarks are available:
GitHub: https://github.com/noon-at-cgn/geforce-sriov
Compatibility reports are welcome as compatibility remains largely unverified beyond my tested setup.
r/VFIO • u/Numerous-Club-1636 • 19d ago
These are all my specs for any necessary reference:
Detailed below is the issue:
In my QEMU setup, I have my GPU passthroughed into the VM, but when I try and inizialize the VM, this error occours:
Error starting domain: internal error: QEMU unexpectedly closed the monitor (vm='win10'): 2026-11-02T22:38:58.440650Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:05:00.0: failed to setup container for group 16: Failed to set group container: Invalid argument
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 67, in cb_wrapper
callback(asyncjob, *args, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb
callback(*args, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
ret = fn(self, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/domain.py", line 1446, in startup
self._backend.create()
~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib64/python3.14/site-packages/libvirt.py", line 1395, in create
raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: QEMU unexpectedly closed the monitor (vm='win10'): 2026-11-02T22:38:58.440650Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:05:00.0: failed to setup container for group 16: Failed to set group container: Invalid argumentThis is the GREP for IOMMU at boot/VM initialization:
[ 0.379018] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.397334] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 15.468791] amdgpu 0000:75:00.0: AMD-Vi: PerfOpt armed on IOMMU0
[ 51.116905] pcieport 0000:04:00.0: Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping
. Contact your platform vendor.Detailed below is my setup fore GRUB, IOMMU and relevent systems to make this feature function:
GRUB Config:
GRUB_DEFAULT='6'
GRUB_DISABLE_RECOVERY='true'
GRUB_DISABLE_SUBMENU='true'
GRUB_ENABLE_BLSCFG='true'
GRUB_TERMINAL_OUTPUT='console'
GRUB_TIMEOUT='10'
GRUB_CMDLINE_LINUX_DEFAULT='splash amd_iommu=on iommu=pt amd_iommu=relax_unity'
GRUB_DISTRIBUTOR='Nobara Linux'
GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt amd_iommu=relax_unity"
GRUB_THEME="/usr/share/grub/themes/sleek/theme.txt"This is my IOMMU Groups: (IOMMU Group 16 has the devices that I need to passthrough)
IOMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 10:
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
IOMMU Group 11:
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0 [1022:14e0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1 [1022:14e1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2 [1022:14e2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3 [1022:14e3]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4 [1022:14e4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5 [1022:14e5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6 [1022:14e6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7 [1022:14e7]
IOMMU Group 13:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
IOMMU Group 14:
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 15:
03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
IOMMU Group 16:
04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 17:
04:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 18:
04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 0c)
IOMMU Group 19:
04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
08:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8851BE PCIe 802.11ax Wireless Network Controller [10ec:b851]
IOMMU Group 2:
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 20:
04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 21:
04:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 22:
04:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 23:
04:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 24:
04:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 25:
04:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
0e:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. OM3PDP3 PCIe 3 NVMe SSD (DRAM-less) [2646:500d] (rev 01)
IOMMU Group 26:
04:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
0f:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 800 Series Chipset USB 3.x XHCI Controller [1022:43fc] (rev 01)
IOMMU Group 27:
04:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 28:
11:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Upstream Port [1b21:2421] (rev 01)
IOMMU Group 29:
12:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
IOMMU Group 3:
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 30:
12:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
IOMMU Group 31:
12:02.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
73:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM4242 USB 3.2 xHCI Controller [1b21:2426] (rev 01)
IOMMU Group 32:
12:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
74:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM4242 USB 4 / Thunderbolt 3 Host Router [1b21:2425] (rev 01)
IOMMU Group 33:
75:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon 610M] [1002:13c0] (rev cb)
IOMMU Group 34:
75:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 35:
75:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP [1022:1649]
IOMMU Group 36:
75:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b6]
IOMMU Group 37:
75:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b7]
IOMMU Group 38:
75:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Ryzen HD Audio Controller [1022:15e3]
IOMMU Group 39:
76:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI [1022:15b8]
IOMMU Group 4:
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 5:
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 6:
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 7:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 8:
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 9:
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]My VFIO Config file:
options vfio-pci ids=10de:1c82,10de:0fb9
softdep nvidia pre: vfio-pci
softdep nouveau pre: vfio-pci———————————————————————————————————————————–
I have seen suggestions to turn of Pre-Kernel DMA Protection, which I would not like to do if possible because I would like to maintain security and I still want to be able to play certain anti-cheat games on Windows.
All suggestions are greatly appreciated.