Proxmox GPU Passthrough
Fast summary of how to passthrough GPU in Proxmox
Your hardware should, support: VT-d, interrupt mapping, and UEFI BIOS
Tested with Proxmox VE 6.0
Activate IOMMU
edit file /etc/default/grub
and edit line GRUB_CMDLINE_LINUX_DEFAULT="quiet"
Example: (line 9 in the example)
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true
For Intel CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
For AMD CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
If it's the default selected GPU in BIOS add video=efifb:off
e.g. (If you don't know... add it)
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"
update grub
update-grub
If later the passthrough still fails
Add this iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off
Example:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"
Add kernel modules
add to /etc/modules
:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
Unsafe interrupt remapping
run this script directly on command line:
if [ $(dmesg | grep ecap | wc -l) -eq 0 ]; then
echo "No interrupt remapping support found"
fi
for i in $(dmesg | grep ecap | awk '{print $NF}'); do
if [ $(( (0x$i & 0xf) >> 3 )) -ne 1 ]; then
echo "Interrupt remapping not supported"
fi
done
if interrupt remapping is not supported run: (If not sure add it)
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
Add also:
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
Blacklist GPU driver
run:
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
Add GPU to VFIO
run lspci -v
Example output:
08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev ef) (prog-if 00 [VGA controller])
Subsystem: XFX Pine Group Inc. Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=2M]
I/O ports at e000 [size=256]
Memory at fe800000 (32-bit, non-prefetchable) [size=256K]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] #15
Capabilities: [270] #19
Capabilities: [2b0] Address Translation Service (ATS)
Capabilities: [2c0] Page Request Interface (PRI)
Capabilities: [2d0] Process Address Space ID (PASID)
Capabilities: [320] Latency Tolerance Reporting
Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
Capabilities: [370] L1 PM Substates
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
Subsystem: XFX Pine Group Inc. Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
Flags: bus master, fast devsel, latency 0, IRQ 5
Memory at fe860000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
In this case the GPU is 08:00.0
so we run lspci -n -s 08:00
Output example:
08:00.0 0300: 1002:67df (rev ef)
08:00.1 0403: 1002:aaf0
then we need 1002:67df
and 1002:aaf0
(Vendor:Device ID) remember to replace the id's with your own
formulate them and run:
echo "options vfio-pci ids=1002:67df,1002:aaf0" > /etc/modprobe.d/vfio.conf
update initramfs:
update-initramfs -u
Reboot for updating everything:
reboot
Windows VM configuration
Download all you need
- Windows ISO 🏴☠️ (I used Win 10 LTSC)
- VirtIO driver for better performance HERE
Upload them to your server using the "Upload" button
Create the VM
Click on the blue button on the top right
-
General:
- Just add a name and move on
-
OS:
- Select Windows ISO image and set Guest OS to the correct one
-
System:
- Graphics card select "none (none)" because we'll be adding a passthrough GPU
-
Hard Disk:
- As Bus/Device you should use "SCSI" but in my case 4k random speed where better using "SATA" as Cache "Default" is more secure in case of a power outage or crash but "writeback (unsafe)" gives the best performance. Although you may lose data in case of an accident (I use "writeback (unsafe)" because it's fast)
-
CPU:
- Add some Cores and on Type use "host"
-
Memory:
- Choose something suitable for yours system
-
Network:
- On Model "VirtIO" gives the best performance. Untick Firewall cause you wont need it on Windows
-
Confirm:
- Yes
DON'T START YOUR VM YET
Edit the file /etc/pve/qemu-server/<vmid>.conf
(use your VM ID instead of <vmid>
) and add theese at the bottom
machine: q35
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
The 2° and 3° line are necessary if you have a NVIDIA GPU
Passthrough the GPU
Under your VM's Options Tab/Window change the bios to UEFI
BIOS: OMVF (UEFI)
Then press the Add button and add:
- CD/DVD Drive: Your VirtIO ISO, it contains the drivers for VirtIO components
- EFI disk: because you are using a UEFI BIOS
- USB device: like your mouse and keyboard
- PCI Device: your GPU
The first 3 of then are self-explanatory. When adding your GPU configure it like so:
All Functions: YES
Rom-Bar: YES
Primary GPU: NO
PCI-Express: YES (Visible after ticking Adavanced)
In my case UEFI didn't work and i had to revert to "SeaBIOS"
Now plug-in your monitor to your GPU hope it works
More troubleshooting
Use dmesg | tail
to see if there's any error and search it with www.google.com
Also check /var/log/
for errors. I had 23GB of them before found out that i needed video=efifb:off
in my grub config
On older GPU you need to setup rom file MORE INFO
No Comments