Proxmox GPU Passthrough

Fast summary of how to passthrough GPU in Proxmox

Your hardware should, support: VT-d, interrupt mapping, and UEFI BIOS

Tested with Proxmox VE 6.0

 Activate IOMMU

edit file /etc/default/grub and edit line GRUB_CMDLINE_LINUX_DEFAULT="quiet"

Example: (line 9 in the example)

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

For Intel CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

For AMD CPUs:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

If it's the default selected GPU in BIOS add video=efifb:off e.g. (If you don't know... add it)

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on video=efifb:off"

update grub

update-grub

If later the passthrough still fails

Add this iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off

Example:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

Add kernel modules

add to /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Unsafe interrupt remapping

run this script directly on command line:

if [ $(dmesg | grep ecap | wc -l) -eq 0 ]; then
  echo "No interrupt remapping support found"
fi

for i in $(dmesg | grep ecap | awk '{print $NF}'); do
  if [ $(( (0x$i & 0xf) >> 3 )) -ne 1 ]; then
    echo "Interrupt remapping not supported"
  fi
done

if interrupt remapping is not supported run: (If not sure add it)

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

Add also:

echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

Blacklist GPU driver

run:

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf 
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf 

Add GPU to VFIO

run lspci -v

Example output:

08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480] (rev ef) (prog-if 00 [VGA controller])
        Subsystem: XFX Pine Group Inc. Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
        Flags: bus master, fast devsel, latency 0, IRQ 10
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=2M]
        I/O ports at e000 [size=256]
        Memory at fe800000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [200] #15
        Capabilities: [270] #19
        Capabilities: [2b0] Address Translation Service (ATS)
        Capabilities: [2c0] Page Request Interface (PRI)
        Capabilities: [2d0] Process Address Space ID (PASID)
        Capabilities: [320] Latency Tolerance Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [370] L1 PM Substates
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
        Subsystem: XFX Pine Group Inc. Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]
        Flags: bus master, fast devsel, latency 0, IRQ 5
        Memory at fe860000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

In this case the GPU is 08:00.0

so we run lspci -n -s 08:00

Output example:

08:00.0 0300: 1002:67df (rev ef)
08:00.1 0403: 1002:aaf0

then we need 1002:67df  and  1002:aaf0 (Vendor:Device ID) remember to replace the id's with your own

formulate them and run:

echo "options vfio-pci ids=1002:67df,1002:aaf0" > /etc/modprobe.d/vfio.conf

update initramfs:

update-initramfs -u

Reboot for updating everything:

reboot

Windows VM configuration

Download all you need

• Windows ISO 🏴‍☠️ (I used Win 10 LTSC)
• VirtIO driver for better performance HERE

Upload them to your server using the "Upload" button

Create the VM

Click on the blue button on the top right

• General:
  • Just add a name and move on
• OS:
  • Select Windows ISO image and set Guest OS to the correct one
• System:
  • Graphics card select "none (none)" because we'll be adding a passthrough GPU
• Hard Disk:
  
  •  
  • As Bus/Device you should use "SCSI" but in my case 4k random speed where better using "SATA" as Cache "Default" is more secure in case of a power outage or crash but "writeback (unsafe)" gives the best performance. Although you may lose data in case of an accident (I use "writeback (unsafe)" because it's fast)
• CPU:
  • Add some Cores and on Type use "host"
• Memory:
  • Choose something suitable for yours system
• Network:
  • On Model "VirtIO" gives the best performance. Untick Firewall cause you wont need it on Windows
• Confirm:
  • Yes

DON'T START YOUR VM YET

Edit the file /etc/pve/qemu-server/<vmid>.conf (use your VM ID instead of <vmid>) and add theese at the bottom

machine: q35
cpu: host,hidden=1,flags=+pcid
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'

The 2° and 3° line are necessary if you have a NVIDIA GPU

Passthrough the GPU

Under your VM's Options Tab/Window change the bios to UEFI

BIOS: OMVF (UEFI)

Then press the Add button and add:

1. CD/DVD Drive: Your VirtIO ISO, it contains the drivers for VirtIO components
2. EFI disk: because you are using a UEFI BIOS
3. USB device: like your mouse and keyboard
4. PCI Device: your GPU

The first 3 of then are self-explanatory. When adding your GPU configure it like so:

All Functions: YES
Rom-Bar: YES
Primary GPU: NO
PCI-Express: YES (Visible after ticking Adavanced)

In my case UEFI didn't work and i had to revert to "SeaBIOS"

Now plug-in your monitor to your GPU hope it works

More troubleshooting

Use dmesg | tail to see if there's any error and search it with www.google.com

Also check /var/log/ for errors. I had 23GB of them before found out that i needed video=efifb:off in my grub config

On older GPU you need to setup rom file MORE INFO

Source: link1 link2 link3