KVM : GPU Passthrough2019/10/09

	Configure GPU Passthrough for Virtual Machines. By this configuration, it's possible to use GPU on Virtual Machines and run GPU Computing by CUDA, Machine learning/Deep Learning by TensorFlow and so on. Before configuration, Enable VT-d (Intel) or AMD IOMMU (AMD) on BIOS Setting first.
[1]	Enable IOMMU feature and [vfio-pci] kernel module on KVM Host.

[root@dlp ~]#

vi /etc/default/grub

# line 6: add (for AMD CPU, set [amd_iommu=on])

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet

intel_iommu=on

"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@dlp ~]#

grub2-mkconfig -o /boot/grub2/grub.cfg

# show PCI identification number and [vendor-ID:device-ID] of Graphic card

# PCI number ⇒ it matchs [02:00.*] below

# vendor-ID:device-ID ⇒ it matchs [10de:***] below

[root@dlp ~]#

lspci -nn | grep -i nvidia

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

[root@dlp ~]#

vi /etc/modprobe.d/vfio.conf

# create new : for [ids=***], specify [vendor-ID:device-ID]

options vfio-pci ids=10de:1b81,10de:10f0

[root@dlp ~]#

echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

[root@dlp ~]#

reboot

# confirm IOMMU is enabled

[root@dlp ~]#

dmesg | grep -E "DMAR|IOMMU"

[    0.000000] ACPI: DMAR 0x00000000DC44CC70 0000BC (v01 A M I  OEMDMAR  00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.000000] DMAR: Host address width 46
[    0.000000] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1
[    0.000000] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.000000] DMAR: RMRR base: 0x000000dc315000 end: 0x000000dc321fff
[    0.000000] DMAR: ATSR flags: 0x0
[    0.000000] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0
[    0.000000] DMAR-IR: IOAPIC id 0 under DRHD base  0xfbffc000 IOMMU 0
[    0.000000] DMAR-IR: IOAPIC id 2 under DRHD base  0xfbffc000 IOMMU 0
[    0.000000] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000
[    0.000000] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.000000] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.238032] DMAR: dmar0: Using Queued invalidation
.....
.....

# confirm vfio_pci is enabled

[root@dlp ~]#

dmesg | grep -i vfio

[    4.783160] VFIO - User Level meta-driver version: 0.3
[    4.794903] vfio_pci: add [10de:1b81[ffff:ffff]] class 0x000000/00000000
[    4.807042] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000

[2]	That's OK. For example, create a CentOS 8 Virtual machine with GPU. Specify PCI identification number of GPU for [--host-device].

[root@dlp ~]#

virt-install \
--name centos8 \
--ram 8192 \
--disk path=/var/kvm/images/centos8.img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant rhel8.0 \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--location 'https://ftp.riken.jp/Linux/centos/8/BaseOS/x86_64/os/' \
--extra-args 'console=ttyS0,115200n8 serial' \
--host-device 01:00.0 \
--features kvm_hidden=on \
--machine q35 \
--boot uefi

[3]	After creating Virtual machine, Confirm GPU is found on it like follows. For next steps, Install NVIDIA Graphic Driver and then Install GPGPU Platform CUDA or other GPGPU applications to process GPU computings.

[root@localhost ~]#

lspci | grep -i nvidia

05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)

Matched Content