Skip to content

M #~: Minor changes Host config #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattrowe-opennebula wants to merge 2 commits into
base: one-7.4
Choose a base branch
from
Open

M #~: Minor changes Host config #1087

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
53b7540
M #~: Minor changes Host config
mattrowe-opennebula Oct 29, 2026
3138437
M #~: Table headers
mattrowe-opennebula Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 20 additions & 18 deletions content/product/cluster_configuration/pci_passthrough_sriov/host_configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@ tags: ['AI','NVIDIA']
weight: "2"
---

This guide describes the common host configuration required to enable PCI passthrough in OpenNebula. These requirements apply independently of the type of PCI device being assigned to virtual machines. Device-specific configuration is covered in the corresponding guides.
The guide assumes a Linux host running the OpenNebula KVM hypervisor. While the examples use common Linux utilities and a GRUB-based bootloader, equivalent procedures can be used with other Linux distributions and bootloaders.
This guide describes the common Host configuration required to enable PCI passthrough in OpenNebula. These requirements apply independently of the type of PCI device being assigned to virtual machines. Device-specific configuration is covered in the corresponding guides.
The guide assumes a Linux Host running the OpenNebula KVM hypervisor. While the examples use common Linux utilities and a GRUB-based bootloader, equivalent procedures can be used with other Linux distributions and bootloaders.

## Prerequisites

Before configuring PCI passthrough, verify that the host satisfies the following requirements:
Before configuring PCI passthrough, verify that the Host satisfies the following requirements:
* Hardware-assisted virtualization enabled in the system firmware.
* An IOMMU implementation supported by the processor:
* Intel VT-d
* AMD-Vi
* Arm SMMUv3
* OpenNebula host packages installed, including KVM, QEMU, and libvirt support.
* OpenNebula Host packages installed, including KVM, QEMU, and libvirt support.

---

Expand Down Expand Up @@ -95,7 +95,7 @@ arm-smmu-v3 ...

## Step 2. Configure VFIO Device Binding

PCI devices assigned to virtual machines must be managed by a VFIO driver instead of their native host driver. OpenNebula supports any mechanism that binds devices to VFIO. The recommended approach is to use the **driverctl** utility, which provides persistent per-device driver overrides using the Linux kernel `driver_override` interface.
PCI devices assigned to virtual machines must be managed by a VFIO driver instead of their native Host driver. OpenNebula supports any mechanism that binds devices to VFIO. The recommended approach is to use the **driverctl** utility, which provides persistent per-device driver overrides using the Linux kernel `driver_override` interface.

For Linux Containers, the PCI device driver shouldn’t be changed. Skip this section entirely if using passthrough in Linux Containers.

Expand Down Expand Up @@ -165,7 +165,7 @@ driverctl set-override 0000:43:00.0 vfio-pci
Many PCI devices expose multiple functions. For example, GPUs often expose both a graphics controller and an audio controller. Repeat the command for every PCI function that will be assigned to the virtual machine. The override is persistent and is automatically restored after reboot.

{{< alert title="Note" type="primary" >}}
Once a device is bound to a VFIO driver, it is no longer available to the host operating system until the override is removed.
Once a device is bound to a VFIO driver, it is no longer available to the Host operating system until the override is removed.
{{< /alert >}}

### Verification
Expand Down Expand Up @@ -196,7 +196,7 @@ driverctl list-overrides

### Removing an Override

To restore the default host driver:
To restore the default Host driver:

```shell
driverctl unset-override 0000:43:00.0
Expand Down Expand Up @@ -258,7 +258,7 @@ hugepages=1024 default_hugepagesz=1G hugepagesz=1G

The appropriate Huge Page size depends on the processor architecture and workload. On x86 platforms, common Huge Page sizes are 2 MB and 1 GB.

Regenerate the bootloader configuration using the appropriate command for your Linux distribution and reboot the host.
Regenerate the bootloader configuration using the appropriate command for your Linux distribution and reboot the Host.

**Notes**:

Expand All @@ -281,7 +281,7 @@ grep Huge /proc/meminfo

## Step 4. Configure PCI Monitoring

OpenNebula discovers PCI devices through the host monitoring subsystem. PCI monitoring is disabled by default to avoid exposing unnecessary hardware resources.
OpenNebula discovers PCI devices through the Host monitoring subsystem. PCI monitoring is disabled by default to avoid exposing unnecessary hardware resources.

Before PCI devices can be assigned to virtual machines, configure the monitoring subsystem to discover the devices intended for passthrough.

Expand Down Expand Up @@ -339,24 +339,26 @@ device_name:
- "Virtual Function"
```

This example monitors only VFs. To [select VFs by their parent PF]({{% relref "product/cluster_configuration/pci_passthrough_sriov/network_interfaces.md#selecting-virtual-functions-by-physical-function" %}}), adjust the filters to include both the PF and its VFs. A network PF with enabled VFs remains visible for monitoring but is not eligible for whole-device passthrough.

### Override Monitoring at Cluster or Host Level

The system-wide configuration can be overridden for individual clusters or hosts using the corresponding template attributes:
The system-wide configuration can be overridden for individual Clusters or Hosts using the corresponding template attributes:

| Parameter | Description |
| **Parameter** | **Description** |
|-----------|-------------|
| PCI_FILTER | (List) Filters by PCI vendor:device:class patterns (same as for lspci) |
| PCI_SHORT_ADDRESS | (List) Filters by short PCI address bus:device.function |
| PCI_DEVICE_NAME | (List) Filters by the PCI device name reported by the Host |

This allows different clusters to expose different classes of PCI devices while sharing the same frontend configuration.
This allows different Clusters to expose different classes of PCI devices while sharing the same frontend configuration.

{{< alert title="Recommendation" type="primary" >}}
Configure the narrowest possible filters to expose only devices intended for passthrough.{{< /alert >}}

### Synchronize the Monitoring Probes

After modifying `pci.conf`, synchronize the updated monitoring probes to all hosts (run as oneadmin):
After modifying `pci.conf`, synchronize the updated monitoring probes to all Hosts (run as oneadmin):

```shell
onehost sync --force
Expand Down Expand Up @@ -388,13 +390,13 @@ Verify that:
* Driver information is correct.
* NUMA information is reported.
* SR-IOV Virtual Functions are detected, when applicable.
* Device availability matches the host configuration.
* Device availability matches the Host configuration.

If a device does not appear:

* Verify that it matches the configured monitoring filters.
* Verify that pci.conf has been synchronized to the hosts.
* Verify that the host has completed a monitoring cycle.
* Verify that pci.conf has been synchronized to the Hosts.
* Verify that the Host has completed a monitoring cycle.
* Verify that the device is visible using lspci.
* Verify that the device is bound to the appropriate VFIO driver.
* Verify that the OpenNebula monitoring service is running.
Expand All @@ -411,11 +413,11 @@ Before deploying virtual machines using PCI passthrough, verify that the followi
* &#10004; PCI monitoring configured
* &#10004; PCI devices discovered by OpenNebula

Once these checks are complete, the host is ready to assign PCI devices to virtual machines.
Once these checks are complete, the Host is ready to assign PCI devices to virtual machines.

## Next Steps

After completing the host configuration:
After completing the Host configuration:

* Continue with the [Generic PCI Devices Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/generic_devices/" %}}) to assign storage controllers, USB controllers, audio devices, and other generic PCI devices.
* Continue with the [Network Interfaces Guide]({{% relref "product/cluster_configuration/pci_passthrough_sriov/network_interfaces/" %}}) to configure PCI passthrough or SR-IOV network devices.
Expand Down
Loading