> ## Documentation Index > Fetch the complete documentation index at: https://docs.siderolabs.com/llms.txt > Use this file to discover all available pages before exploring further. # NVIDIA GPU (OSS drivers) > In this guide you'll follow the procedure to support NVIDIA GPU using OSS drivers on Talos. > Enabling NVIDIA GPU support on Talos is bound by [NVIDIA EULA](https://www.nvidia.com/en-us/drivers/nvidia-license/). > The Talos published NVIDIA OSS drivers are bound to a specific Talos release. > The extensions versions also needs to be updated when upgrading Talos. We will be using the following NVIDIA OSS system extensions: * `nvidia-open-gpu-kernel-modules`: the NVIDIA open kernel modules. * `nvidia-container-toolkit`: the NVIDIA driver userspace libraries and the NVIDIA persistence daemon. Despite its name, this extension no longer includes the NVIDIA Container Toolkit; the [NVIDIA GPU Operator](#deploying-nvidia-gpu-operator) installs the toolkit. Create the [boot assets](https://docs.siderolabs.com/talos/v1.15/platform-specific-installations/boot-assets.md) which includes the system extensions mentioned above (or create a custom installer and perform a machine upgrade if Talos is already installed). > Make sure the driver version matches for both the `nvidia-open-gpu-kernel-modules` and `nvidia-container-toolkit` extensions. > The `nvidia-open-gpu-kernel-modules` extension is versioned as `-` and the `nvidia-container-toolkit` extension is versioned as ``. ## Proprietary vs OSS Nvidia Driver Support The NVIDIA Linux GPU Driver contains several kernel modules: `nvidia.ko`, `nvidia-modeset.ko`, `nvidia-uvm.ko`, `nvidia-drm.ko`, and `nvidia-peermem.ko`. Two "flavors" of these kernel modules are provided, and both are available for use within Talos: * Proprietary, This is the flavor that NVIDIA has historically shipped. * Open, i.e. source-published/OSS, kernel modules that are dual licensed MIT/GPLv2. With every driver release, the source code to the open kernel modules is published on [https://github.com/NVIDIA/open-gpu-kernel-modules](https://github.com/NVIDIA/open-gpu-kernel-modules) and a tarball is provided on [https://download.nvidia.com/XFree86/](https://download.nvidia.com/XFree86/). The choice between Proprietary/OSS may be decided after referencing the Official [NVIDIA announcement](https://developer.nvidia.com/blog/nvidia-transitions-fully-towards-open-source-gpu-kernel-modules/). > [!TIP] > > Some hardware may not work with the OSS drivers without additional configuration. Grace Blackwell (GB10) devices running with arm64 CPUs like the NVIDIA DGX Spark should use the [NVIDIA proprietary drivers](https://docs.siderolabs.com/talos/v1.15/configure-your-talos-cluster/hardware-and-drivers/nvidia-gpu-proprietary.md). The `-new-feature` OSS drivers (615 or later) also work on these devices if the `nvidia_uvm` module is loaded with the `uvm_disable_hmm=1` parameter. ## Enabling the NVIDIA OSS modules Patch Talos machine configuration using the patch `gpu-worker-patch.yaml`: ```yaml theme={null} apiVersion: v1alpha1 kind: KernelModuleConfig name: nvidia --- apiVersion: v1alpha1 kind: KernelModuleConfig name: nvidia_uvm --- apiVersion: v1alpha1 kind: KernelModuleConfig name: nvidia_drm --- apiVersion: v1alpha1 kind: KernelModuleConfig name: nvidia_modeset ``` Now apply the patch to all Talos nodes in the cluster having NVIDIA GPU's installed: ```bash theme={null} talosctl patch mc --patch gpu-worker-patch.yaml ``` The NVIDIA modules should be loaded and the system extension should be installed. This can be confirmed by running: ```bash theme={null} talosctl get modules ``` which should produce an output similar to below: ```text theme={null} NODE NAMESPACE TYPE ID VERSION STATE 10.5.0.3 runtime LoadedKernelModule nvidia_uvm 1 Live 10.5.0.3 runtime LoadedKernelModule nvidia_drm 1 Live 10.5.0.3 runtime LoadedKernelModule nvidia_modeset 1 Live 10.5.0.3 runtime LoadedKernelModule nvidia 1 Live ``` ```bash theme={null} talosctl get extensions ``` which should produce an output similar to below: ```text theme={null} NODE NAMESPACE TYPE ID VERSION NAME VERSION 172.31.41.27 runtime ExtensionStatus 0 1 nvidia-container-toolkit-lts 580.178.04 172.31.41.27 runtime ExtensionStatus 1 1 nvidia-open-gpu-kernel-modules-lts 580.178.04-v1.15.0-alpha.2 ``` ## Deploying NVIDIA GPU Operator Follow the [upstream instructions](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html) with only passing Helm chart values specific to Talos. Disable the driver component of the GPU operator, because the driver is already installed as a system extension on Talos. Enable the toolkit component: the GPU operator installs the NVIDIA Container Toolkit under `/var/lib/nvidia` and registers it with containerd through [NRI](https://github.com/containerd/nri). Pods get the GPU through [CDI](https://github.com/cncf-tags/container-device-interface), which is enabled in the Talos containerd configuration, so no `nvidia` `RuntimeClass` is needed. Further custom values may be needed to be passed to the helm chart depending on the cluster configuration. ```bash theme={null} kubectl create ns gpu-operator kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged ``` > Note: Make sure to install GPU operator version v26.3.1 or higher ```bash theme={null} helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm repo update helm upgrade --wait --install -n gpu-operator gpu-operator nvidia/gpu-operator \ --set driver.enabled=false \ --set toolkit.enabled=true \ --set toolkit.installDir=/var/lib/nvidia \ --set cdi.enabled=true \ --set cdi.nriPluginEnabled=true \ --set hostPaths.driverInstallDir=/usr/local ``` ### Verification Follow the [upstream instructions](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html#verification-running-sample-gpu-applications) to verify the GPU operator installation. GPU workloads request the `nvidia.com/gpu` resource and don't set `runtimeClassName`: ```yaml theme={null} apiVersion: v1 kind: Pod metadata: name: cuda-test spec: restartPolicy: OnFailure containers: - name: cuda-test image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0 resources: limits: nvidia.com/gpu: 1 ``` A successful run ends with `Test PASSED` in the pod logs. ## Collecting NVIDIA GPU debug data When debugging NVIDIA GPU issues (for example, `NVRM: GPU has fallen off the bus` messages in the kernel log), NVIDIA support will often ask for the output of `nvidia-bug-report.sh`. Talos does not allow direct shell access on the nodes, but you can still generate this report by following the steps below: 1. Start a debug pod on the affected node: ```bash theme={null} kubectl -n kube-system \ run debug-gpu \ --rm -it \ --image=ubuntu \ --overrides='{ "spec": { "hostPID": true, "hostNetwork": true, "containers": [{ "name": "debug-gpu", "image": "ubuntu", "stdin": true, "tty": true, "securityContext": { "privileged": true }, "volumeMounts": [{ "name": "host-root", "mountPath": "/host" }] }], "volumes": [{ "name": "host-root", "hostPath": { "path": "/" } }] } }' \ --restart=Never \ -- /bin/bash ``` 2. This will drop you into a shell inside a container running on the node. From here, you can install the necessary tools to run `nvidia-bug-report.sh` and generate the report. ```bash theme={null} apt update && apt install --no-install-recommends -y \ dmidecode \ pciutils \ usbutils \ mesa-utils \ kmod \ vulkan-tools \ infiniband-diags \ acpidump \ mstflint ``` 3. Inside the debug container, run `nvidia-bug-report.sh`: ```bash theme={null} /host/usr/local/bin/nvidia-bug-report.sh ``` This will generate `nvidia-bug-report.log.gz` in the current directory. 4. To copy the report of the cluster: From your local machine, run the following command to copy the report from the debug pod to your local machine: ```bash theme={null} kubectl cp \ "kube-system/debug-gpu:/nvidia-bug-report.log.gz" \ ./nvidia-bug-report.log.gz ``` You can now upload `nvidia-bug-report.log.gz` to NVIDIA support. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.