GPU troubleshooting: nvidia-smi errors, CUDA mismatches and memory

When the GPU does not work, the error message is usually precise. Start with the check below, then find your message.

Step 1: Collect the basics

lspci -nn | grep -iE "vga|3d|display"
uname -r
nvidia-smi
dmesg | grep -iE "nvidia|amdgpu|nouveau" | tail -30

On AMD servers, run rocminfo instead of nvidia-smi. Keep the output; it also helps us if you open a ticket.

"NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver"

The NVIDIA kernel module is not loaded. Check, in this order:

1. Was the server restarted after the driver install? Reboot.
2. Is the module there? lsmod | grep nvidia. If not, and you installed a DKMS driver, check dkms status for a failed build; install the headers for the running kernel (linux-headers-$(uname -r)) and reinstall the driver.
3. Secure Boot. mokutil --sb-state shows whether it is on. DKMS-built modules must be signed with an enrolled machine owner key (MOK). On Ubuntu, use ubuntu-drivers, which installs pre-built signed drivers.
4. nouveau holds the GPU. If lsmod | grep nouveau shows the open-source driver, the NVIDIA driver installation did not finish; reinstall it and reboot.

"Failed to initialize NVML: Driver/library version mismatch"

The driver packages were updated, but the old kernel module is still running. Reboot. If it persists, packages from two driver sources are mixed; see install NVIDIA drivers and CUDA.

A shared vGPU does not work: "No devices were found" or Code 43

On a shared vGPU, the driver inside your server must be the vGPU guest driver that matches the vGPU software on the host. Our images ship with it. If a standard consumer or desktop driver, or another driver version, was installed over it, NVIDIA states that the vGPU fails to load:

• on Linux, nvidia-smi reports No devices were found;
• on Windows, Device Manager shows the GPU with "Windows has stopped this device because it has reported problems. (Code 43)".

To fix it, reinstall the matching guest driver, using the driver version listed in your service details or provided by HyperDC support, or reinstall the operating system image from the client area. Do not install drivers from your distribution or the GPU maker's consumer downloads on a shared vGPU.

Verify: nvidia-smi lists the vGPU, or Device Manager shows the GPU without an error.

vGPU features become limited after a while

If a shared vGPU works at first but its features or performance become limited after some time, contact support. HyperDC manages the vGPU host software.

A dedicated GPU shows Code 43 on Windows

Install the GPU maker's current data-center driver for the card and Windows Server version (NVIDIA, Intel Data Center GPU or AMD), then restart. Microsoft's guidance for GPUs assigned to a virtual machine is the same: install the vendor's supported driver and check the device state in Device Manager. If the error stays, open a ticket.

"CUDA driver version is insufficient for CUDA runtime version"

The application (or container image) was built for a newer CUDA than your driver supports. nvidia-smi shows the highest CUDA version the driver supports in its header. Update the driver, or use an image built for an older CUDA version.

"CUDA out of memory"

The GPU's memory is full.

• See what uses it: the process list at the bottom of nvidia-smi.
• Stop processes you do not need, or limit how many workers or models run at once.
• Use a smaller or quantized model, a shorter context or a smaller batch.
• A shared vGPU has a fixed share of memory; a dedicated GPU or a GPU dedicated server gives you all of it.

The GPU is idle while the job runs

The application runs on the CPU. Check that it was installed with GPU support (for example a CUDA or ROCm build of PyTorch), and in containers that the GPU was passed in; see GPUs in Docker containers.

The GPU slows down under load

Check the performance state and why clocks are reduced:

nvidia-smi -q -d PERFORMANCE,TEMPERATURE

If the output reports thermal or power limits for long periods on a dedicated GPU, open a ticket with this output.

AMD: rocminfo shows no GPU

• Your user must be in the render and video groups (groups), and you must log in again after adding them.
• The amdgpu-dkms module must be built for the running kernel; check dkms status and reboot.
• The GPU must be on AMD's ROCm compatibility list for your ROCm release.

See install AMD ROCm.

Next steps

• Back to the start: GPU servers: getting started.
• Containers: GPUs in Docker containers.