Use your GPU in Docker containers: NVIDIA, AMD and Intel

Docker containers are isolated from the host's devices, so a GPU must be passed in on purpose. How depends on the GPU maker: NVIDIA uses the NVIDIA Container Toolkit, AMD and Intel GPUs are passed in as device files. This guide sets up both and shows the Compose syntax.

Before you start

• A GPU server with a working driver: nvidia-smi (NVIDIA) or rocminfo (AMD) answers on the host; see install NVIDIA drivers and CUDA or install AMD ROCm.
• Docker Engine from Docker's repository: install Docker on Ubuntu.

NVIDIA: install the NVIDIA Container Toolkit

Add NVIDIA's repository with its signing key, then install the toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

NVIDIA's guide pins an exact version of the four toolkit packages; installing nvidia-container-toolkit without a version installs the current one with its dependencies.

Register the NVIDIA runtime with Docker and restart Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify with NVIDIA's sample workload:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

The container prints the same nvidia-smi table as the host.

NVIDIA GPUs in Docker Compose

Reserve the GPU in the service definition:

services:
  app:
    image: ollama/ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Use count: all to give the service every GPU.

AMD: pass the ROCm devices

AMD GPUs need two device paths: /dev/kfd (the compute interface) and /dev/dri (one node per GPU). AMD's example:

docker run -it --rm --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined rocm/rocm-terminal bash

Inside the container, rocminfo lists the GPU. --security-opt seccomp=unconfined is optional; AMD recommends it for HPC workloads that use numactl.

In Compose:

services:
  app:
    image: rocm/rocm-terminal
    devices:
      - /dev/kfd
      - /dev/dri

Intel: pass the render devices

Intel GPUs are reached through /dev/dri. Pass it in and use an image that contains Intel's compute or media runtime for your GPU:

docker run --rm --device /dev/dri ubuntu ls -l /dev/dri

Troubleshooting

could not select device driver "" with capabilities: [[gpu]]. Docker does not know the NVIDIA runtime. Run sudo nvidia-ctk runtime configure --runtime=docker and restart Docker.

nvidia-container-cli: initialization error. The NVIDIA driver on the host is not working. Check nvidia-smi on the host first.

AMD: Permission denied on /dev/kfd. The user running the container's process lacks access to the devices. Run as root inside the container, or add the host's video and render group IDs with --group-add.

The application in the container uses the CPU. The image may not include GPU support, or it was started without the GPU flags. Check the image's documentation for the GPU tag or variant.

Next steps

• Chat with local models on your GPU: Open WebUI with Ollama.
• Something does not work: GPU troubleshooting.