Why do my containers fail to see the GPU on this server?

Most often this happens because the GPU is visible to the host, but the container runtime isn’t actually passing the device into the container.

Common causes:

1. No NVIDIA container runtime / toolkit installed

   • Docker/Podman won’t expose GPUs unless NVIDIA Container Toolkit is installed and configured.

2. Container started without GPU access

   • For Docker you usually need something like:

     docker run --gpus all ...
     

   • Without --gpus, the container won’t see the GPU.

3. Wrong container image

   • The image may not include NVIDIA user-space libraries or CUDA support.
   • Some images are CPU-only even if run on a GPU host.

4. Driver mismatch

   • The host has a driver, but the container expects different CUDA/runtime versions.
   • Usually the host driver must be new enough for the CUDA version in the container.

5. Permissions / cgroup / device access issue

   • The container may not have access to /dev/nvidia* devices.
   • Security settings, rootless Docker, or restrictive cgroups can block them.

6. GPU driver not healthy on the host

   • Check the host first:

     nvidia-smi
     

   • If that fails on the host, containers won’t work either.

Quick checks:

• On host:

  nvidia-smi
  docker info | grep -i nvidia
  

• In container:

  ls -l /dev/nvidia*
  nvidia-smi
  

If you want, I can help you debug it step-by-step if you paste:

• host OS
• container runtime (Docker/Podman/Kubernetes)
• output of nvidia-smi on the host
• the exact container run command or compose file