How do I fix CUDA version conflicts across containers?

CUDA version conflicts across containers usually come from one of two things:

1. The container’s CUDA runtime/toolkit doesn’t match the host GPU driver
2. Different containers expect different CUDA versions, but they’re sharing the same host driver/GPU stack

A key rule:

• The host NVIDIA driver must be new enough for the CUDA version used inside the container
• Containers can bring their own CUDA runtime, but they share the host kernel driver

Quick checklist

1) Check the host driver

On the host:

nvidia-smi

Look at:

• Driver version
• Supported CUDA version shown by nvidia-smi

If the driver is too old for the container’s CUDA version, you’ll get failures like:

• CUDA driver version is insufficient for CUDA runtime version
• failed to initialize NVML
• libcudart.so version mismatch

2) Check the container CUDA version

Inside the container:

nvcc --version

or:

cat /usr/local/cuda/version.json

or inspect installed packages.

3) Make sure the NVIDIA container runtime is set up correctly

Use --gpus with Docker:

docker run --gpus all ...

And ensure the NVIDIA Container Toolkit is installed on the host.

---

Common fixes

Fix A: Upgrade the host NVIDIA driver

If containers need a newer CUDA version than the host supports, update the host driver.

This is often the best fix if:

• You manage the host
• You want multiple containers using newer CUDA images

Fix B: Use a container image with an older CUDA version

If you can’t change the host driver, use a CUDA image compatible with it.

For example:

• Host driver supports CUDA 11.8
• Don’t use a container built for CUDA 12.4

Use an image like:

FROM nvidia/cuda:11.8.0-runtime-ubuntu22.04

Fix C: Use separate images per workload

If different apps need different CUDA versions, build/tag separate images:

• myapp:cuda11.8
• myapp:cuda12.2

This avoids mixing incompatible dependencies.

Fix D: Avoid installing a full CUDA toolkit unless needed

If your app only needs runtime libraries, prefer:

• runtime
• base
• devel only when compiling

Example:

• nvidia/cuda:12.3.2-runtime-ubuntu22.04 for running
• nvidia/cuda:12.3.2-devel-ubuntu22.04 for building

Fix E: Don’t bundle conflicting CUDA libraries into the image

Sometimes conflict comes from:

• Conda packages
• pip wheels
• system libraries in /usr/local/cuda
• older libcudart, cublas, cudnn

Check LD_LIBRARY_PATH and library precedence. A wrong path can cause the container to load the wrong CUDA libs.

---

If multiple containers conflict with each other

Containers don’t normally interfere with each other through their own CUDA libs, but they do share:

• Host driver
• GPU devices
• NVIDIA runtime setup

If one container works and another doesn’t, usually the broken one has:

• A newer CUDA runtime than the host supports
• Wrong library paths
• A mismatched PyTorch/TensorFlow build

---

For Python ML stacks

If using PyTorch or TensorFlow, match framework build to CUDA version:

PyTorch example

Install a wheel built for the CUDA version you want:

• cu118 for CUDA 11.8
• cu121 for CUDA 12.1

Example:

pip install torch --index-url https://download.pytorch.org/whl/cu118

TensorFlow

Use a TensorFlow version that supports your CUDA/cuDNN combo, or use the official container image.

---

Diagnose inside the container

Run:

python -c "import torch; print(torch.version.cuda); print(torch.cuda.is_available())"

or:

python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

Also check loaded libraries:

ldd $(which python) | grep cuda

or:

find / -name 'libcudart.so*' 2>/dev/null

---

Best practices

• Keep the host driver updated
• Pin container images to a specific CUDA version
• Use official NVIDIA CUDA images when possible
• Match ML framework builds to CUDA version
• Avoid mixing multiple CUDA installs in one image
• Use nvidia-container-toolkit rather than manually mounting GPU devices

---

Rule of thumb compatibility

In practice:

• Driver version must be >= the minimum required for the container CUDA version
• Newer drivers are generally backward compatible with older CUDA runtimes

So it’s usually safer to:

• update the host driver
• keep containers pinned and reproducible

If you want, I can help you debug a specific error message or compare your host driver version and container CUDA version.