33

I have a GPU application that does unit-testing during the image building stage. With Docker 19.03, one can specify nvidia runtime with docker run --gpus all but I also need access to the gpus for docker build because I do unit-testing. How can I achieve this goal?

For older version of docker that use nvidia-docker2 it was not possible to specifiy runtime during build stage, BUT you can set the default runtime to be nvidia, and docker build works fine that way. Can I do that in Docker 19.03 that doesn't need nvidia-docker anymore? If so, how?

danny
  • 1,101
  • 1
  • 12
  • 34
  • Related to: https://stackoverflow.com/questions/70157364/docker-make-nvidia-gpus-visible-during-docker-build-process – Evandro Coan Jul 04 '22 at 14:24

3 Answers3

60

You need use nvidia-container-runtime as explained in docs: "It is also the only way to have GPU access during docker build".

Steps for Ubuntu:

  1. Install nvidia-container-runtime:

    sudo apt-get install nvidia-container-runtime

  2. Edit/create the /etc/docker/daemon.json with content:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    },
    "default-runtime": "nvidia" 
}

  1. Restart docker daemon:

    sudo systemctl restart docker

  2. Build your image (now GPU available during build):

    docker build -t my_image_name:latest .

Anton Ganichev
  • 2,184
  • 1
  • 18
  • 17
  • This is due to a syntax error (extra comma in 6-th line). Fixed now. – Anton Ganichev Jun 16 '20 at 09:12
  • 4
    FYI if you need this because you want to compile custom kernels with pytorch, you can use the nvidia development base image (e.g. nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04) and set `ENV TORCH_CUDA_ARCH_LIST=Turing `, then you can also build them without having a GPU available during build – RunOrVeith Mar 25 '21 at 09:48
  • 7
    So, literally the *only* way to get `build` to use the nvidia runtime (or I guess, any other runtime) is to set it as the *default*? Geez. How is this not an option? – juanpa.arrivillaga Sep 24 '21 at 23:01
  • If you do this and are still having issues, make sure that docker-buildx-plugins is NOT installed – Mason Aug 08 '23 at 19:05
8

A "solution" I found is to first run a base image with the host nvidia drivers mounted on it

docker run -it --rm --gpus ubuntu

And then build my app within the container manually and commit the resulting image. This is not ideal and it would be best to have access to nvidia-smi during the build phase.

danny
  • 1,101
  • 1
  • 12
  • 34
7

IMPORTANT NOTICE
(in addition to the existing answer)

Currently (march 2023), if you have docker compose installed, just configuring the default runtime may still not be enough.

In addition to configuring the default runtime, you have to disable the default docker build kit, with:

DOCKER_BUILDKIT=0 docker build <blah>

This applies even if you're not using docker compose, but it applies to docker compose as well of course.

See also:

Sam De Meyer
  • 2,031
  • 1
  • 25
  • 32
  • Thanks! you saved me! But WHY does this happen? isn't there any fix for this? – Cypher Mar 05 '23 at 15:03
  • I think this is a regression between docker 23.x and docker 20.x. https://forums.developer.nvidia.com/t/nvidia-driver-is-not-available-on-latest-docker/246265 Installing 5:20.10.24~3-0~ubuntu-focal resolves the error. As of this writing 5:23.0.4-1~ubuntu.20.04~focal seems to have the problem. Instructions for installing a specific version of docker: https://docs.docker.com/engine/install/ubuntu/ in the "Install Docker Engine". I did have to run a `sudo apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin` to have everything removed properly. – Kevin Vasko Apr 21 '23 at 13:56
  • 2
    This didn't help ... building it with DOCKER_BUILDKIT=0 or without it, during docker build the GPU is still not available. (Docker engine version 23.0.5) – Jan Apr 27 '23 at 12:37
  • same issue with me. I am trying to build deepstream container for the PC but unable build. getting /usr/bin/ld: warning: libcuda.so.1, needed by /opt/nvidia/deepstream/deepstream-6.0/lib/libnvbufsurftransform.so, not found. If I run the image in bash, and build works as I have include --gpu all in the run. would be good to make it automatic than manual. – Paul Aug 14 '23 at 09:56