-1

I have the following issue.

I am using Jupyter DockerSpawner with GPU. When I log in as a user and run nvidia-smi I get the output I am expecting. However, when another user logs in Failed to initialize NVML: Unknown Error and another user gets expected output using nvidia-smi. However, after couple of hours, both users get the same error after running nvidia-smi.

I took a look at this question: Failed to initialize NVML: Unknown Error in Docker after Few hours but it didn't work for me.

EDIT: My jupyterhub_config:

c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.remove_containers = True
c.DockerSpawner.extra_create_kwargs = {'user': 'root'}
c.DockerSpawner.extra_host_config = {'runtime': 'nvidia'}
c.Spawner.environment = {'GRANT_SUDO': 'yes'}
# c.Spawner.args = ['--allow-root']
c.Spawner.cmd = ['start-notebook.sh', '--allow-root']


def create_dir_hook(spawner):
    username = spawner.user.name  # get the username
    volume_path = os.path.join('/home/', username)
    if not os.path.exists(volume_path):
        os.mkdir(volume_path, 0o755)
    spawner.environment = {
        "NB_USER": username,
        "CHOWN_HOME": "yes",
        "CHOWN_HOME_OPTS": "-R"
    }
c.Spawner.pre_spawn_hook = create_dir_hook
eemamedo
  • 325
  • 1
  • 6
  • 14

1 Answers1

0

Found a solution. Convert

c.DockerSpawner.extra_host_config = {'runtime': 'nvidia'}

to

c.DockerSpawner.extra_host_config = {'runtime': 'nvidia', 'privileged': True}
eemamedo
  • 325
  • 1
  • 6
  • 14