1

I run the distributed training code in the pytorch tutorial, when I use the 'gloo' backend, the code raises RuntimeError when init_process_group.

"""run.py:"""
#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    """ Distributed function to be implemented later. """
    pass

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)


if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

While getting the error:

Process Process-2:
Traceback (most recent call last):
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-1-6cc8a94a6551>", line 16, in init_process
    dist.init_process_group(backend, rank=rank, world_size=size)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 416, in init_process_group
    timeout=timeout)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 484, in _new_process_group_helper
    timeout=timeout)
RuntimeError: [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/device.cc:198] ifa != nullptr. Unable to find interface for: [0.0.0.27]
Process Process-1:
Traceback (most recent call last):
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-1-6cc8a94a6551>", line 16, in init_process
    dist.init_process_group(backend, rank=rank, world_size=size)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 416, in init_process_group
    timeout=timeout)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 484, in _new_process_group_helper
    timeout=timeout)
RuntimeError: [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/device.cc:198] ifa != nullptr. Unable to find interface for: [0.0.0.27]

If I change the backbone from 'gloo' to 'NCCL', the code runs correctly.

weleen
  • 41
  • 3
  • I find on the other server, the code runs no problem. So I think there are issues in network configuration. ``` lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:471171 errors:0 dropped:0 overruns:0 frame:0 TX packets:471171 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:27339830 (27.3 MB) TX bytes:27339830 (27.3 MB) ``` – weleen Apr 19 '20 at 12:01

0 Answers0