Long paths in Python on Ubuntu

Question

Is there a solution for paths that are too long in Python on Ubuntu? This is the error I am getting

OSError: [Errno 36] File name too long

The thread here provides a solution for Windows Long paths in Python on Windows which does not work on Ubuntu.

I am using PyTorch's SummaryWriter. This is the error I get.

Traceback (most recent call last):
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/site-packages/tensorboardX/record_writer.py", line 47, in directory_check
    factory = REGISTERED_FACTORIES[prefix]
KeyError: '/home_nfs/haziq/runs/(vanilla)(1e-3)(1337 1337)(64)(walker)(63 1024 1024 256)(264 1024 1024 63)(512 256 8)(0 0)(attention)(63 1024)(6 96)(1120 128 128)(128 1)(1152 1024 1024 63)(attention)(63 1024)(6 96)(2 32)(128 512 128)(128 1024 1)(1152 1024 1024 1) Tue Jan  5 00'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 99, in <module>
    writer = SummaryWriter(args.log_save_path+" "+datetime)
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/site-packages/tensorboardX/writer.py", line 275, in __init__
    self._get_file_writer()
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/site-packages/tensorboardX/writer.py", line 323, in _get_file_writer
    self.file_writer = FileWriter(logdir=self.logdir,
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/site-packages/tensorboardX/writer.py", line 94, in __init__
    self.event_writer = EventFileWriter(
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/site-packages/tensorboardX/event_file_writer.py", line 104, in __init__
    directory_check(self._logdir)
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/site-packages/tensorboardX/record_writer.py", line 51, in directory_check
    os.makedirs(path)
  File "/home_nfs/haziq/cenvs/hpf/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 36] File name too long: '/home_nfs/haziq/runs/(vanilla)(1e-3)(1337 1337)(64)(walker)(63 1024 1024 256)(264 1024 1024 63)(512 256 8)(0 0)(attention)(63 1024)(6 96)(1120 128 128)(128 1)(1152 1024 1024 63)(attention)(63 1024)(6 96)(2 32)(128 512 128)(128 1024 1)(1152 1024 1024 1) Tue Jan  5 00:48:53 2021'

How long is this path? What API are you using that produces this error? There are OS/file system limits; [typically the name of a single entity is limited to 255 bytes and complete paths on most POSIX APIs are limited to 4096 bytes](https://askubuntu.com/a/859953). — ShadowRanger, Dec 23 '20 at 13:41
Linux ext2/3/4 has no actual limit for the path length so this might be an issue related to the library you are using. Can you provide details about the path you are trying to open and the code/library you are using? — Cristian Ramon-Cortes, Dec 23 '20 at 13:44
You may need to double check your assumptions... the path says `/home_nfs/...` and NFS may have path limitations that your Ubuntu ext4 file system does not have. A quick web search got results starting at 255 bytes. `mount | grep /home_nfs` will tell what the file system is. [Edit] your question and add this information, or do a web search on what that file system supports. Either way, move your files --- no matter what the limit is, you're clearly hitting it. — Robert, Jan 05 '21 at 03:53

Long paths in Python on Ubuntu

0 Answers0