32

I have a problem when programming in Python running under Windows. I need to work with file paths, that are longer than 256 or whatsathelimit characters. Now, I've read basically about two solutions:

  1. Use GetShortPathName from kernel32.dll and access the file in this way.

That is nice, but I cannot use it, since I need to use the paths in a way

shutil.rmtree(short_path)

where the short_path is a really short path (something like D:\tools\Eclipse) and the long paths appear in the directory itself (damn Eclipse plugins).

  1. Prepend "\\\\?\\" to the path

I haven't managed to make this work in any way. The attempt to do anything this way always result in error WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: <path here>

So my question is: How do I make the 2nd option work? I stress that I need to use it the same way as in the example in option #1.

OR

Is there any other way?

EDIT: I need the solution to work in Python 2.7

EDIT2: The question Python long filename support broken in Windows does give the answer with the 'magic prefix' and I stated that I know it in this question. The thing I do not know is HOW do I use it. I've tried to prepend that to the path but it just failed, as I've written above.

Community
  • 1
  • 1
Jiří Kantor
  • 744
  • 1
  • 5
  • 12
  • possible duplicate of [Python long filename support broken in Windows](http://stackoverflow.com/questions/1365797/python-long-filename-support-broken-in-windows). It has a different string to prepend. Have you tried using that string? – AncientSwordRage Apr 10 '15 at 09:31
  • @Pureferret Yes but the accepted answer there is exactly what I am not able to make work :D Well the string I have here is the same as the one that is there, but this one is escaped. – Jiří Kantor Apr 10 '15 at 09:32
  • Also, this is a solution using the kernal32.dll, not sure if you've seen *this* version: http://stackoverflow.com/q/11420689/1075247 – AncientSwordRage Apr 10 '15 at 09:32
  • 2
    So you've tried `"\\?\"` and not just `"\\\\?\\"` ? – AncientSwordRage Apr 10 '15 at 09:33
  • @Pureferret: 1st: Yes I have seen it and it is not what I need. If I get the long path, I am where I was again. The path is too long for Python too work with it 2nd: What? :D I am prepending \\?\ to the string, what I've written in the question is just escaped. – Jiří Kantor Apr 10 '15 at 09:36
  • Yeah I realise now that getting the right characters is difficult in the back-ticks. The problem is that the path is too long for *windows* Python does not have an issue with it. In that second link they assign it to a variable then run it. Did you follow the exact example in that question? – AncientSwordRage Apr 10 '15 at 09:51
  • 2
    One important trick : your long file name must be of `unicode` and not `str` type if you use Python 2.x, so prefix must be written like this : `u'\\\\?\\UNC\\'` – herve-guerin Sep 18 '17 at 22:27
  • According to https://stackoverflow.com/questions/36219317/pathname-too-long-to-open/36237176 the individual components in the path have still 255 character limit. – Roland Pihlakas Dec 24 '21 at 19:04

4 Answers4

19

Well it seems that, as always, I've found the answer to what's been bugging me for a week twenty minutes after I seriously ask somebody about it.

So I've found that I need to make sure two things are done correctly:

  1. The path can contain only backslashes, no forward slashes.
  2. If I want to do something like list a directory, I need to end the path with a backslash, otherwise Python will append /*.* to it, which is a forward slash, which is bad.

Hope at least someone will find this useful.

Jiří Kantor
  • 744
  • 1
  • 5
  • 12
  • 2
    @Pureferret Lol :D Thanks for the cultural enrichment, I've never heard of Rubber duck debugging before – Jiří Kantor Apr 10 '15 at 10:21
  • 2
    Given issue 2 above, it seems you're not using a `unicode` path, i.e. `u'\\'.join([ur'\\?', path.decode('mbcs')])`. Python appends `\*.*` to `unicode` paths and `/*.*` to byte string paths. By adding a trailing ``\`` it's only appending `*.*`, but you're still calling the ANSI API `FindFirstFileA`, with a limit of `MAX_PATH` characters. – Eryk Sun Apr 10 '15 at 14:42
  • Read the documentation for [`FindFirstFile`](https://msdn.microsoft.com/en-us/library/aa364418). It tells you straight away in the description of `lpFileName` that you need to use the Unicode ([W]ide character) version of the function, i.e. `FindFirstFileW` (the name is in "Requirements"). In most cases Python uses the Windows wide-character API when passed a `unicode` argument, especially in the os module. – Eryk Sun Apr 10 '15 at 14:50
  • @eryksun: the fact that it works for the OP provided he puts the backslash at the end of the path strongly suggests that Python itself calls the Unicode API regardless of what sort of string is used. Presumably, Python converts the narrow-character string into a wide-character string and then calls FindFirstFileW, perhaps in order to ensure that the conversion occurs according to Python's rules rather than Windows? – Harry Johnston Apr 11 '15 at 05:01
  • @HarryJohnston, see [`posix_listdir`](https://hg.python.org/cpython/file/648dcafa7e5f/Modules/posixmodule.c#l2234) in Modules/posixmodule.c (lines 2239-2389). It first parses the argument tuple as a [U]nicode string object. If successful, it allocates `wnamebuf` and copies the string, appends `\*.*` or `*.*`, and calls `FindFirstFileW`. Otherwise if the arg is an `str` bye string, it copies to `char namebuf[MAX_PATH+5]`, appends `/*.*` or `*.*` (forward slash), and calls `FindFirstFileA`. If the byte string is too long, parsing the arg will raise an exception in Python. – Eryk Sun Apr 11 '15 at 05:37
  • 4
    @HarryJohnston, in case you're unfamiliar with Python 2.x, string literals are `str` byte strings unless prefixed with `u` to create a `unicode` instance. Concatenating `str` and `unicode` uses the default ASCII encoding to decode the `str` instance, so if `short_path` is `unicode`, prepending `'\\\\?\\'` results in `unicode`; otherwise it creates a new `str`. The OP should prepend `u'\\\\?\\'`. Also, read [parsing arguments and building values](https://docs.python.org/2/c-api/arg.html) to understand what's happing with the `PyArg_ParseTuple` calls in `posix_listdir`. – Eryk Sun Apr 11 '15 at 05:43
  • @eryksun: are you certain that the OP is using `posix_listdir` ? The behaviour he describes cannot be explained by the implementation you describe. – Harry Johnston Apr 11 '15 at 05:47
  • @HarryJohnston. the OP says "do something like list a directory". For that you call `os.listdir`, and `os.listdir.__module__ is nt`. For the Windows build `posixmodule.c` is compiled as a built-in module [named "nt"](https://hg.python.org/cpython/file/648dcafa7e5f/Modules/posixmodule.c#l9395) (line 9397). – Eryk Sun Apr 11 '15 at 05:54
  • 3
    Just to add my experience, I had the same problem as OP. A trailing slash didn't help me but converting to a unicode type string did. – Ben Aaronson Sep 18 '17 at 10:24
  • Downvoting because this answer does not seem to answer the original question. It seems like the correct application of `'\\\\?\\'` is the true solution. Correct me if I'm wrong. – Dr_Zaszuś Jun 15 '21 at 15:51
14

Let me just simplify this for anyone looking for a straight answer:

  1. For python < 3: Path needs to be unicode, prepend string with u like u'C:\\path\\to\\file'
  2. Path needs to start with \\\\?\\ (which is escaped into \\?\) like u'\\\\?\\C:\\path\\to\\file'
  3. No forward slashes only backslashes: / --> \\
  4. It has to be an absolute path; it does not work for relative paths
Roelant
  • 4,508
  • 1
  • 32
  • 62
Viktor Tóth
  • 423
  • 6
  • 12
  • 2
    Note: Item #1 is specific to Python 2, which was no longer supported at the time you wrote this answer. In Python 3, it just needs to be `str` (requires no special prefix to make a `str` literal). – ShadowRanger Dec 23 '20 at 13:40
2

py 3.8.2

# Fix long path access:
import ntpath
ntpath.realpath = ntpath.abspath
# Fix long path access.

In my case, this solved the problem of running a script from a long path. (https://developers.google.com/drive/api/v3/quickstart/python) But this is not a universal fix. It looks like the ntpath.realpath implementation has problems. This code replaced it with a dummy.

uDev
  • 31
  • 3
  • 8
    While this code may resolve the OP's issue, it is best to include an explanation as to how your code addresses the OP's issue. In this way, future visitors can learn from your post, and apply it to their own code. SO is not a coding service, but a resource for knowledge. Also, high quality, complete answers are more likely to be upvoted. These features, along with the requirement that all posts are self-contained, are some of the strengths of SO as a platform, differentiating it from forums. You can edit to add info &/or supplement your explanation with quotes and links to source documentation – SherylHohman May 21 '20 at 16:31
2

it works for me

import os
str1=r"C:\Users\manual\demodfadsfljdskfjslkdsjfklaj\inner-2djfklsdfjsdklfj\inner3fadsfksdfjdklsfjksdgjl\inner4dfhasdjfhsdjfskfklsjdkjfleioreirueewdsfksdmv\anotherInnerfolder4aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\5qbbbbbbbbbbbccccccccccccccccccccccccsssssssssssssssss\tmp.txt"
print(len(str1)) #346

path = os.path.abspath(str1)

if path.startswith(u"\\\\"):
    path=u"\\\\?\\UNC\\"+path[2:]
else:
    path=u"\\\\?\\"+path

with open(path,"r+") as f:
    print(f.readline())

if you get a long path(more then 258 char) issue in windows then try this .

Sandeep Makwana
  • 31
  • 2