1

I'm having trouble with os.walk() in python 2.7.8 on Windows.

When I supply it with a 'normal' path such as "D:\Test\master" it works as expected. However when I supply it with a UNC path such as "\\?\D:\Test\master" it will report the root directory as expected but it will not drill down into the sub directories, nor will it raise an exception.

My research: I read on the help page that os.walk() accepts a function argument to handle errors. By default this argument is None so no error is reported.

I passed a simple function to print the error and received the following for every directory.

def WalkError(Error):
    raise Exception(Error)

Stack trace:

Traceback (most recent call last):
  File "Compare.py", line 988, in StartServer
    for root, dirs, files in os.walk(ROOT_DIR,True,WalkError):
  File "C:\Program Files (x86)\Python2.7.8\lib\os.py", line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "C:\Program Files (x86)\Python2.7.8\lib\os.py", line 281, in walk
    onerror(err)
  File "Compare.py", line 62, in WalkError
    raise Exception(Error)
Exception: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\D:\\Test\\master\\localization/*.*'
Darrick Herwehe
  • 3,553
  • 1
  • 21
  • 30
admin0389
  • 51
  • 5
  • 4
    You can actually post an answer to your own question. http://blog.stackoverflow.com/2011/07/its-ok-to-ask-and-answer-your-own-questions/ – Mingye Wang Apr 18 '16 at 22:18
  • The ``\\?\`` prefix isn't a UNC path. A UNC path such as `\\server\share` becomes `\\?\UNC\server\share` using the extended path prefix. This ``\\?\`` prefix bypasses normal path processing. Windows simply replaces it with the NT DOS devices prefix ``\??\``, which has the object manager search for the device in the logon session DOS devices and then the `\Global??` object directory. This means a ``\\?\`` path must be a `unicode` string, since NT paths are Unicode, and must use only backslash as the path separator since a forward slash is just a regular name character in the NT namespace. – Eryk Sun Apr 18 '16 at 22:35
  • 1
    If you use `'//?/D:/Test/master'`, it does not bypass regular path processing and the path is limited to `MAX_PATH` (260) characters. Use `u'\\\\?\\D:\\Test\\master'`. – Eryk Sun Apr 18 '16 at 22:38
  • @eryksun - This is the correct solution. My code is all in unicode, so I assumed that wasn't the issue, but it transmits path names over a socket and therefore the unicode 'format' is lost. The receiving end takes the bytes from the socket and puts them into a string of the default encoding. I take that string and use it as top to walk a directory and it fails. Converting to unicode and running the walk makes it work perfectly. - Thank you. – admin0389 Apr 18 '16 at 23:44

1 Answers1

2

Answer from the original author (originally posted as an edit to the question):

Instant update: In the process of inspecting \lib\os.py, I discovered the error stems from os.listdir(). I searched for the above error message in relation to os.listdir() and found this solution which worked for me.

It looks like if you're going to use UNC style paths with os. modules they need to Unixised (have their \ converted to /). \\\\?\\D:\\Test\\master\\ becomes //?/D:/Test/master/ (note: you no longer need to escape the \ which is handy).

This runs counter to the UNC 'spec' so be aware if you're working with other modules which respect Microsoft's UNC implementation.

(Sorry for the self-solution, I was going to close the tab but felt there was knowledge here which couldn't be found elsewhere.)

Community
  • 1
  • 1
vacip
  • 5,246
  • 2
  • 26
  • 54
  • No such problem exists for UNC paths, and the ``\\?\`` prefix is not even a UNC path. You can't use slash as a path separator in an NT path, which Python violates by appending `/*.*` in the 8-bit `str` branch. If it appended `\*.*` you'd be able to use ``\\?\`` in a path that's not a `unicode` string, but it's only documented to allow up to 260 characters since older versions of Windows decode 8-bit ANSI strings to a statically allocated buffer that imposes this limit. Anyway, the 8-bit string branch of Python's `listdir` uses a stack allocated buffer that's also limited to 260 characters. – Eryk Sun Apr 19 '16 at 16:54
  • @eryksun I don't manage this answer, I don't take credit for it, I didn't write this. It was marked for deletion in the review queue, and it seemed to have helped the OP, so I merely salvaged it. Feel free to edit it, or even better - post your own answer. If you do so, please notify me so that I can delete this one. Thanks! :) – vacip Apr 19 '16 at 17:06
  • I have related answers [here](http://stackoverflow.com/a/36237176/205580) and [here](http://stackoverflow.com/a/24064783/205580). I could add another answer to discuss using ``\\?\`` with `listdir` in Python 2, but problems only come up when you try to use it with an 8-bit `str` path, which should be deprecated even in 2.7, just like `bytes` paths are deprecated for Windows in Python 3. – Eryk Sun Apr 19 '16 at 17:57