0

I want to run subprocess.call (or any other subprocess function) with a cwd that is really long (longer than 260 characters). I am using a recent Windows 10.

I read here that in order to support long paths, you either have to set a registry key or add \\?\ in front of the path. I did both.

It works if the executable I want to run has a long path. But it does not work if the cwd is a long path:

import os, sys
import subprocess

PATH_TO_WRITE_EXE = r"C:\Windows\write.exe"
print(os.path.isfile(PATH_TO_WRITE_EXE))

# error:
my_cwd = "\\\\?\\C:\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path\\a\\really\\long\\path"
print(os.path.isdir(my_cwd))

# no error:
#my_cwd = "\\\\?\\C:\\a\\not\\so\\long\\path"
#print(os.path.isdir(my_cwd))

o = subprocess.call([PATH_TO_WRITE_EXE], timeout=None, cwd=my_cwd)
print(o)

Note that os.path.isdir() returns True on both the short and the long path.

How can I use a long path as cwd on Windows 10?

langlauf.io
  • 3,009
  • 2
  • 28
  • 45
  • 1
    The `lpCurrentDirectory` parameter of [`CreateProcess`](https://learn.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-createprocessw) is limited to `MAX_PATH` (260) characters, even in Windows 10 with long paths enabled. Using `NULL` to inherit a long path as the working directory also fails. Maybe this is for the sake of compatibility since they don't know whether the executable is long-path aware. They could check the executable's manifest for long-path support, so maybe they just haven't gotten around to updating `CreateProcess`. – Eryk Sun Jul 25 '18 at 18:22
  • Note that if a process is not long-path aware, [`SetCurrentDirectory`](https://learn.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-setcurrentdirectory) does not allow paths that exceed `MAX_PATH`, even with the "\\?\" prefix. The current `SetCurrentDirectory` documentation is a mess in many ways. Someone mistakenly added: "To extend this limit to 32,767 wide characters, call the Unicode version of the function and prepend "\?" to the path." That's not right and wasn't in the docs previously. They even botched the backslash escaping. – Eryk Sun Jul 25 '18 at 18:28
  • I'm not running Windows to check this out, but can you chdir() into the directory, not set the cwd and allow Windows CreateProcess to inherit the cwd of the process? – sehafoc Jul 25 '18 at 19:15
  • @sehafoc I tried to `chdir()` into the directory first, also gives the same error. Worth a try though, thanks. – langlauf.io Jul 25 '18 at 19:23
  • 1
    @stackoverflowwww rats, Only other ideas I have are not as good... If you're on py3.6 (I think 3.2 or greater maybe) you can try to create a symlink at a shorter path to your longer path and use that. Other ideas are more work, like checking through the win32 and seeing if there is a better option and then accessing the win32 lib directly – sehafoc Jul 25 '18 at 20:18
  • @sehafoc, as I already said, `SetCurrentDirectory` only allows setting a long path in a long-path aware application (e.g. Python 3.6+), and for `CreateProcess`, "[u]sing `NULL` to inherit a long path as the working directory also fails". The latter case is a mess. It fails with `ERROR_INVALID_PARAMETER` (87), but `NULL` is a valid parameter (i.e. `cwd=None` in Python), so it's not obvious what the problem is unless you know the working directory is a long path and know that this makes `CreateProcess` fail. – Eryk Sun Jul 26 '18 at 02:20
  • 1
    Creating a symbolic link requires SeCreateSymbolicLinkPrivilege, which by default is only assigned to elevated administrators. (However, it can be explicitly assigned to users and groups.) If `os.symlink` raises `OSError`, you can create a junction via `_winapi.CreateJunction` or CMD's `mklink /j` command. – Eryk Sun Jul 26 '18 at 02:36
  • @eryksun Thanks for the detailed analysis, I've added your comments to some POC code in an answer below. I couldn't find a CreateJunction win32 call, but linked another answer that detailed how to do it. The symlink seems to work fine. If you have a working example of that it would probably be a fine answer. Cheers! – sehafoc Jul 27 '18 at 19:46
  • Derp, just found it, I'm running py 3.4 haha – sehafoc Jul 27 '18 at 19:50

1 Answers1

1

This is just a proof of concept, and you will would probably want to do something different, but here is "an" answer that will work if you run your script as admin (which is a bad idea... maybe? (depending on the scope)).

import os
import win32file
import subprocess

long_path = '\\\\?\\C:\\Temp\\3d\\RsTYjcEwAA26\\aFmtI0e\\v\\ZZ7\\AWgMBtUP5\\JRGtyZXFj2\\f2rqXnYX3yJ4\\39X11fdRbYEA\\NtPySHqx\\htyDGAtZWv8NDK\\d2VRFFJPuBUVXET\\2QSlBOlMkgO8h\\mES\\sQfPZ1nBAKZNIogOb\\wyGm5Z0RwHV\\n54Si\\2BqDwGnK6TOxjs2P\\p4SnwEre4\\KQzs1NXu5QEZcuZOIct\\YrMfsGq5g5gnMN69ko\\QFIq\\J4IKjZ3vxNrC\\OVDWtz\\Jp1H0M1UclBJqeBuX\\bjN7dA\\lCFmKDg7G1\\OhYtim9AxgX9Bm9\\vrLaaL\\KLvkkJeI0ofdwb\\Es\\ZJi3Q54oIXbQ8NOi10\\VR\\HH3\\O\\5\\zn7\\7EKj96k3BC\\8Q1OqP\\FdX8RLhl1Ce\\mPG\\OtmJWbzFk\\AheYZ8Ypwo\\085dmIvlrg\\Y8tmeJt\\cDYqXPq\\G6EYcqVXaLxv\\XXq6tIfVDhv8WoF\\xM\\PCYkVfFT1Uam9N0e\\G9PfRMOv\\GUWbc6eot4aEuVQIMd\\0NMEq9iDzqgLGOJx09\\HpUN5rBfaq9\\Ve\\Tp0E\\wpXyehjLotcDa4x\\HlBy1LMD83sxzQF0\\1\\NH1be07kdb61aomggou\\D0\\SF\\n0NLPfYTEh\\3k1AooSmx4y2CS6Mrp\\sgAd9N6x1v31jZ\\hof1X6XGdBAU8\\zyzuxVDHuX54PiYW0\\nVJc8\\r\\ukx63N2kY\\6gf8dhUTYad\\L8w4JWwZq\\iixvKOcH13FXljY5D\\zgGuUlXFH1hd\\2Ykw1isPKOKXR4Osv1U\\ncmRIMWf\\i1ioae6pqcsfDsI\\AU7fhnbPCtpaOphXL\\Vxn\\gJFO1o6JAMBmBWP\\8EKwcdps\\JGd\\SgfwKrEd5\\pGSxLp\\DuA8th1\\YRx8u0LF8Cgs6JEfwA\\dIV0Ay\\PEc2\\CSli\\nyRaOzgBtLuM8S09st\\vMd9Ctvc8c6\\2\\H5tpHh\\K6TsNhH\\jXmon6\\BqvEDk\\gsMH20FxEgwlY'
file_name = "test_file"
symlink_name = "C:\\Temp\\long_link"

os.makedirs(long_path)
with open(os.path.join(long_path, file_name), "w") as file:
    file.write("I'm some test data in a long path!")
win32file.CreateSymbolicLink(symlink_name, long_path, 0x3)

subprocess.call("type %s" % file_name, shell=True, timeout=None, cwd=symlink_name)
I'm some test data in a long path!0

As @eryksun mentioned in the comments: Creating a symbolic link requires SeCreateSymbolicLinkPrivilege, which by default is only assigned to elevated administrators. (However, it can be explicitly assigned to users and groups.) If os.symlink raises OSError, you can create a junction via _winapi.CreateJunction or CMD's mklink /j command.

Finally here is another answer which should enable the same behavior if you create a junction. I have not tested this answer in conjunction with your question, but it should work.

Edit: If you're running >= Python 3.5 you can use the CreateJunction call to replace the symlink above.

import _winapi
_winapi.CreateJunction(source, target)
sehafoc
  • 866
  • 6
  • 9
  • IIRC, CMD limits strings to 8K characters prior to Windows 10. That's adequate because reparse points (junctions, symlinks, etc) are limited to about 4K characters. (The reparse buffer is limited to 16 KiB, which has to include both the DOS display path and NT substitute path.) – Eryk Sun Jul 27 '18 at 22:07
  • It's been awhile since I've tested any MS redirector or file system code, but IIRC the deep path tests were always a bit... odd. For many of the applications to take long dirs they started converting to the "short name" of a directory and even then they ran into limitations well before anything in "win32" did. It looks like the OP is AFK on this one tho, and I'm too lazy to check for the purpose of this conversation. Tho CreateSymbolicLinkA does state it will work up to a 32K target. Meaning that this code above should work with any path len. the link traversal is another question I guess. – sehafoc Jul 29 '18 at 17:35
  • 1
    `CreateSymbolicLinkW` can accept paths of up to 32K, the standard NT path limit. But file-system reparse points in particular are limited to `MAXIMUM_REPARSE_DATA_BUFFER_SIZE` (16 KiB). If no print path is set, this limits the target to about 8K characters, but `CreateSymbolicLinkW` and most others set a print path for the target, which reduces the limit to about 4K characters. This is approximately equivalent to the 4K limit on device reparse points that `DefineDosDevice` sets (e.g. as used by subst.exe to assign a path to a drive letter). – Eryk Sun Jul 29 '18 at 19:34
  • See [`FSCTL_SET_REPARSE_POINT](https://learn.microsoft.com/en-us/windows-hardware/drivers/ifs/fsctl-set-reparse-point), the file-system control that's used to set a reparse point on a file or directory. – Eryk Sun Jul 29 '18 at 19:35
  • Ah, gotcha. Well you (or I in this case) learn something new everyday! :) It seems like to get this to work right with paths greater than 4K you'd have to make a pile of relative links (though it probably doesn't matter at this point)? I'll remove the invalid assertion from the answer. Additionally I'm not sure why but the SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE Doesn't seem to work as expected on my machine. more mysteries – sehafoc Jul 29 '18 at 21:33
  • 1
    The unprivileged-create flag requires the system to be in developer mode. Also it's an invalid parameter prior to Windows 10. On my personal system I don't really care who can symlink. I grant SeCreateSymbolicLinkPrivilege to the well-known group "Authenticated Users". This also allows administrators to symlink without elevating. Problem solved. – Eryk Sun Jul 29 '18 at 22:45