Subprocess.run .vs. running command in the cmd directly

Question

I am getting a weird "Access denied - \" error\warning when I run the following script:

import os
import subprocess

directory = r'S:\ome\directory'
subprocess.run(['find', '/i', '"error"', '*.txt', '>Errors.log'], shell=True, cwd=directory)

I have also checked:

print(os.access(directory, os.R_OK))  # prints True
print(os.access(directory, os.W_OK))  # prints True

and they both print True.

The error message is printed while the subprocess command is running but the process is not killed; nothing is raised. As a result, wrapping it into a try-except even without specifying the exception is not catching anything. When the process finishes, the file is created (Error.log) but contains the wrong results.

Running the exact same command (find /i "fatal" *.txt >Error.log) from a cmd opened in the specified directory produces the correct results.

So in which way are the two approaches different?

Approach 1 (from Python):

subprocess.run(['find', '/i', '"error"', '*.txt', '>Errors.log'], shell=True, cwd=r'S:\ome\directory')

Approach 2 (from cmd):

S:\ome\directory>find /i "error" *.txt >Errors.log

FYI, `os.access` isn't very useful on Windows. It only checks the file attributes, which tells you whether a file exists and whether it's read-only. For filesystems such as NTFS that support permissions, it does nothing to check whether the user is actually permitted to read, write, or execute the file. — Eryk Sun, Jan 23 '18 at 04:58
Calling `CreateFile` with the desired access and open-existing disposition is a simple check. Normally if it fails it's either due to access denied or a sharing violation. — Eryk Sun, Jan 23 '18 at 09:26
If you instead go the route of getting the security descriptor via `GetNamedSecurityInfo` and checking via `AccessCheck`, then you may also have to check the parent directory. Specifically, read-attributes access is allowed if you can list (read) the parent, and delete is allowed if the parent allows delete-child. (The filesystem normally checks this for you.) In practice, you still may be denied read, write, or delete access due to the sharing mode of existing File objects that reference the file. — Eryk Sun, Jan 23 '18 at 09:30

score 2 · Accepted Answer · answered Jan 22 '18 at 17:53

2

I am still not sure what exactly the problem is but changing:

subprocess.run(['find', '/i', '"error"', '*.txt', '>Errors.log'], shell=True, cwd=r'S:\ome\directory')

to:

subprocess.run('find /i "error" *.txt >Errors.log', shell=True, cwd=directory)

does the trick.

As it appears, manually stitching the command works. If anybody has more info on the matter, I would be very grateful.

answered Jan 22 '18 at 17:53

Ma0

15,057
4
35
65

The redirection `>file` is a shell feature. Passing a list with `shell=True` isn't well-defined. – tripleee Jan 22 '18 at 18:01
WinAPI `CreateProcess` takes a command line string, not an `argv` array, so `Popen` has to join the list back as a string by calling `subprocess.list2cmdline`. This implements white-space quoting and quote escaping according to the rules that VC++ and `CommandLineToArgvW` use to parse the command line into an `argv` array. – Eryk Sun Jan 23 '18 at 04:47
1

You've asked for `'"error"'` as an argument, including the quotes. According to VC++ rules, a literal quote has to be escaped with a backslash, e.g. `'find /i \"error\" *.txt >Errors.log'`. However, find.exe doesn't parse its command line according to VC++ rules. It uses a C++ library named ulib.dll (used by many Windows utility programs), specifically its `ARGUMENT_LEXEMIZER` class. It parses the first backslash as one of the files to search. Obviously trying to open the root directory as a regular file isn't allowed. Then it also searches for ``error\`` in the files that match "*.txt". – Eryk Sun Jan 23 '18 at 04:49
@eryksun If you find the time to formulate an answer with all that info I would happily accept it. This is the problem I was having. Thanks a lot! – Ma0 Jan 23 '18 at 09:10

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

0

From Popen constructor (which called by subprocess.run)

On Windows, if args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows. This is because the underlying CreateProcess() operates on strings.

The problem is that CreateProcess does not support redirection.

see : How do I redirect output to a file with CreateProcess?

edited Jun 20 '20 at 09:12

Community

1
1

answered Jan 22 '18 at 18:23

napuzba

6,033
3
21
32

The redirection works but the results listed in the file `Erros.log` are not the correct. It is not the redirection that is the problem, but thanks for the comments and the link. – Ma0 Jan 23 '18 at 09:07
Yes, It seems that with shell=True , cmd.exe /c "Converted argument sequence string" is invoked instead of find.exe. The problem is the manner the argument sequence is converted to string. – napuzba Jan 23 '18 at 09:44

Sanjar Adilov · Answer 3 · 2018-01-23T10:52:48.170

0

Try to pass an output file handler for the stdout argument:

import shlex
import subprocess

with open("Errors.log", "w") as output_fh:
    # command = ['find', '/i', '"fatal"', '*.txt']
    command = shlex.split(r'find /i \"fatal\" *.txt')
    try:
        subprocess.run(command, shell=True, stdout=output_fh)
    except subprocess.CalledProcessError:
        pass

Perhaps, this is the only way to implement your task, because subprocess.run doesn't run redirections (> or >>) described by its arguments.

edited Jan 23 '18 at 10:52

answered Jan 22 '18 at 18:48

Sanjar Adilov

1,039
7
15

1

after the splitting you get `['find', '/i', 'fatal', '*.txt']`. Putting that into `subprocess.list2cmdline` returns `'find /i fatal *.txt'` which will fail in the cmd with `FIND: Parameter format not correct` due to the `fatal` being without quotes. – Ma0 Jan 23 '18 at 09:16

Subprocess.run .vs. running command in the cmd directly

3 Answers3