Passing string to subprocess.run seems to encase it in single quotes negating a wildcard search?

Question

I have a set of directories on my machine related to PCI devices (GPUs). Within these directories are various hwmon interfaces. I am attempting to find the specific path for each PCI device by running the command, for example,

$ find /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/hwmon/hwmon* -maxdepth 0

Here the wildcard will match the single directory located under .../hwmon/ for each PCI device. The above command outputs the following in my terminal

/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/hwmon/hwmon2

I am attempting to automate this process in python with subprocess.run

subprocess.run('find', gpu_pci_device_path + '/hwmon/hwmon*', '-maxdepth', '0', stdout=subprocess.PIPE).stdout.decode('utf-8')

Here I have already located (what I have termed) the PCI device path for each GPU in the variable gpu_pci_device_path. Then I am build the rest of the wildcard path to pass to find.

Though it seems subprocess is encasing the path I have built in single quotes based on the error it produces

find: ‘/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/hwmon/hwmon*’: No such file or directory

Thus it is negating my wildcard expression during find. How can I supply this wildcard expression to find during the call to subprocess.run?

My evidence that the encasing is happening and the wildcard is negated is from the output of the command

$ find '/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/hwmon/hwmon*' -maxdepth 0
find: ‘/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:00.0/0000:03:00.0/hwmon/hwmon*’: No such file or directory

That wildcard would normally be expanded by the shell before it's passed to `find`, but you're running this (implicitly) with `shell=False`, so that doesn't happen. You could use `shell=True`, but then `find` isn't actually doing anything other than printing the names it gets from the shell, which seems pointless. And the output is potentially ambiguous (if the filenames contain weird characters). Why not do it directly in Python with [`glob.glob` or something similar](https://stackoverflow.com/questions/3348753/search-for-a-file-using-a-wildcard)? — Gordon Davisson, Jan 09 '21 at 04:11

Charles Duffy · Accepted Answer · 2021-12-29T15:15:24.260

The Easy (Shell-Free) Answer

Using shell=True is bad practice, and you have no reason to do so here: Python can perform globbing itself with standard-library functionality.

subprocess.run((['find'] + 
                glob.glob(gpu_pci_device_path + '/hwmon/hwmon*') +
                ['-maxdepth', '0']),
               stdout=subprocess.PIPE).stdout.decode('utf-8')

This is not quite like real shell behavior: If no paths match, this will simply make the first argument to find be -maxdepth, which a shell would only do if the non-default nullglob option were set. To fix that, see below.

The Longer (Shell-Compatible) Answer

To get a closer emulation of the shell's behavior, we might add a wrapper around glob.glob() that makes default semantics more POSIX-y, and to support bash's failglob and nullglob options (which in bash can be enabled with shopt) to tune behavior when no matching files are found, and globstar to make ** optionally able to recurse:

import errno
import glob
import os
import subprocess

def glob_or_literal(expression, nullglob=False, failglob=False, globstar=False):
    literal_result = glob.glob(expression, recursive=globstar)
    if literal_result or nullglob:
        return literal_result
    if failglob:
        raise FileNotFoundError(errno.ENOENT, os.strerr(errno.ENOENT), expression)
    return [expression]

subprocess.run((['find'] + 
                glob_or_literal(gpu_pci_device_path + '/hwmon/hwmon*') +
                ['-maxdepth', '0']),
               stdout=subprocess.PIPE).stdout.decode('utf-8')

...in which case, if no hwmon directory exists, you'll get the literal glob expression passed to find to let it write an appropriately useful error message (just like POSIX-family shells do by default), unless you set the nullglob option (instructing the function, or a shell, to make a non-matching glob not expand to any arguments -- which may have unwanted behavior, like GNU find deciding to start the search in the current directory), or the failglob option (which makes the shell, or in this case the glob_or_literal Python function, abort before starting find at all).

Passing string to subprocess.run seems to encase it in single quotes negating a wildcard search?

1 Answers1

The Easy (Shell-Free) Answer

The Longer (Shell-Compatible) Answer

Related