How do I embed my shell scanning-script into a Python script?

Question

Iv'e been using the following shell command to read the image off a scanner named scanner_name and save it in a file named file_name

scanimage -d <scanner_name> --resolution=300 --format=tiff --mode=Color 2>&1 > <file_name>

This has worked fine for my purposes. I'm now trying to embed this in a python script. What I need is to save the scanned image, as before, into a file and also capture any std output (say error messages) to a string

I've tried

    scan_result = os.system('scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name))

But when I run this in a loop (with different scanners), there is an unreasonably long lag between scans and the images aren't saved until the next scan starts (the file is created as an empty file and is not filled until the next scanning command). All this with scan_result=0, i.e. indicating no error

The subprocess method run() has been suggested to me, and I have tried

with open(file_name, 'w') as scanfile:

    input_params = '-d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} '.format(scanner, file_name)
    scan_result = subprocess.run(["scanimage", input_params], stdout=scanfile, shell=True)

but this saved the image in some kind of an unreadable file format

Any ideas as to what may be going wrong? Or what else I can try that will allow me to both save the file and check the success status?

You can't use `subprocess.run([... list ...], shell=True)`, it's either pass a single sting with `shell=True` or pass a list of tokens with no shell. — tripleee, Oct 28 '19 at 13:19

score 1 · Accepted Answer · answered Oct 28 '19 at 13:24

subprocess.run() is definitely preferred over os.system() but neither of them as such provides support for running multiple jobs in parallel. You will need to use something like Python's multiprocessing library to run several tasks in parallel (or painfully reimplement it yourself on top of the basic subprocess.Popen() API).

You also have a basic misunderstanding about how to run subprocess.run(). You can pass in either a string and shell=True or a list of tokens and shell=False (or no shell keyword at all; False is the default).

with_shell = subprocess.run(
    "scanimage -d {} --resolution=300 --format=tiff --mode=Color 2>&1 > {} ".format(
        scanner, file_name), shell=True)

with open(file_name) as write_handle:
    no_shell = subprocess.run([
        "scanimage", "-d", scanner, "--resolution=300", "--format=tiff",
            "--mode=Color"],  stdout=write_handle)

You'll notice that the latter does not support redirection (because that's a shell feature) but this is reasonably easy to implement in Python. (I took out the redirection of standard error -- you really want error messages to remain on stderr!)

If you have a larger working Python program this should not be awfully hard to integrate with a multiprocessing.Pool(). If this is a small isolated program, I would suggest you peel off the Python layer entirely and go with something like xargs or GNU parallel to run a capped number of parallel subprocesses.

For more details perhaps see also https://stackoverflow.com/questions/4256107/running-bash-commands-in-python/51950538#51950538 — tripleee, Oct 28 '19 at 13:24
I am skeptical of threading but https://stackoverflow.com/questions/26774781/python-multiple-subprocess-with-a-pool-queue-recover-output-as-soon-as-one-finis has some hints. https://stackoverflow.com/questions/13279813/subprocess-multiprocessing-multiple-commands-in-sequence is older (so maybe take it with a grain of salt) but closer to what I would suggest. — tripleee, Oct 28 '19 at 13:25

RightmireM · Answer 2 · 2019-10-28T09:49:10.613

I suspect the issue is you're opening the output file, and then running the subprocess.run() within it. This isn't necessary. The end result is, you're opening the file via Python, then having the command open the file again via the OS, and then closing the file via Python.

JUST run the subprocess, and let the scanimage 2>&1> filename command create the file (just as it would if you ran the scanimage at the command line directly.)

I think subprocess.check_output() is now the preferred method of capturing the output.

I.e.

from subprocess import check_output
# Command must be a list, with all parameters as separate list items
command = ['scanimage', 
           '-d{}'.format(scanner), 
           '--resolution=300', 
           '--format=tiff', 
           '--mode=Color', 
           '2>&1>{}'.format(file_name)]

scan_result = check_output(command)
print(scan_result)

However, (with both run and check_output) that shell=True is a big security risk ... especially if the input_params come into the Python script externally. People can pass in unwanted commands, and have them run in the shell with the permissions of the script.

Sometimes, the shell=True is necessary for the OS command to run properly, in which case the best recommendation is to use an actual Python module to interface with the scanner - versus having Python pass an OS command to the OS.

You can't use shell features like redirection if you pass a list of tokens and `shell=False`. — tripleee, Oct 28 '19 at 13:20

How do I embed my shell scanning-script into a Python script?

2 Answers2