Python 2.6 : piping bash commands containing python variables(inside python script)

Question

I want to run the below bash command from my python script:

stat --printf='%U%G%a' /tmp/file1.csv &&md5sum /tmp/file1.csv |awk '{print $1}'

I have done it using subprocess.Popen as below:

Command=subprocess.Popen(["stat --printf='%U%G%a' file1.csv &&md5sum file1.csv|awk '{print $1}'"],stdout=subprocess.PIPE,shell=True)

But instead of hard coding the filename I need to pass a python variable. I tried

filevar="/tmp/file.csv"
Command=subprocess.Popen(["stat --printf='%U%G%a' filevar &&md5sum filevar|awk '{print $1}'"],stdout=subprocess.PIPE,shell=True)

But the above code is not working.

I have been through all the answers related to How to pass a python variable to subprocess

The best answer I got till now is piping python variable value to bash script (inside python script)

Based on this I tried:

Command=subprocess.Popen(["stat","--printf='%U%G%a'",filevar],stdout=subprocess.PIPE)

Which works great. But when I try to include more commands like md5sum it throws error.

Command=subprocess.Popen(["stat","--printf='%U%G%a'",filevar,"&&","md5sum",filevar],stdout=subprocess.PIPE)

Please suggest how this could be done.

score 1 · Accepted Answer · edited May 23 '17 at 11:54

To support spaces and other shell meta-characters, use pipes.quote():

#!/usr/bin/env python
import pipes
from subprocess import check_output

path = "/path/to/file.csv"
output = check_output("stat --printf='%U%G%a' {path} && md5sum {path}"
                      .format(path=pipes.quote(path))
                      + "|awk '{print $1}'", shell=True)

To get check_output() on Python 2.6, see What's a good equivalent to python's subprocess.check_call that returns the contents of stdout?

Note: pipes.quote() is not bullet-proof. Don't pass path to the shell unless it comes from a trusted source otherwise you risk an arbitrary shell command being executed (shell injection).

As an alternative, you could use plumbum to emulate the pipeline:

#!/usr/bin/env python
from plumbum.cmd import stat, md5sum, awk  # $ pip install plumbum

path = "/path/to/file.csv"
stat["--printf=%U%G%a", path]()
output = (md5sum[path] | awk['{print $1}'])()

See How do I use subprocess.Popen to connect multiple processes by pipes?

Depending on your case, it might make sense to implement the command in pure Python without external commands.

@Sebastian Thanks a lot , I used plumbum to execute the commands — rdm_, Jun 22 '16 at 06:07
@rdm_ check whether you need to remove quotes around `--printf` value. — jfs, Jun 22 '16 at 07:07

score 0 · Answer 2 · answered Jun 14 '16 at 12:07

0

When you call Popen you simply pass a string to the function. So you could use string formatting to put the correct variable in there:

filevar="/tmp/file.csv"
Command=subprocess.Popen(["stat --printf='%U%G%a' %(filevar) &&md5sum filevar|awk '{print $1}'" % {"filevar": filevar}],stdout=subprocess.PIPE,shell=True)

answered Jun 14 '16 at 12:07

datosh

498
7
20

thanks @datosh, but im getting the below error using the above command `Traceback (most recent call last): File "linuxx.py", line 8, in Command=subprocess.Popen(["stat --printf='%U%G%a' %(filevar) &&md5sum filevar|awk '{print $1}'" % {"filevar": filevar}],stdout=subprocess.PIPE,shell=True) ValueError: unsupported format character 'U' (0x55) at index 16` . Please suggest. – rdm_ Jun 15 '16 at 12:38

Python 2.6 : piping bash commands containing python variables(inside python script)

2 Answers2