Running multiple bash commands with individual output files in a single subprocess calls

Question

I wanted to see if this was feasible. I'd like to execute two separate bash commands, where the second command needs to take the output of the first command as an input. Both commands would need to generate an output file as well. Is this possible to do in a single subprocess call? I'd like to combine this with psrecord and have both commands graphed in the same plot, which is why I'm trying to combine them:

psrecord 'program --arg1 foo --arg2 bar --output out1.txt && program out1.txt --arg3 baz --output out2.csv' --log activity.txt --plot plot.png

Apologies if this question has been asked before; I know there have been posts about executing multiple bash commands, but I'm more concerned about the 2 output files, passing the first output to the second command, and being able to record both commands in one psrecord graph. Thank you in advance for your help.

This reads like you already have a single compound shell command. Does that command do what you want? If not, exactly how does its behavior differ from what you intend? If so, have you tried invoking it from the subprocess module? What specific problem did that have? — Charles Duffy, Aug 07 '18 at 17:52
@CharlesDuffy I think it's because I want to pass that to a subprocess call, and I'm not sure what the stdout should be: `cmd = psrecord 'program --arg1 foo --arg2 bar --output out1.txt && program out1.txt --arg3 baz --output out2.csv' --log activity.txt --plot plot.png` `process = subprocess.Popen(shlex.split(cmd), stdout=??)` — Melissa, Aug 07 '18 at 18:02
Don't use `shlex.split()` -- if unwilling to split it across multiple `Popen` calls you need a shell for this (pipes and the like are shell syntax), so use `shell=True`. That said, be sure that if your filenames come from variables, you have the script refer to positional parameters rather than using string substitution to avoid security vulnerabilities. — Charles Duffy, Aug 07 '18 at 18:05
...which of the filenames here are hardcoded in your real code, and which ones come from the values of Python variables? Also, are you targeting Python 2.x or 3.x? — Charles Duffy, Aug 07 '18 at 18:07
I'm writing it in Python 3.x; the input filenames are given in the command line when executing the script, and I parse the output filename (as there can be multiple input files) from that. Most of the args and values are given in the command line and parsed out using argparse. I'm working on a script that uses `psrecord` to see resource usage of different programs, and one of those programs require that compound bash command to get an equivalent output as the other programs, if that makes sense. — Melissa, Aug 07 '18 at 18:14
...btw, I'm not sure quite where you see the multiple output files adding to the complexity here -- `subprocess` doesn't care what output files a program creates as a side effect; it's only responsible for wiring up stdin, stdout and stderr, and passing in an argument list. — Charles Duffy, Aug 07 '18 at 18:15
...also, I'm assuming in my answer that the `psrecord` command given actually works when you run it at a shell (thus, that it implicitly passes its arguments to `sh -c` or another `system()` equivalent; that doesn't speak well of its author's judgment, but that's something that's on them). — Charles Duffy, Aug 07 '18 at 18:18
...and if you're passing arguments through from your program's command line, that makes the approach given in my answer (constructing a list of strings) quite advisable -- that way your command line can contain `'$(rm -rf ~)'` without your program generating a shell program that tries to run that as a command, f/e. :) — Charles Duffy, Aug 07 '18 at 18:20
Thank you to Charles and Jeremy; between your comments and suggestions I was able to get it to work. — Melissa, Aug 07 '18 at 19:50

score 2 · Answer 1 · answered Aug 07 '18 at 17:39

2

It sounds like your goal is to write a bash one-liner and then shove it in python, so I'll focus on the bash.

I'm assuming your stdout for cmd1 needs to go both to the next command and to a file, and your cmd2 just needs to pass stdout to file. tee is perfect for this.

cmd1 | tee /path/to/cmd1file.log | cmd2 > /path/to/cmd2file.log

In general, | takes the output of one command and makes in the input for the next command in the line. tee takes input, pushes one copy to stdout and puts one copy somewhere else; in this case, to a log file. > writes the stdout to a file.

answered Aug 07 '18 at 17:39

jeremysprofile

10,028
4
33
53

**If** this answer is responsive, then presumably you believe the question should be closed as a duplicate of [How to redirect output to a file and stdout](https://stackoverflow.com/questions/418896/how-to-redirect-output-to-a-file-and-stdout)? – Charles Duffy Aug 07 '18 at 17:54
Would this work passing through `subprocess.Popen()`, leaving it as `stdout=None`? I think that's where I'm worried issues would occur. But I will try this, thank you. – Melissa Aug 07 '18 at 18:06

score 2 · Accepted Answer · answered Aug 07 '18 at 18:12

A version of your code which tries to be as safe as possible might look like:

import shlex, pipes
if hasattr(shlex, 'quote'):
    quote = shlex.quote # Python 3
else:
    quote = pipes.quote # Python 2

inner_cmds = [
  [
    'program',
    '--arg1', 'foo',
    '--arg2', 'bar',
    '--output', 'out1.txt'
  ], [
    'program',
    'out1.txt',
    '--arg3', 'baz',
    '--output', 'out2.csv'
  ]
]

inner_cmd_str = ' && '.join(' '.join(shlex.quote(word) for word in inner_cmd) for inner_cmd in inner_cmds)
cmd = [
  'psrecord', inner_cmd_str,
  '--log', 'activity.txt',
  '--plot', 'plot.png'
]

p = subprocess.Popen(cmd)

Running multiple bash commands with individual output files in a single subprocess calls

2 Answers2