2

I am trying to include this line in a python script.

!#/bin/bash/env python

import os

os.system("paste <(awk '!/^($|[:space:]*#)/{print $0}' file1) <(awk '!/^($|[:space:]*#)/{print $0} file2) > out_file")

The command is perfectly fine when run from bash directly. However, inside the script, I get:

sh: -c: line0: syntax error near unexpected token `('

The problem persists when using simply:

os.system("paste <(cat file1) > output_file")

Any ideas?

astabada
  • 1,029
  • 4
  • 13
  • 26
  • 6
    It looks like your default shell is `sh`, not `bash`. And both of your command lines are invalid with `sh`. – abarnert Apr 15 '13 at 11:09
  • @MartijnPieters: He's not using `subprocess`, he's using `os.system`. Which, on Unix systems, just calls the libc function `system`, which uses… well, that's up to your OS, and we don't know what OS the OP is using. – abarnert Apr 15 '13 at 11:12
  • @abarnert: right, indeed; my mistake. I'll point to [`man system`](http://linux.die.net/man/3/system) instead then, which states it uses `/bin/sh -c`. – Martijn Pieters Apr 15 '13 at 11:13
  • @MartijnPieters: Did the OP say he's using linux somewhere, or are you just assuming that everyone who tries to use `system` is on linux? – abarnert Apr 15 '13 at 11:16
  • @abarnert: I am assuming it's a POSIX system, since `sh` and `bash` are installed. – Martijn Pieters Apr 15 '13 at 11:17
  • @MartijnPieters: POSIX only requires a shell that acts as if it were `sh` when given a valid `sh` command (which means it could extend `sh` in ways that don't prevent running valid `sh` commands). Some platforms (like cygwin) use `bash` with the right flags as `sh`, but use different flags as `sh` vs. as `system`. (For that matter, it also allows, but "discourages", following the `SHELL` env variable—because at least one commercial UNIX did so.) – abarnert Apr 15 '13 at 11:26

2 Answers2

4

The command is perfectly fine when run from bash directly. However, inside the script, I get:

sh: -c: line0: syntax error near unexpected token `('

That's because inside the script, you're running the command with sh rather than bash. Both this command, and the simpler one, use bash-specific features. Try running an sh shell and typing the same lines, and you'll get the same error.

The os.system call doesn't document what shell it uses, because it's:

implemented by calling the Standard C function system()

On most Unix-like systems, this calls sh. You probably shouldn't rely on that… but you definitely shouldn't rely on it calling bash!

If you want to run bash commands, use the subprocess module, and run bash explicitly:

subprocess.call(['bash', '-c', 'paste <(cat file1) > output_file'])

You could, I suppose, try to get the quoting right to run bash as a subshell within the shell system uses… but why bother?

This is one of the many reasons that the documentation repeatedly tells you that you should consider using subprocess instead of os.system.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • And you should add that in order to run the command in `bash` you can use subprocess, as in [this answer](http://stackoverflow.com/a/4256153/418609). – adrianp Apr 15 '13 at 11:12
  • @adrianp: That answer doesn't show how to use `bash` explicitly; it shows how to run the `cwm` command without using a shell, which is not what the OP wants. – abarnert Apr 15 '13 at 11:15
1

Kill two birds with one awk script:

awk -v DELIM=' ' '!/^($|[[:space:]]*#)/{a[FNR]=a[FNR]DELIM$0}END{for(i=1;i<=FNR;i++)print substr(a[i],2)}' file1 file2

This eliminates the need for process substitution and is therefor sh compliant.

Chris Seymour
  • 83,387
  • 30
  • 160
  • 202
  • Thanks for your suggestion, but the situation is a bit more complicated: I don't specify the number of files directly, but rather they are arguments of the script, e.g. script.py file1 file2 file3 ... In other words: I have no idea on how to extend your example to N files. Do I have to do it a couple at a time? file1+file2=part1 part1+file3=part2 partN-1+fileN=final? – astabada Apr 15 '13 at 12:10
  • @astabada You would just parse the list of files after the script `file1 file2 ... fileN` . Calling `awk` multiple times (once per file) is the wrong approach when a single script can handle it. Subsequently this eliminates your original problem. – Chris Seymour Apr 15 '13 at 12:16
  • ok, it works... One further (and last, hopefully) question. Do you know of any comprehensive place where to look to understand the command? man awk does not seem sufficient. – astabada Apr 15 '13 at 15:51
  • Read effective awk programming here is a line http://www.gnu.org/software/gawk/manual/ – Chris Seymour Apr 15 '13 at 17:21