Send 'jq' output to a file using subprocess

Question

I am not entirely sure how jq works here with subprocess so there is a misunderstanding but I am trying to get the extact format from using jq on the command line ( jq > file.json) with subprocess. This is what I have but the following produces an empty file.

os.makedirs(os.path.dirname(filePath))
open(filePath, 'a').close()

call(["bw", "list", "--session", session_key, \
    "collections", "|", "jq", ".", ">", filePath])

I have also tried

with open(filePath, "w+") as output:
        call(["bw", "list", "--session", session_key, \
         "collections", "|", "jq", "."], stdout=output)

But this produces a string instead of the actual formatting of jq. Is there a way for me to obtain the stdout jq on the commandline into a file using python?

`|` and `>` to configure pipes and redirect output are features offered by your shell. `subprocess.call()` runs `bw` *directly*, with no intermediary shell process. So either handle connecting process pipes and and output file yourself in Python or use `shell=True` and pass in a single string. — Martijn Pieters, Feb 10 '19 at 13:15

score 2 · Accepted Answer · edited Feb 11 '19 at 08:41

Without a shell, you need two subprocesses.

os.makedirs(os.path.dirname(filePath))
dst = open(filePath, 'a')

p0 = Popen(["bw", "list", "--session", session_key, "collections"], 
    stdout=PIPE, check=True)
p1 = Popen(["jq", "."], stdout=dst, stdin=p0.stdout, check=True)
p1.communicate()
dst.close()

This is almost literally lifted from the documentation. There is a pipes module in the standard library which somewhat simplifies this, though it is arguably clunky; thus, there are also several third-party replacements.

On the other hand, perhaps this is one of those situations where you can defend shell=True.

os.makedirs(os.path.dirname(filePath))
dst = open(filePath, 'a')
subprocess.run("""
    bw list --session '{}' collections |
    jq .""".format(session_key),
    stdout=dst, shell=True, check=True)
dst.close()

On the third hand, if you really want a shell script, why are you writing Python?

The first solution worked perfect, thanks. – Chris Feb 10 '19 at 13:51 — Chris, Feb 10 '19 at 13:51
What is the `check= true` doing here? – Chris Feb 10 '19 at 13:54 — Chris, Feb 10 '19 at 13:54
It causes an exception to be raised if the command fails. – tripleee Feb 10 '19 at 18:48 — tripleee, Feb 10 '19 at 18:48

Send 'jq' output to a file using subprocess

1 Answers1