How to run jq command in python?

Question

I am following a bigquery geospatial guide, and my json file is changed via a jq command. Specifically this one:

cat ~/file1.json | jq -c '.features[]' > converted.json

I'm having difficulties running the equivalent of above in python. I'm using the subprocess module, and it's not working.

cmd = ['cat', './sample.json', '|', 'c', '.features[]']

print(subprocess.check_output(cmd))

the output im getting is this:

cat: |: No such file or directory
cat: -c: No such file or directory
cat: .features[]: No such file or directory
Traceback (most recent call last):
  File "/Users/rtom/kml2geo/main.py", line 12, in <module>
    result = subprocess.check_output(cmd)
  File "/Users/rtom/opt/anaconda3/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/rtom/opt/anaconda3/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cat', './sample.json', '|', '-c', '.features[]']' returned non-zero exit status 1.
(base) rtom@Ryans-MacBook-Pro kml2geo %

Any help here?

There's no reason to use `cat` here _at all_, and it's far easier if you don't. — Charles Duffy, Dec 28 '21 at 01:52
(and why `subprocess.check_output()` when you want your output to go _direct to a file_?) — Charles Duffy, Dec 28 '21 at 01:56
BTW, the shell equivalent to a `stdin=open('filename', 'r')` argument to `subprocess.Popen`-family functions in Python is to use ` — Charles Duffy, Dec 28 '21 at 19:04

Charles Duffy · Accepted Answer · 2021-12-28T02:01:10.917

You don't need cat (you don't ever need cat when reading from a single file; its purpose is to concatenate multiple input sources together). Take it out, and everything gets far easier:

subprocess.run(['jq', '-c', '.features[]'],
               stdin=open(os.path.expanduser('~/file1.json'), 'r'),
               stdout=open('converted.json', 'w'))

You can also avoid the stdin= argument by passing the input filename (after replacing the ~ with a fully-qualified path, since there's no shell to do it for you) on jq's command line:

subprocess.run(['jq', '-c', '.features[]',
                os.path.expanduser('~/file1.json')
               ], stdout=open('converted.json', 'w'))

If you want to read output to a variable instead of writing it to a file, take out the stdout= argument and add capture_output=True.

That did it! Thank you so much. Happy new year friend. – Ryan Tom Dec 28 '21 at 02:05 — Ryan Tom, Dec 28 '21 at 02:05

How to run jq command in python?

1 Answers1