42

My request sounds trivial but I could not find a way to do it. I have as input an array of JSON objects:

[
    {
        "foo": 1,
        "bar": 2
    },
    {
        "foo": 3,
        "bar": 4
    },
    (...)
]

and I want as output the JSONL version of the same, aka one object per line, not an array:

    { "foo": 1, "bar": 2 }
    { "foo": 3, "bar": 4 }
    (...)

This is not the same as using --compact-output, as that would preserve the array and give me:

    [ { "foo": 1, "bar": 2 }, { "foo": 3, "bar": 4 }, (...) ]

Thank you in advance.

peak
  • 105,803
  • 17
  • 152
  • 177
giacecco
  • 661
  • 1
  • 5
  • 8
  • This should be a feature: `--line-output`. Then you can feed the data into all the unix line-based tools. – NeilG Oct 17 '21 at 21:19

3 Answers3

57

The answer to the original question is to use the filter .[] together with the -c command-line option:

$ jq -c '.[]'

If the input file is very large (notably, if it is too large to fit into memory), it may be better to use jq's --stream command-line option, or a companion tool. If the preservation of numerical precision is important, then you may wish to consider jm, which is particularly easy to use: invoking jm without any parameters suffices. See the jq Cookbook: https://github.com/stedolan/jq/wiki/Cookbook

peak
  • 105,803
  • 17
  • 152
  • 177
  • This didn't work with the direct output, but for some reason running it twice did: `$ curl | jq '.[]' | jq -c '.[]'` – NeilG Oct 17 '21 at 21:41
2

If the input array is too large to fit into memory, you can use jq's so-called "streaming parser".

Here is an illustration using a generic approach, that is, it makes no assumptions about the items in the top-level array:

$ echo '[{"foo":"bar"},99,null,{"foo":"baz"}]' |
  jq -cn --stream 'fromstream( inputs|(.[0] |= .[1:]) | select(. != [[]]) )'
{"foo":"bar"}
99
null
{"foo":"baz"}
$
peak
  • 105,803
  • 17
  • 152
  • 177
  • This didn't work for me until I captured the response data and then used `cat` to push it through the construct again. In other words, this works: `$ curl | jq -cn --stream 'fromstream( inputs|(.[0] |= .[1:]) | select(. != [[]]) )' | jq -cn --stream 'fromstream( inputs|(.[0] |= .[1:]) | select(. != [[]]) )'`. Perhaps you could explain what each part of the command does? – NeilG Oct 17 '21 at 21:28
-2

Found: it's

map(tostring) | reduce .[] as $item (""; . + $item + "\n")

You also need to use --raw-output.

giacecco
  • 661
  • 1
  • 5
  • 8
  • This doesn't work either, but at least you can split on `},{`, at least on my data set it worked. – NeilG Oct 17 '21 at 21:17