37

I would like to convert a list into JSON array. I'm looking at jq for this but the examples are mostly about parsing JSON (not creating it). It would be nice to know proper escaping will occur. My list is single line elements so the new line will probably be the best delimiter.

jcalfee314
  • 4,642
  • 8
  • 43
  • 75
  • 1
    BTW, the -1 wasn't me, but if I had to guess as to why it's there, it's probably the "please do my work for me" scent this (otherwise useful and worthy, hence my answer) question gives off. Showing your work and issues you've hit trying to solve this yourself would avoid that. – Charles Duffy Oct 09 '14 at 21:53
  • thank you, yes I did google around. I was looking for a small one-liner maybe something I missed in the `jq` man pages. Thanks for the feedback, that up-vote was me. – jcalfee314 Oct 09 '14 at 23:42

5 Answers5

46

I was also trying to convert a bunch of lines into a JSON array, and was at a standstill until I realized that -s was the only way I could handle more than one line at a time in the jq expression, even if that meant I'd have to parse the newlines manually.

jq -R -s -c 'split("\n")' < just_lines.txt
  • -R to read raw input
  • -s to read all input as a single string
  • -c to not pretty print the output

Easy peasy.

Edit: I'm on jq ≥ 1.4, which is apparently when the split built-in was introduced.

chbrown
  • 11,865
  • 2
  • 52
  • 60
  • Looks easy .. I can't verify though. I get an error on ubuntu jq version 1.3 `error: split is not defined` – jcalfee314 Jan 18 '15 at 16:18
  • 1
    Unfortunately, the `jq` documentation doesn't say when each built-in function was introduced, nor does `jq --version` tell me what version I've got, but it's at least 1.4, installed with `brew install jq --HEAD`. Good point, though, I'll edit the answer. – chbrown Jan 18 '15 at 20:15
  • 6
    I like this answer except that for a multi-line output that has a trailing newline(\n), the `split` built-in function will yield an extra empty string item in the array :( Anyway to fix that? – Devy Oct 21 '16 at 15:46
  • @Devy one way could be to remove the trailing newline before processing. But how do I remove the string quotes that arise after this command? – Koustuv Sinha Jun 22 '17 at 20:58
  • 3
    My current hack is to repipe it into `jq` to use its slicing feature ¯\_(ツ)_/¯ `echo $(jq -R -s -c 'split("\n")' < urls.txt) | jq '.[:-1]'` – Derek Argueta Feb 18 '18 at 06:03
  • How do you reverse this to original? – user31986 Oct 16 '19 at 00:04
  • @user31986 `... | jq -r '.[]'` will dump a JSON array of strings to newline-separated lines, without quotes (because `-r`). – chbrown Oct 17 '19 at 01:08
  • 3
    Consider `'split("\n") | map(select(length > 0))'` to remove empty lines, too. – Justin W. Nov 05 '19 at 21:11
  • `cat lines.txt | jq -s -R 'split("\n")[:-1]'` – Nino Filiu Apr 16 '23 at 19:58
26

--raw-input, then --slurp

Just summarizing what the others have said in a hopefully quicker to understand form:

cat /etc/hosts  | jq  --raw-input .  | jq --slurp .

will return you:

[
  "fe00::0 ip6-localnet",
  "ff00::0 ip6-mcastprefix",
  "ff02::1 ip6-allnodes",
  "ff02::2 ip6-allrouters"
]

Explanation

 --raw-input/-R:

       Don´t parse the input as JSON. Instead, each line of text is passed
       to  the  filter  as  a  string.  If combined with --slurp, then the
       entire input is passed to the filter as a single long string.

 --slurp/-s:

       Instead of running the filter for each JSON object  in  the  input,
       read  the entire input stream into a large array and run the filter
       just once.
Sridhar Sarnobat
  • 25,183
  • 12
  • 93
  • 106
9

You can also use jq -R . to format each line as a JSON string and then jq -s (--slurp) to create an array for the input lines after parsing them as JSON:

$ printf %s\\n aa bb|jq -R .|jq -s .
[
  "aa",
  "bb"
]

The method in chbrown's answer adds an empty element to the end if the input ends with a linefeed, but you can use printf %s "$(cat)" to remove trailing linefeeds:

$ printf %s\\n aa bb|jq -R -s 'split("\n")'
[
  "aa",
  "bb",
  ""
]
$ printf %s\\n aa bb|printf %s "$(cat)"|jq -R -s 'split("\n")'
[
  "aa",
  "bb"
]

If the input lines don't contain ASCII control characters (which have to be escaped in strings in valid JSON), you can use sed:

$ printf %s\\n aa bb|sed 's/["\]/\\&/g;s/.*/"&"/;1s/^/[/;$s/$/]/;$!s/$/,/'
["aa",
"bb"]
nisetama
  • 7,764
  • 1
  • 34
  • 21
  • When pipeing from `jq` to `jq` pretty printing is not necessary: `jq -cR . | jq -s .` – ceving Oct 27 '16 at 10:00
  • 1
    Rather than slurping the inputs in a separate call, just throw the raw inputs into an array. `jq -Rn '[inputs]' input.json`. These are both equivalent: `jq -s '.' input.json` and `jq -n '[inputs]' input.json` – Jeff Mercado Aug 10 '17 at 03:18
8

Update: If your jq has inputs you can simply write:

jq -nR [inputs] /etc/hosts

to produce a JSON array of strings. This avoids having to read the text file as a whole.

peak
  • 105,803
  • 17
  • 152
  • 177
4

I found in the man page for jq and through experimentation what seems to me to be a simpler answer.

$ cat test_file.txt | jq -Rsc '. / "\n" - [""]'
["aa","bb"]

The -R is to read without trying to parse json, the -s says to read all of the input as one string, and the -c is for one-line output - not necessary, but it's what I was looking for.

Then in the string I pass to jq, the '.' says take the input as it is. The '/ \n' says to divide the string (split it) on newlines. The '- [""]' says to remove from the resulting array any empty strings (resulting from an extra newline at the end).

It's one line and without any complicated constructs, using just simple built in jq features.

Corey Woodfield
  • 350
  • 2
  • 10