What is the most efficient way of writing a JSON file with Bash?

Question

So I have to write a JSON file with bash script, and I know I can do something like echo 'something' >> $file to slowly build a file, but echo redirection instead of real file output seems kind of "hacky." If that is the best way, and not a hacky way at all, I am happy to use echo, but I was just wondering if there is a better way to output a file from bash script.

Answer 1

Efficiently generating output

echo is a built-in, not an external command, so it's not nearly as inefficient as you think. What is inefficient is putting >> filename on the end of each echo.

This is bad:

# EVIL!
echo "something" >file
echo "first line" >>file
echo "second line" >>file

This is good:

# NOT EVIL!
{
  echo "something" >&3
  printf '%s\n' "first line" "$second line" >&3
  # ... etc ...
} 3>file

...opens the output file only once, eliminating the major inefficiency.

To be clear: Calling echo, say, 20 times is considerably more efficient than calling cat once, since cat is an external process, not part of the shell. What's highly inefficient about running echo "foo" >>file 20 times is opening and closing the output file 20 times; it's not echo itself.

---

Correctly generating JSON

Don't use cat, echo, printf, or anything else of the sort. Instead, use a tool that understands JSON -- any other approach will lead to potentially incorrect (perhaps even exploitable via injection attacks) results.

For instance:

jq \
  --arg something "$some_value_here" \
  --arg another "$another_value" \
  '.["something"]=$something | .["another_value"]=$another' \
  <template.json >output.json

...will generate a JSON file, based on template.json, with something set to the value in the shell variable "$some_value_here" and another_value set to, well, a second value. Unlike naive approaches, this will correctly handle variable values which contain literal quotes or other characters which need to be escaped to be correctly represented.

---

An aside on echo

All the above having been said -- echo should be avoided in favor of printf (with an appropriate, static format string). Per the POSIX sh standard:

APPLICATION USAGE

It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.

The printf utility can be used portably to emulate any of the traditional behaviors of the echo utility as follows (assuming that IFS has its standard value or is unset):

[...]

New applications are encouraged to use printf instead of echo.

RATIONALE

The echo utility has not been made obsolescent because of its extremely widespread use in historical applications. Conforming applications that wish to do prompting without s or that could possibly be expecting to echo a -n, should use the printf utility derived from the Ninth Edition system.

As specified, echo writes its arguments in the simplest of ways. The two different historical versions of echo vary in fatally incompatible ways.

The BSD echo checks the first argument for the string -n which causes it to suppress the that would otherwise follow the final argument in the output.

The System V echo does not support any options, but allows escape sequences within its operands, as described for XSI implementations in the OPERANDS section.

The echo utility does not support Utility Syntax Guideline 10 because historical applications depend on echo to echo all of its arguments, except for the -n option in the BSD version.

Answer 2

You can use cat and here-document format:

cat <<'EOF' > output.json
{
    "key": "value",
    "num": 5,
    "tags": ["good", "bad"],
    "money": "$0"
}
EOF

Note the single ticks around the here-document anchor. This prevents interpolation of the document's contents. Without it, the $0 can be substituted.

If you define efficiency as raw speed as opposed to readability, you should consider using Charles Duffy's answer instead as it's almost an order of magnitude faster for small number of lines (echo 0.01s vs cat 0.1s).
If you need to create files larger than a few hundred lines, you should consider a method other than cat/echo.

Answer 3

Construct the data in an environment variable, and echo it once.

var=something
var="$var something else"
var="$var and another thing"
echo "$var" > file

Answer 4

Besides echo, you could use cat:

cat > myfile << EOF
Hello
World
!
EOF

Answer 5

You can use cat and "heredocs" to minimize the number of calls you have to make.

$ cat foo.sh
cat <<'HERE' > output
This
that
the other
    indentation is 
        preserved

as are 

    blank lines

The end.
HERE
$ sh foo.sh
$ cat output
This
that
the other
    indentation is 
        preserved

as are 

    blank lines

The end.