jq to replace text directly on file (like sed -i)

Question

I have a json file that needs to be updated on a certain condition.

Sample json

{
   "Actions" : [
      {
         "value" : "1",
         "properties" : {
            "name" : "abc",
            "age" : "2",
            "other ": "test1"
          }
      },
      {
         "value" : "2",
         "properties" : {
            "name" : "def",
            "age" : "3",
            "other" : "test2"
          }
      }
   ]
}

I am writing a script that makes use of Jq to match a value and update, as shown below

cat sample.json |  jq '.Actions[] | select (.properties.age == "3") .properties.other = "no-test"'

Output (printed to terminal)

{
  "value": "1",
  "properties": {
    "name": "abc",
    "age": "2",
    "other ": "test1"
  }
}
{
  "value": "2",
  "properties": {
    "name": "def",
    "age": "3",
    "other": "no-test"
  }
}

While this command makes the needed change, it outputs the entire json on the terminal and does not make change to the file itself.

Please advise if there is an option to have jq make changes on the file directly (similar to sed -i).

Answer 1

This post addresses the question about the absence of the equivalent of sed's "-i" option, and in particular the situation described:

I have a bunch of files and writing each one to a separate file wouldn't be easy.

There are several options, at least if you are working in a Mac or Linux or similar environment. Their pros and cons are discussed at http://backreference.org/2011/01/29/in-place-editing-of-files/ so I'll focus on just three techniques:

One is simply to use "&&" along the lines of:

jq ... INPUT > INPUT.tmp && mv INPUT.tmp INPUT

Another is to use the sponge utility (part of GNU moreutils):

jq ... INPUT | sponge INPUT

The third option might be useful if it is advantageous to avoid updating a file if there are no changes to it. Here is a script which illustrates such a function:

#!/bin/bash

function maybeupdate {
    local f="$1"
    cmp -s "$f" "$f.tmp"
    if [ $? = 0 ] ; then
      /bin/rm $f.tmp
    else
      /bin/mv "$f.tmp" "$f"
    fi
}

for f
do
    jq . "$f" > "$f.tmp"
    maybeupdate "$f"
done

Answer 2

instead of sponge :

cat <<< $(jq 'QUERY' sample.json) > sample.json

Answer 3

You ran into two issues:

• This is a common problem for text processing, not solved in the base Linux distribution.
• jq did not write special code to overcome this problem.

One good solution:

• Install moreutils using brew install moreutils or your favorite package manager. This contains the handy program sponge, for just this purpose.
• Use cat myfile | jq blahblahblah | sponge myfile. That is, run jq, capturing the standard out, when jq has finished, then write the standard output over myfile (the input file).

Answer 4

You'll want to update the action objects without changing the context. By having the pipe there, you're changing the context to each individual action. You can control that with some parentheses.

$ jq --arg age "3" \
'(.Actions[] | select(.properties.age == $age).properties.other) = "no-test"' sample.json

This should yield:

{
  "Actions": [
    {
      "value": "1",
      "properties": {
        "name": "abc",
        "age": "2",
        "other ": "test1"
      }
    },
    {
      "value": "2",
      "properties": {
        "name": "def",
        "age": "3",
        "other": "no-test"
      }
    }
  ]
}

You can redirect the results to a file to replace the input file. It won't do in-place updates to a file as sed does.

Answer 5

I use yq, For advanced users this -i (in-place update) is needed, hope be added to jq

yq -iP '.Email.Port=3030' config.json -o json

• -i in place update
• -P pretty print
• -o output should be json

---

yq --version
yq (https://github.com/mikefarah/yq/) version 4.21.1

Answer 6

use tee command

➜ cat config.json|jq '.Actions[] | select (.properties.age == "3") .properties.other = "no-test"'|tee config.json
{
  "value": "1",
  "properties": {
    "name": "abc",
    "age": "2",
    "other ": "test1"
  }
}
{
  "value": "2",
  "properties": {
    "name": "def",
    "age": "3",
    "other": "no-test"
  }
}

➜ cat config.json
{
  "value": "1",
  "properties": {
    "name": "abc",
    "age": "2",
    "other ": "test1"
  }
}
{
  "value": "2",
  "properties": {
    "name": "def",
    "age": "3",
    "other": "no-test"
  }
}

Answer 7

Using my answer to a duplicate question

Assignment prints the whole object with the assignment executed so you could assign a new value to .Actions of the modified Actions array

.Actions=([.Actions[] | if .properties.age == "3" then .properties.other = "no-test" else . end])

I used an if statement but we can use your code to do the same thing

.Actions=[.Actions[] | select (.properties.age == "3").properties.other = "no-test"]

The above will output the entire json with .Actions edited. jq does not had sed -i like functionality, but all you need to do is pipe it back into a sponge to the file with | sponge

 jq '.Actions=([.Actions[] | if .properties.age == "3" then .properties.other = "no-test" else . end])' sample.json | sponge sample.json

Answer 8

It's possible to do something like:

echo "$(jq '. + {"registry-mirrors": ["https://docker-mirror"]}' /etc/docker/daemon.json)" > /etc/docker/daemon.json

So it gets text in sub-shell using jq and echoes it to file in 'main' shell.

Note: The main idea here is to illustrate how it can be achieved without additional tools like sponge or so. Instead of echo you can use any command which can write to stdout e.g. printf '%s' "$(jq ... file)" > file.

P.S Issue in jq project is still open: https://github.com/stedolan/jq/issues/105

Answer 9

The simplest way of accomplishing it is to load the file into a variable first and then send it into jq.

content=$(cat sample.json) && 
jq '<your jq script>' <<<$content >sample.json

This will only work if your JSON file fits in shell variable. To find out the maximum size the bash variable on your system run:

getconf ARG_MAX

Mine shows 2MB.

Answer 10

This bash (probably sh compatible) function jqi will take care of everything.

Usage: jqi [-i] <filename> [jq options] <jq filter>

e.g.:

fix-node-sass() 
{ 
    jqi -i package.json '.resolutions += {"node-sass": "6.0.1"}' \
                  '| .devDependencies += {"node-sass": "6.0.1"}'

}

Much like sed or perl, specify -i as the leading argument to force rewriting of the original file. If -i is not specified, it will be a "dry run" and output will go to stdout.

If for some arcane reason you want to do something weird like:

cat in.json | jq -i - > out.json

Then out.json will hold either the result, or the original contents of in.json on error -- i.e., out.json should be valid json.

Note: an output of less than 7 characters (e.g. null) is considered an error, and will not overwrite. You can disable this safety feature if you wish.

jqi () 
{ 
    local filename=$1;
    shift;
    local inplace=;
    local stdin=;
    if [[ $filename == "-i" ]]; then
        echo "jqi: in-place editing enabled" 1>&2;
        inplace=y;
        filename=$1;
        shift;
    fi;
    if [[ $filename == "-" ]]; then
        echo "jqi: reading/writing from stdin/stdout" 1>&2;
        if [ -n "$inplace" ]; then
            stdin=y;
            inplace=;
        fi;
        filename="/dev/stdin";
    fi;
    local tempname="$( mktemp --directory --suffix __jq )/$( dirname "$filename" ).$$.json";
    local timestamp="${tempname%json}timestamp";
    local -i error=0;
    cat "$filename" > "$tempname";
    touch "$timestamp";
    while :; do
        if jq "${*}" "$filename" > "$tempname"; then
            if test "$tempname" -nt "$timestamp"; then
                local ls_output=($( ls -Lon "$tempname" ));
                filesize=${ls_output[3]};
                if [[ $filesize -lt 7 ]]; then
                    echo "jqi: read only $filesize bytes, not overwriting" 1>&2;
                    error=1;
                    break;
                fi;
                if [ -n "$inplace" ]; then
                    cat "$tempname" > "$filename";
                else
                    echo "jqi: output from dry run" 1>&2;
                    cat "$tempname";
                fi;
                error=0;
                break;
            else
                echo "jqi: output not newer, not overwriting" 1>&2;
                error=1;
                break;
            fi;
        else
            echo "jqi: jq error, not overwriting" 1>&2;
            error=1;
            break;
        fi;
    done;
    if [ -n "$stdin" ] && [ $error -eq 1 ]; then
        echo "jqi: output original to stdout" 1>&2;
        cat "$filename";
    fi;
    rm "$tempname" "$timestamp";
    rmdir "$( dirname "$tempname" )"
}

Answer 11

in one line : cat file.json | jq '.' | tee file.json.json >/dev/null