Trying to swap JSON element using sed editor on MAC

Question

I am trying to create a script to find a JSON element and update it with the arg values.

 #!/bin/bash

# Shell script to verify the end to end D1 request flow
placeLocation=$1 
vehicleHeading=$2
message=$3
file=one.txt

sed -i '' '/location/c\   \"location\" : \"$placeLocation\",' $file
sed -i '' '/heading/c\   \"heading\" : \"$vehicleHeading\",' $file
sed -i '' '/message/c\   \"message\" : \"$message\",' $file

One.txt

"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",

But getting following error

sed: 1: "/location/c\   \"locati ...": extra characters after \ at the end of c command
sed: 1: "/heading/c\   \"heading ...": extra characters after \ at the end of c command
sed: 1: "/message/c\   \"message ...": extra characters after \ at the end of c command
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f

I have just started learning about sed editor and tried out multiple things but couldn't able to figure it out. Any help is much appreciated!

Update: This issue is specifically on MacOS sed package. The script works fine on Ubuntu sed. — Mohit Sharma, Jun 14 '19 at 04:28

Alex Harvey · Accepted Answer · 2019-06-18T13:22:45.177

Note that you probably should consider using a tool like jq for editing JSON files. But I assume you have a good reason for using sed, so you have a couple of problems there.

The first is that you're trying to use GNU sed features on your Mac OS X version of sed that doesn't have those features. If you want GNU sed on Mac OS X, then install it:

▶ brew install gnu-sed

Fixing up your code for GNU sed (and also for other Bash style guide recommendations about quoting strings):

cat > FILE <<EOF
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
EOF

placeLocation=myPlaceLocation
vehicleHeading=myVehicleHeading
message=myMessage

file=FILE

gsed -i -e '/location/c\' -e '"location": "'"$placeLocation"'",' "$file"
gsed -i -e '/heading/c\' -e '"heading": "'"$vehicleHeading"'",' "$file"
gsed -i -e '/message/c\' -e '"message": "'"$message"'",' "$file"

As noted in the GNU sed manual, use of multiple -e commands on the same line with the c\ command is a GNU extension.

If you want to use Mac OS X's sed, just may be able to write it this way:

sed -i '' '
  s/"location".*/"location": "'"$placeLocation"'",/
  s/"heading".*/"heading": "'"$vehicleHeading"'",/
  s/"message".*/"message": "'"$message"'",/
' "$file"

But note you would have to sanitise the input if you need the code to be robust to all inputs.

Try that with `message='what is 1/2?'` or `message='what is 1&2?'` or `message='what is \1?'` or .... You can't do this job robustly with sed unless you sanitize the text first, see https://stackoverflow.com/q/29613304/1745001. — Ed Morton, Jun 17 '19 at 15:03
Right @EdMorton I did say it's best to use JQ for this whereas the OP's question seems to be asking about confusions about sed's syntax. — Alex Harvey, Jun 18 '19 at 13:24
That's fine but when there's non-obvious chars or strings that need to be avoided in a solution and reasonable cause to think the OP might encounter them (e.g. with field names like "message") then it's good to give the OP a heads up about that constraint. — Ed Morton, Jun 18 '19 at 13:28

Ed Morton · Answer 2 · 2019-06-17T15:24:24.087

To do this robustly you need to use a tool that understands literal strings (which sed doesn't - see Is it possible to escape regex metacharacters reliably with sed) e.g. awk:

$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1/2?' one.txt
"message":"what is 1/2?"

$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1&2?' one.txt
"message":"what is 1&2?"

$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is \1?' one.txt
"message":"what is \1?"

the above will work robustly using any awk in any shell on every UNIX box. Try using those as replacement strings in a sed command.

The full script to do what you want would be:

#!/bin/env bash

# Shell script to verify the end to end D1 request flow
placeLocation=$1 
vehicleHeading=$2
message=$3
file=one.txt
tmp=$(mktemp)

awk '
    BEGIN {
        split("location heading message", tags)
        for (i in tags) {
            vals["\"" tags[i] "\""] = "\"" ARGV[i] "\""
            ARGV[i] = ""
        }
        FS=OFS=":"
    }
    $1 in vals {
        tag = $1
        sub(FS".*","")
        $0 = tag OFS vals[tag]
    }
1' "$placeLocation" "$vehicleHeading" "$message" "$file" > "$tmp" && mv "$tmp" "$file"

If you had jq and valid json then you should but the OP is asking about a standard UNIX tool, sed, instead so I'm showing him why if he's restricted to standard UNIX tools he should use awk instead (or update his shell script to sanitize his text before passing it to sed but IMHO escaping all metachars in the text to make it act like a literal string is less attractive than just using literal strings) — Ed Morton, Jun 18 '19 at 13:26

Trying to swap JSON element using sed editor on MAC

2 Answers2