How delete last comma in json file using Bash?

Question

I wrote some script that takes all user data of aws ec2 instance, and echo to local.json. All this happens when I install my node.js modules. I don't know how to delete last comma in the json file. Here is the bash script:

#!/bin/bash
export DATA_DIR=/data
export PATH=$PATH:/usr/local/bin

#install package from git repository
sudo -- sh -c "export PATH=$PATH:/usr/local/bin; export DATA_DIR=/data; npm install git+https://reader:secret@bitbucket.org/somebranch/$1.git#$2"

#update config files from instance user-data
InstanceConfig=`cat /instance-config`
echo '{' >> node_modules/$1/config/local.json
while read line
do
   if [ ! -z "$line" -a "$line" != " " ]; then
      Key=`echo $line | cut -f1 -d=`
      Value=`echo $line | cut -f2 -d=`
      if [ "$Key" = "Env" ]; then
         Env="$Value"
      fi
      printf '"%s" : "%s",\n' "$Key" "$Value" >> node_modules/*/config/local.json
   fi
done <<< "$InstanceConfig"
sed -i '$ s/.$//' node_modules/$1/config/local.json
echo '}' >> node_modules/$1/config/local.json

To run him im doing that way: ./script I get json(OUTPUT), but with comma in all lines. Here is local.json that I get:

{
    "Env" : "dev",
    "EngineUrl" : "engine.url.net",
}

All I trying to do, is delete in last line of the json file - comma(","). I try many ways, that I found in internet. I know that it should be after last "fi"(end of the loop). And I know that it should be something like this line:

sed -i "s?${Key} : ${Value},?${Key} : ${Value}?g" node_modules/$1/config/local.json

Or this:

sed '$ s/,$//' node_modules/$1/config/local.json

But they not work for me. Can someone help me with that? Who knows Bash scripting well? Thanks!

Answer 1

If you know that it is the last comma that needs to be replaced, a reasonably robust way is to use GNU sed in "slurp" mode like this:

sed -zr 's/,([^,]*$)/\1/' local.json

Output:

{
  "Env" : "dev",
  "EngineUrl" : "engine.url.net"
}

Answer 2

If you'd just post some sample input/output it'd remove the guess-work but IF this is your input file:

$ cat file
Env=dev
EngineUrl=engine.url.net

Then IF you're trying to do what I think you are then all you need is:

$ cat tst.awk
BEGIN { FS="="; sep="{\n" }
{
    printf "%s    \"%s\" : \"%s\"", sep, $1, $2
    sep = ",\n"
}
END { print "\n}" }

which you'd execute as:

$ awk -f tst.awk file
{
    "Env" : "dev",
    "EngineUrl" : "engine.url.net"
}

Or you can execute the awk script inline within a shell script if you prefer:

awk '
BEGIN { FS="="; sep="{\n" }
{
    printf "%s    \"%s\" : \"%s\"", sep, $1, $2
    sep = ",\n"
}
END { print "\n}" }
' file
{
    "Env" : "dev",
    "EngineUrl" : "engine.url.net"
}

The above is far more robust, portable, efficient and better in every other way than the shell script you posted because it's using the right tool for the job. A UNIX shell is an environment from which to call tools with a language to sequence those calls. It is NOT a language to process text which is why it's so difficult to get it right. The UNIX tool for general text processing is awk so when you need to process text in UNIX, you just have shell call awk, that's all.

Answer 3

Here a jq version if it's available:

jq --raw-input 'split("=") | {(.[0]):.[1]}' /instance-config | jq --slurp 'add'

There might be a way to do it with one jqpass, but I couldn't see it.

Answer 4

You an remove all trailing commas from invalid json with:

sed -i.bak ':begin;$!N;s/,\n}/\n}/g;tbegin;P;D' FILE

• sed -i.bak = creates a backup of the original file, then applies changes to the file

• ':begin;$!N;s/,\n}/\n}/g;tbegin;P;D' = anything ending with , followed by
  "new line and }". Remove the , on the previous line

• FILE = the file you want to make the change to

Answer 5

If you're willing to use it, xidel is rather forgiving for trailing commas:

xidel -s local.json -e '$json'
{
  "Env": "dev",
  "EngineUrl": "engine.url.net"
}

xidel - -se '$json' <<< '{"Env":"dev","EngineUrl":"engine.url.net",}'
#or
xidel - -se 'parse-json($raw,{"liberal":true()})' <<< '{"Env":"dev","EngineUrl":"engine.url.net",}'
{
  "Env": "dev",
  "EngineUrl": "engine.url.net"
}