JQ create json array using bash

Question

I am currently trying to get information about my file hosting accounts. As I keep a lot of my backup media on different accounts. I am using megatools to query information about the account, which I then parse into an array. The array is then flattened using \n for raw input.

The script works wonderfully but its not creating valid json. I am not sure what I am missing to make it valid. Thanks for the help in advance.

script

function join_by { local IFS="$1"; shift; echo "$*"; }

while IFS='' read -r line || [[ -n "$line" ]]; do
    IFS=, read -ra array <<< "$line"

    nickname=${array[0]}
    user=${array[1]}
    pass=${array[2]}
    data=($(megadf --username=$user --password=$pass))
    data[${#data[@]}]+="$nickname"

    stats=$(join_by $'\n' ${data[@]})

    echo $stats | jq --slurp --raw-input 'split("\n")[:-1] | map([ split(" ")[] ]) | map({
    nick: .[6],
    total: .[1],
    used: .[3],
    free: .[5]
    })' >> /opt/stats/json/accounts.json

done < .accounts

json output

[
  {
    "nick": "alt",
    "total": "53687091200",
    "used": "7885201595",
    "free": "45801889605"
  }
]
[
  {
    "nick": "main",
    "total": "214748364800",
    "used": "87240914483",
    "free": "127507450317"
  }
]

What should be

[
  {
    "nick": "alt",
    "total": "53687091200",
    "used": "7885201595",
    "free": "45801889605"
  },
  {
    "nick": "main",
    "total": "214748364800",
    "used": "87240914483",
    "free": "127507450317"
  }
]

.accounts

nickname,user,pass

You shouldn't call `jq` inside the loop. Pipe the output of the loop to `jq`, so it can create a JSON array of everything. — Barmar, Aug 30 '18 at 00:33
`data[${#data[@]}]+="$nickname"` would be more clearly (and more correctly, should an array be sparse) written as `data+=( "$nickname" )`. — Charles Duffy, Aug 30 '18 at 00:43

score 1 · Answer 1 · answered Aug 30 '18 at 00:40

1

@barmar found the simplest solution that I overlooked....

Needed to pipe the loop into jq

while IFS='' read -r line || [[ -n "$line" ]]; do
    IFS=, read -ra array <<< "$line"
    nickname=${array[0]}
    user=${array[1]}
    pass=${array[2]}
    data=($(megadf --username=$user --password=$pass))
    data[${#data[@]}]+="$nickname"
    stats=$(join_by $'\n' ${data[@]})
    echo $stats
done < .accounts  | jq --slurp --raw-input 'split("\n")[:-1] | map([ split(" ")[] ]) | map({
    nick: .[6],
    total: .[1],
    used: .[3],
    free: .[5]
    })' >> /opt/stats/json/accounts.json

answered Aug 30 '18 at 00:40

Ryahn

537
7
24

3

Consider running your code through http://shellcheck.net/ -- there are a bunch of quoting errors in here. Look at what happens if a user has `hi * world` as their password -- the `*` will be replaced with a list of filenames in the current directory. Similarly, `echo $stats` is removes the newlines you just put in with `join_by`, by splitting on all whitespace, passing each word as a separate argument to `echo`, and letting `echo` re-join content with regular spaces. – Charles Duffy Aug 30 '18 at 00:44
@CharlesDuffy Thank you for that tool. Going to help fix all my own code, than having to pester all the wonderful people here <3 – Ryahn Aug 30 '18 at 01:36

JQ create json array using bash

1 Answers1

Linked