JQ: Enrich primary array with secondary array

Question

I have two arrays containing prices for items:

[{"id":"a","p":5},{"id":"b","p":7},{"id":"c","p":8}]

and

[{"id":"a","p":9},{"id":"b","p":7},{"id":"d","p":4}]

I want the primary array to be enriched with the prices from the secondary array, if available. If the price hasn't been updated, the entry should be removed. Thus the wanted output is ("b" removed because prices are equal and "c" kept):

[{"id":"a","p":5,"c":9},{"id":"c","p":8,"c":null}]

I tried using select but this removes items not available in both arrays:

echo '$pri' | jq -c --argjson sec '$sec' '.[]|. as {id:$id,p:$c}|{id,p,"c":($sec[]|select(.id==$id and .p!=$c).p)}'

EDIT: peak's answer brought me in the right direction. Once it was working I improved it even more.

The updated prices are always an array containing all items in an unsorted manner. I want to avoid disc writes when there's no real update.

Thus I save the last price response already in dictionary format like:

wget ... | jq -c 'reduce .[] as $x ({}; .[$x|.id|tostring]=$x)' >lastprices

This step converts the input array

{"id":28460,"price":"1.119","time":"2018-03-05T18:33:43Z","camp":null}
{"id":11438,"price":"1.119","time":"2018-03-05T18:58:26Z","camp":null}
{"id":11627,"price":"1.119","time":"2018-03-05T18:47:45Z","camp":null}
{"id":6905,"price":"1.119","time":"2018-03-05T19:03:59Z","camp":null}

to an index-sorted output array (a dictionary)

{ "6905"  : { "id":6905,  "price":"1.119", "time":"2018-03-05T19:03:59Z", "camp":null },
  "11438" : { "id":11438, "price":"1.119", "time":"2018-03-05T18:58:26Z", "camp":null },
  "11627" : { "id":11627, "price":"1.119", "time":"2018-03-05T18:47:45Z", "camp":null },
  "28460" : { "id":28460, "price":"1.119", "time":"2018-03-05T18:33:43Z", "camp":null } }

In the next iteration I compare the update.time != last.time and only keep (=select) items that have been updated:

wget ... | jq -c --argfile last lastprices '[.[]|select(.time!=$last[.id|tostring].time)]'

If the comparison evaluated a non-empty array I convert it to CSV and append it to the "database" (saving as JSON would require to rewrite the whole array every time):

echo "$COMPARISON" | jq -rc '(.[0] | keys_unsorted) as $keys | map([.[ $keys[] ]])[]|@csv'>>gasdb.csv

score 0 · Accepted Answer · answered Mar 01 '18 at 04:17

The requirement that "If the price hasn't been updated, the entry should be removed." seems to be at odds with the example, so the following assumes the example is definitive.

First, a small helper function for producing a (fast) lookup table:

def dict(f): reduce .[] as $x ({}; .[$x|f] = $x);

In the following, as in the text of the question, $pri refers to the array with the old prices, and $sec to the array with new prices:

($sec|dict(.id)) as $new
| reduce $pri[] as $x ([];
    $new[$x.id] as $y
    | if ($y.p == $x.p) then .
      else . + [$x | (.c = $y.p)]
      end )

The result is as given.

JQ: Enrich primary array with secondary array

1 Answers1