Replacing a value in a JSON structure with randomly-chosen strings from a list using jq

Question

There are lots of similar questions but none for dynamically joining 2 files. What I'm trying to do is to dynamically edit the following structure:

{
  "features": [
    {
      "type": "Feature",
      "properties": {
        "name": "0",
        "height": 0.7
      }
    },
    {
      "type": "Feature",
      "properties": {
        "name": "1",
        "height": 0
      }
    }
  ]
}

I want to replace only the one field .features[].properties.name with a random value from a 1d-array inside another txt file. There are 8,000 features and around 100 names I've prepared.

This is what I've got now failing with errors:

#!/bin/bash
declare -a names=("name1" "name2" "name3")
jq '{
    "features" : [
        "type" : "Feature",
        "properties" : {
            "name" : `$names[seq 0 100]`,
            "height" : .features[].properties.height
        },
        .features[].geometry
    ]
}' < areas.json

Is it even possible to do in a single command or I should use python or js for such tasks?

Answer 1

Your document (https://echarts.baidu.com/examples/data-gl/asset/data/buildings.json) is actually small enough that we don't need to do any crazy memory-conservation tricks to make it work; the following functions as-is:

# create sample data
[[ -e words.txt ]] || printf '%s\n' 'First Word' 'Second Word' 'Third Word' >words.txt

# actually run the replacements
jq -n --slurpfile buildings buildings.json '
  # define a jq function that changes the current property name with the next input
  def replaceName: (.properties.name |= input);
  # now, for each document in buildings.json, replace each name it contains
  $buildings[] | (.features |= map(replaceName))
' < <(shuf -r words.txt | jq -R .)

This works because shuf -r words.txt creates an unending stream of words randomly chosen from words.txt, and the jq -R . inside the process substitution quotes those as strings. (Because we only call input once per item in buildings.json, we don't try to keep running after that file's contents have been completely consumed).

---

For the tiny two-record document given in the question, the output looks like:

{
  "features": [
    {
      "type": "Feature",
      "properties": {
        "name": "Third Word",
        "height": 0.7
      }
    },
    {
      "type": "Feature",
      "properties": {
        "name": "Second Word",
        "height": 0
      }
    }
  ]
}

...with the actual words varying each run; it's similarly been smoketested with the full externally-hosted file.

Answer 2

Here's a solution to the problem of choosing the names randomly with replacement, using the very simple PRNG written in jq copied from https://rosettacode.org/wiki/Random_numbers#jq

Invocation:

jq  --argjson names '["name1","name2","name3","name4"]' \
  -f areas.jq areas.json

areas.jq

# The random numbers are in [0 -- 32767] inclusive.
# Input: an array of length at least 2 interpreted as [count, state, ...]
# Output: [count+1, newstate, r] where r is the next pseudo-random number.
def next_rand_Microsoft:
  .[0] as $count | .[1] as $state
  | ( (214013 * $state) + 2531011) % 2147483648 # mod 2^31
  | [$count+1 , ., (. / 65536 | floor) ] ;

# generate a stream of random integers < $n
def randoms($n):
  def r: next_rand_Microsoft
    | (.[2] % $n), r;
  [0,11] | r ;


. as $in
| ($names|length) as $count
| (.features|length) as $n
| [limit($n; randoms($count))] as $randoms
| reduce range(0; $n) as $i (.;
    .features[$i].properties.name = $names[$randoms[$i]] )

Answer 3

Assuming your areas.json is valid JSON, then I believe the following would come close to accomplishing your intended edit:

names='["name1","name2","name3","name4"]'
jq --argjson names "$names" '.features[].properties.name = $names
  ' < areas.json

However, given your proposed solution, it's not clear to me what you mean by a "random value from a 1d-array". If you mean that the index should be randomly chosen (as by a PRNG), then I would suggest computing it using your favorite PRNG and passing in that random value as another argument to jq, as illustrated in the following section.

So the question becomes how to transform the text

['name1','name2','name3','name4']

into a valid JSON array. There are numerous ways this can be done, whether using jq or not, but I believe that is best left as a separate question or as an exercise, because the selection of the method will probably depend on specific details which are not mentioned in this Q. Personally, I'd use sed if possible; you might also consider using hjson, as also illustrated in the following section.

Illustration using hjson and awk

hjson -j <<< "['name1','name2','name3','name4']" > names.json.tmp

function randint {
  awk -v n="$(jq length names.json.tmp)" '
    function randint(n) {return int(n * rand())}
    BEGIN {srand(); print randint(n)}'
}

jq --argfile names names.json.tmp --argjson n $(randint) '
  .features[].properties.name = $names[$n]
' < areas.json

Addendum

Currently, jq does not have a builtin PRNG, but if you want to use jq and if you want a value from the "names" array to be chosen at random (with replacement?) for each occurrence of the .name field, then one option would be to pre-compute an array of the randomly selected names (an array of length features | length) using your favorite PRNG, and passing that array into jq:

jq --argjson randomnames "$randomnames" ' 
  reduce range(0; .features[]|length) as $i (.;
    .features[$i].properties.name = $randomnames[$i]) 
  ' < areas.json

Another option would be to use a PRNG written in jq, as illustrated elsewhere on this page.