How to use jq as SED for large json file (optionally with a mapping table)

Question

I have a large json file which contains several IPs within json files. I want to replace specific IP addresses within the json file (without having to write the putout into a new file). The equivalent SED command would be sed -i 's/old_ip/new_ip/g'

The structure is as follows (of a single json entry in the json file):

{

   "destination":{
      "port":53,
      "bytes":100,
      "ip":"1.1.1.1"
   },
   "source":{
      "port":54894,
      "bytes":84,
      "ip":"10.1.1.49",
   },
   "related":{
      "ip":"10.5.5.45"
   },
   "event":{
      "code":"0000000013",
      "action":"accept",
      "type":[
         "allowed",
         "connection",
         "end",
         "protocol"
      ],
      "outcome":"success"
   },
   "@timestamp":"2021-08-29T04:47:10.000+02:00"
}

I have now two problems which I can't figure to solve.

Problem 1: I want to replace every IP by a corresponding IP address. So imagine I want to replace the IP 10.1.1.49 with 10.10.10.10

I know I can use the command jq '(.. | .ip?)' | select(.ip == "10.2.1.49")| .ip |= "10.10.10.10"' but unfortunately this just outputs the new IP instead of replacing it in the file like SED does (this refers to all solutions I found on the web).

Problem 2: Now suppose I have a mapping table (mapping.json) with the following structure:

{
 "10.2.1.49":"10.10.10.10",
 "10.15.15.15":"10.10.10.10"
}

Can I provide and combine this mapping table with jq for a find and replace? I have the feeling that this is not possible in jq meaning I would need to figure out my first problem and then writing a bash script. Someone had similar question (see here) by using --sluprfile. But apparently I can't use the solution from this link as a ID or unique value for selection is required which I don't have in my json.

Is there a simple solution or should I just use SED combined with a shellscript? Thanks

pmf · Accepted Answer · 2022-01-11T11:53:23.670

Re Problem 1

You can update the relevant IP addresses using

jq '(.. | .ip? | select(. == "10.1.1.49")) |= "10.10.10.10"'

{
  "destination": {
    "port": 53,
    "bytes": 100,
    "ip": "1.1.1.1"
  },
  "source": {
    "port": 54894,
    "bytes": 84,
    "ip": "10.10.10.10"
  },
  "related": {
    "ip": "10.5.5.45"
  },
  "event": {
    "code": "0000000013",
    "action": "accept",
    "type": [
      "allowed",
      "connection",
      "end",
      "protocol"
    ],
    "outcome": "success"
  },
  "@timestamp": "2021-08-29T04:47:10.000+02:00"
}

Demo

But you cannot update the file as you can with sed -i - at least not with just using a flag. Using temporary files or sponge etc. will, however, work fine.

Re Problem 2

Read in the mappings e.g. using --argfile and update using a lookup in combination with the alternative operator // to fall back to the original value, if it doesn't exist in the mapping

jq --argfile mapping mapping.json '(.. | .ip? // empty) |= ($mapping[.]? // .)'

Demo

Thank you very muchm it id its job. However I haven, how do I have to edit the command if the IPs happen to be inside an array like: "related": { "ip": [ "111.1621.131", "107.11.242.204" ] ? — programmerskitchen, Jan 17 '22 at 13:34
@programmerskitchen Use `arrays[]` to filter for arrays and select its elements, and add that before the update: [Problem 1](https://jqplay.org/s/q2fFgZSx5V) [Problem 2](https://jqplay.org/s/J9LQLngCR2). If you want both (finding `.ip` **and** `.ip[]`) then include the original input as well by adding `.`: [Problem 1](https://jqplay.org/s/wV-wvosuMA) [Problem 2](https://jqplay.org/s/02sdsPOPCf). — pmf, Jan 17 '22 at 14:13

How to use jq as SED for large json file (optionally with a mapping table)

1 Answers1

Re Problem 1

Re Problem 2