How to Iterate Json key value pair and get the value and store in another json using Bash Script with JQ utility

Question

Im writing bash script for AWS Secret Manager .

Question:

I have one aws secret manager it will contain multiple secret

Example: aws secretsmanager get-secret-value --secret-id Example | jq -r ".SecretString"

input.json

{"Username":"admin","Password":"admin","Endpoint":"localhost","DatabaseName":"example","Port":"3306"}

Using this command got a json output

Now I want to extract All values like (Username , password,..)

After get all extract values add in another json file called (secondfile.json)

I have another json file it will contain

this is secondfile.json

 {
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": {
    "Username": "",
    "Password": "",
    "Endpoint": "",
    "DatabaseName": "",
    "Port": ""
  }
}

Sample Output: extracted values now stored second file.json file in Aurora[] array elements.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": {
    "Username": "admin",
    "Password": "admin",
    "Endpoint": "localhost",
    "DatabaseName": "example",
    "Port": "3306"
  }
}

Need Help on do with bash script .

Note: Input.json value not fixed , it may be contain 3 json key/value or n number of key/value

This might be helpful: https://stackoverflow.com/questions/42716734/modify-a-key-value-in-a-json-using-jq-in-place — ExceptionNotThrownException, Oct 13 '22 at 09:04
But output should be stored in another json value with correct key and value — Muthu, Oct 13 '22 at 09:08
I understand. You have to pick these values first, then create a temporary file where You store them and lastly You move them into the desired file. Jq can not write to th file. You have to use the shell commands. — ExceptionNotThrownException, Oct 13 '22 at 09:10
I'm having a hard time understanding the question. What is your goal and the final output? What for do you need the output? — knittl, Oct 13 '22 at 09:16
Hi @knittl I have input.json file and get all the value and store in to output,.json file — Muthu, Oct 13 '22 at 09:20
{ "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning" } }, "Aurora": { "Username": "admin", "Password": "admin", "Endpoint": "localhost", "DatabaseName": "example", "Port": "3306" } } This is the exact output — Muthu, Oct 13 '22 at 09:24
What should happen when keys are missing from the input or from the expected output? e.g. `secondfile.json` contains a key "Environment", but the `input.json` file does not? Similarly, what should happen with keys in `input.json` that are not listed in `secondfile.json`? You only specified _that_ it can happen, but not _what_ should happen in that case. What would the expected output be? — knittl, Oct 13 '22 at 10:46
Is `input.json` created from the _output_ of the `aws` command? — glenn jackman, Oct 13 '22 at 13:17
@glennjackman yes i have created json from that command , which ever value i have got need to add in secondfile.json file — Muthu, Oct 13 '22 at 19:00

score 0 · Answer 1 · answered Oct 13 '22 at 09:24

0

If I understand your question correctly, you simply want to embed/wrap the input document in a top level document.

echo '{"Username":"admin","Password":"admin","Endpoint":"localhost","DatabaseName":"example","Port":"3306"}' | jq '{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": .
}'

Output:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": {
    "Username": "admin",
    "Password": "admin",
    "Endpoint": "localhost",
    "DatabaseName": "example",
    "Port": "3306"
  }
}

The key "Aurora" will contain your input document. If you need only specific fields, then replace . with {Username,Password}

answered Oct 13 '22 at 09:24

knittl

246,190
53
318
364

Hi @knittl . Im not just echo and embedded the value , Input json file iterate and get all value and store in to output.json file in aurora json value should be in dynamic in nature – Muthu Oct 13 '22 at 09:27
@Muthu I don't understand. If you have an input file redirect with ` – knittl Oct 13 '22 at 09:30
Please check now let me know if you are able to unserstand – Muthu Oct 13 '22 at 09:47
Please let me know if you are able to understand – Muthu Oct 13 '22 at 10:01

score 0 · Answer 2 · answered Oct 13 '22 at 10:24

Here is one solution how to jq's --slurpfile option to use another file as "template" to format your input and then replace the parts according to your requirements:

$ cat template.json
{
    "Logging": {
        "LogLevel": {
            "Default": "Information",
            "Microsoft.AspNetCore": "Warning"
        }
    },
    "Aurora": {
        "Username": "",
        "Password": "",
        "Endpoint": "",
        "DatabaseName": "",
        "Port": "",
        "Environment": ""
    }
}

$ cat input.json
{
  "Username": "admin",
  "Password": "admin",
  "Endpoint": "localhost",
  "DatabaseName": "example",
  "Port": "3306",
  "SomethingElse": "ignore me"
}

$ jq --slurpfile tpl template.json '$tpl[0] + {Aurora: with_entries(select(.key | IN($tpl[0].Aurora|keys[])))}' <input.json
{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": {
    "Username": "admin",
    "Password": "admin",
    "Endpoint": "localhost",
    "DatabaseName": "example",
    "Port": "3306"
  }
}

Or swapping template and input files:

$ jq --slurpfile in input.json '.Aurora |= keys as $keys | $in[0] | with_entries(select(.key | IN($keys[])))' <template.json
{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": {
    "Username": "admin",
    "Password": "admin",
    "Endpoint": "localhost",
    "DatabaseName": "example",
    "Port": "3306"
  }
}

score 0 · Answer 3 · answered Oct 13 '22 at 10:34

If you can change your "template" file, I would suggest not keeping it as JSON, but make it a jq program instead:

$ cat input.json
{
  "Username": "admin",
  "Password": "admin",
  "Endpoint": "localhost",
  "DatabaseName": "example",
  "Port": "3306",
  "SomethingElse": "ignore me"
}

$ cat filter.jq
{
    "Logging": {
        "LogLevel": {
            "Default": "Information",
            "Microsoft.AspNetCore": "Warning"
        }
    },
    "Aurora": {
        "Username": .Username,
        "Password": .Password,
        "Endpoint": .Endpoint,
        "DatabaseName": .DatabaseName,
        "Port": .Port
    }
}

$ jq -f filter.jq < input.json
{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Aurora": {
    "Username": "admin",
    "Password": "admin",
    "Endpoint": "localhost",
    "DatabaseName": "example",
    "Port": "3306"
  }
}

Note that the filter program can be written with shorthand syntax:

$ cat filter.jq
{
    "Logging": {
        "LogLevel": {
            "Default": "Information",
            "Microsoft.AspNetCore": "Warning"
        }
    },
    "Aurora": {
        Username,
        Password,
        Endpoint,
        DatabaseName,
        Port
    }
}

How to Iterate Json key value pair and get the value and store in another json using Bash Script with JQ utility

3 Answers3