Find and replace in JSON files

Question

I am new to scripting.

I have the below three files:

Availability.json
Source URLs.txt
Target URLs.txt

My requirement is I need to search for URLs present in Source URLs.txt and replace them with Target URLs.txt in Availability.json file.

How can we achieve this in shell scripting? We have some 50 URLs.

Content: Availability.json

"drillDownUrl": "https://appdync-mex-cgn-2u.mex.group.net:8090/controller/#/location=METRIC_BROWSER&timeRange=last_30_minutes.BEFORE_NOW.-1.-1.180&application=1000&metrics=APPLICATION_COMPONENT.647.4743559",

         "label": null,
        "description": null,
        "drillDownUrl": "https://appdync-mex-cpo-7u.mex.group.net:8090/controller/#/location=METRIC_BROWSER&timeRange=last_30_minutes.BEFORE_NOW.-1.-1.180&application=1&metrics=APPLICATION_COMPONENT.647.474",

Source_URLs.txt

 https://appdync-mex-cgn-2u.mex.group.net:8090/controller/#/location=METRIC_BROWSER&timeRange=last_30_minutes.BEFORE_NOW.-1.-1.180&application=1000&metrics=APPLICATION_COMPONENT.647.4743559

https://appdync-mex-cpo-7u.mex.group.net:8090/controller/#/location=METRIC_BROWSER&timeRange=last_30_minutes.BEFORE_NOW.-1.-1.180&application=1&metrics=APPLICATION_COMPONENT.647.474

Target_URLs.txt:

https://www.apdyn.com/Application#=12
https://www.appdyn.com/Application#=123

You'll need to break the problem down a bit and show us also what's in some of those files. — Alex Harvey, Jul 25 '18 at 01:38
In the json file we have some 50 URLs similar to http://appdyn.net/Application=12/components=12345....In Source_URLs.txt we have all 50 URLs which are present in json and in Target_URLs.txt we have URLs to be replaced...Need to replace all 50 URLs from Target_URLs file injson file...hope it make sense... — Arun kumar, Jul 25 '18 at 04:41
You need to show a sample of your JSON and an example of the source and target URL. That link you pasted here doesn't work. — Alex Harvey, Jul 25 '18 at 05:16
What shell are you using? Bash? Do you need it to work with just your shell, or are you aiming for wide compatibility? It is good to be specific here. — halfer, Jul 25 '18 at 06:47
Please read [Under what circumstances may I add “urgent” or other similar phrases to my question, in order to obtain faster answers?](//meta.stackoverflow.com/q/326569) - the summary is that this is not an ideal way to address volunteers, and is probably counterproductive to obtaining answers. Please refrain from adding this to your questions. — halfer, Jul 25 '18 at 06:47
Thanks for the reponse. We will be using bash and no aim for other. — Arun kumar, Jul 26 '18 at 01:15

score 1 · Answer 1 · answered Jul 26 '18 at 08:37

Updated version - readarray used as suggested and separators in sed are now |. I used your filenames. Please remove blank line and leading space in your Source_URLs.txt

readarray -t SRC < Source_URLs.txt
echo "${SRC[1]}"

readarray -t TRGT < Target_URLs.txt
echo "${TRGT[1]}"

for (( i=0; i<${#SRC[@]}; i++ ));
do
  sed -i "s|${SRC[$i]}|${TRGT[$i]}|g" Availability.json
done

Note: Your Availability.json is not valid - last character is , there is duplicate key and there's no object - check it in https://jsonformatter.curiousconcept.com/

Add this line at the beginning of the script to get rid of the blank lines from **Source_URLs.txt:** `sed -i "/^[[:space:]]*$/d" Source_URLs.txt` — devd, Jul 28 '18 at 04:08

score 0 · Answer 2 · answered Jul 25 '18 at 08:26

0

load your urls into arrays (check https://stackoverflow.com/a/11393884/2235381) According to your OS IFS may vary.

IFS=$'\r\n' GLOBIGNORE='*' command eval  'SRC=($(cat src.txt))'
echo "${SRC[0]}"

IFS=$'\r\n' GLOBIGNORE='*' command eval  'TRGT=($(cat trgt.txt))'
echo "${TRGT[0]}"

backup your json and then iterate over array with sed -i, e.g.:

for (( i=0; i<${#SRC[@]}; i++ ));
do
  sed -i "s/${SRC[$i]}/${TRGT[$i]}/g" your.json
done

I know it's bit brute force, but for 50 lines ok. Suggest better solution if you have.

answered Jul 25 '18 at 08:26

lojza

1,823
2
13
23

Thanks for the idea...I will give a try – Arun kumar Jul 25 '18 at 22:26
Getting the below error, can you help? sed: -e expression #1, char 11: unknown option to `s' sed: -e expression #1, char 11: unknown option to `s' – Arun kumar Jul 26 '18 at 00:41
Updated my question with file content, please enlighten me – Arun kumar Jul 26 '18 at 01:16
1

URLs have `/`s in them. You can't use `s/$foo/$bar/` when `$foo` and `$bar` have slashes. (This will also misbehave for any URL that uses characters that parse as regular expressions; for example, any `.` in a URL will be treated as a wildcard, matching any character). – Charles Duffy Jul 26 '18 at 01:20
1

...and the `eval` usage here is major-security-bugs level unfortunate, and utterly unnecessary. `readarray -t src – Charles Duffy Jul 26 '18 at 01:23
Thanks for the update...can you let me know how to use sed for URLs? – Arun kumar Jul 26 '18 at 01:48

Find and replace in JSON files

2 Answers2