Editing a JSON parameter file on MacOS

Question

Obviously in a Unix environment using .*$ will match any number of characters till the end of the line.

However, when executing a sed using

"ParameterValue": ".*$"

as my regex on a mac, I get the error

sed: 1: "s/"ParameterValue": ".* ...": unterminated substitute in regular expression

I expect it to match against "ParameterValue": "buildname" where buildname could be any of three dozen different options in different formats, preventing a more exact regex.

The script will always be run from a mac environment so I have to have a solution that will work on a mac.

TL;DR: How do I match till the end of the line in bash on a mac?

EDIT: By request, my script. For security reasons I can only show this section. Image and grepname are both defined in earlier lines.

REPLACE='"ParameterValue":.*'
REPLACETO='"ParameterValue": "${IMAGE}"'

grep -A 1 -i "RTSMImageTag" ${GREPNAME}_parameters.json | xargs sed -i '' -E "s/$REPLACE/$REPLACETO"

And an excerpt from the relevant file;

"ParameterKey": "RTSMImageTag",
"ParameterValue": "buildname"

Essentially I need to replace this particular parameter value which is located in a newly generated file with one passed into the script. I was trying to avoid asking for someone to solve my problems for me by focusing on just the error, but if you have a better solution overall please let me know.

@anubhava Identical error; `sed: 1: "s/"ParameterValue":.*/" ...": unterminated substitute in regular expression` — Alex, Dec 06 '16 at 18:08
@Jean-FrançoisFabre Apologies but I'm quite new to bash and scripting in general. Could you elaborate on what you mean by simple quotes? — Alex, Dec 06 '16 at 18:09
It would appear you aren't completing the `s` command: `s/"ParameterValue": ".*$"/whatever/`. The trailing `/` is not optional. As an aside, don't use `sed` to process JSON; use a proper JSON processor like `jq`. — chepner, Dec 06 '16 at 18:16
@Alex: Can you show us your input file to be clear on your requirement? and what is that you want to retrieve? — Inian, Dec 06 '16 at 18:17
@chepner You're quite right! That's my bad, forgot to replace it from a previous edit, silly me. I'm now getting the error `sed: ParameterKey:: No such file or directory` - is this the same root issue do you think, or was I just an idiot and this new bit is unrelated? — Alex, Dec 06 '16 at 18:26
It's unrelated. `xargs` is not passing the values to `sed` via standard input, but as an argument, which means `sed` interprets it as the name of a file to open, not as the text to operate on. — chepner, Dec 06 '16 at 18:29
@chepner: That makes sense! I'm still learning this stuff so thank you for explaining. I suppose I'll just delete this question as the answer is "it's a typo", but I wanted to say thank you for your patience first! — Alex, Dec 06 '16 at 18:32
`sed` regex syntax is different from bash regex syntax (the former is POSIX BRE, the latter is POSIX ERE). — Charles Duffy, Dec 06 '16 at 18:35
BTW, generally speaking, when you're asked for your script, what we really want is a [**M**inimal, **C**omplete, **V**erifiable **E**xample](http://stackoverflow.com/help/mcve) -- that is, a chunk of code that anyone can copy-and-paste (without needing to have files that exist, variables set up, etc) that reproduces the problem (but is the smallest possible thing that will do so, with unrelated contents factored out). Creating a MCVE will generally also fulfil your confidentiality requirements, by virtue of being so tightly focused on demonstrating the question at hand. — Charles Duffy, Dec 06 '16 at 18:37
@CharlesDuffy Makes sense, thank you for explaining! As you can tell I'm new here and absolutely terrified of screwing up... I'll try and be better about posting scripts in the future. — Alex, Dec 06 '16 at 18:43

score 4 · Accepted Answer · edited Dec 06 '16 at 18:43

4

Do not use grep. Do not use sed. Use jq.

$ cat tmp.json
{
  "ParameterKey": "RTSMImageTag",
  "ParameterValue": "buildname"
}
$ jq --arg image "FOO" 'select(.ParameterKey == "RTSMImageTag") | .ParameterValue = $image' tmp.json
{
  "ParameterKey": "RTSMImageTag",
  "ParameterValue": "FOOO"
}

edited Dec 06 '16 at 18:43

mklement0

382,024
64
607
775

answered Dec 06 '16 at 18:33

chepner

497,756
71
530
681

Could you point me in the direction of reading on jq? I honestly have no idea what you're proposing but I'm willing to research. – Alex Dec 06 '16 at 18:46
1

@mklement0 kindly edited my answer to provide a link to the `jq` home page. – chepner Dec 06 '16 at 18:47
Ah...I see it's a download. This script needs to be able to run from any of a number of macs without any setup being required. Is there any way to call it via url or something similar rather than needing a local download? (and if this is out of the scope of this SO question I apologize, I don't know what is/isn't allowed here) – Alex Dec 06 '16 at 18:51
@Alex, ...are those systems guaranteed to have a Python interpreter, by any chance? Python has a JSON module built in, and we have several similar questions and answers already in our knowledgebase where using a short Python scriptlet from shell for this kind of edit is already demonstrated. – Charles Duffy Dec 06 '16 at 18:52
3

Frankly, if you plan on processing JSON, part of the setup should include the proper tools. Too many people assume that the set of tools chosen as the "standard" set 30 years ago are still sufficient for data formats introduced decades later. – chepner Dec 06 '16 at 18:54
Yes and no - this will eventually be set up as a Jenkins 2 job and I'm trying to avoid any compatibility issues from the start. All the local computers have python, but I'd be hard pressed to guarantee that the jenkins slaves do/will. – Alex Dec 06 '16 at 18:57
What OS/distro are you using for the slaves? It's exceedingly unusual to have something without an interpreter -- except in single-purpose mini-distros it's pretty much ubiquitous within scripts used to operate the distro itself; this is true in both Red Hat and Debian-derived distro families. Arch has gone ahead of the rest of the world by using Python 3 rather than Python 2 for its internal use, but it's Python still, and has the JSON module still. – Charles Duffy Dec 06 '16 at 18:58
...and frankly, if you don't have a real JSON parser/generator, the fix is to install one, not to approximate one. Approximations will (not *may*, but *will*) be buggy. – Charles Duffy Dec 06 '16 at 18:59
I believe they're all linux boxes but that's handled by my boss and I've yet to be caught up to speed. I think my course of action will be to touch base with him and see if we can include jq in the setup, and if so proceed with your script. Either way thank you for your help, hopefully I can come back and successfully mark your reply as the answer! – Alex Dec 06 '16 at 19:02

score 3 · Answer 2 · answered Dec 06 '16 at 19:05

3

Using Python:

# this is a shell function that wraps a short Python script
# usage: update_json key value <in.json >out.json
update_json() {
  python -c '
import sys, json
content = json.load(sys.stdin)
content[sys.argv[1]] = sys.argv[2]
json.dump(content, sys.stdout)
sys.stdout.write("\n")
' "$@"
}

Thereafter:

update_json ParameterValue NewValue <<<'{"ParameterKey": "RTSMImageTag", "ParameterValue": "buildname"}'

...properly emits:

{"ParameterValue": "NewValue", "ParameterKey": "RTSMImageTag"}

answered Dec 06 '16 at 19:05

Charles Duffy

280,126
43
390
441

++ for a handy alternative in cases where installing `jq` is not an option. An explanation for why the output's key ordering differs can be found [here](http://stackoverflow.com/a/23820416/45375), where a solution that preserves input key ordering (which isn't necessary for the _correctness_ of the output) is also presented. – mklement0 Dec 06 '16 at 20:48

Editing a JSON parameter file on MacOS

2 Answers2