Add a new role to a Chef .json file with Sed

Question

I have a typical .json file for Chef e.g. servername.json

{

  "name": "myserver123",

  "chef_environment": "test",

  "run_list": [

    "role[base-pkg]",

    "role[interesting_stuff]",

    "role[user_apps]"


  ]

}

What I would like to do is use a "one liner" to add a new role after the last role found in the file. Since I never know what roles or how many are in a file i thought to search for the final closing "]" bracket and add the new role above that.

I tried the following:

tac servername.json | sed -i '0,/\]/a "role[My_New_Role]"'

thinking that this would find the (now) first "]" and add the new line after it. However when I run the cmd it adds the line "role[My_New_Role]" 3 times. Twice before the "]" and once in the correct place after the "]"

Questions:

1) Why is the line added 3 times when the "0" says match once ?

2) Would this be better done using AWK, Perl or Python (2.7.5) ? How ?

3) Should I use regex lookahead/behind instead of tac ?

4) Since I will need to process the file again to add a comma to the second to last role perhapes my whole approach is wrong ? What would be a better approach ?

You shouldn't use regexes or sed/awk for this task at all. Using a decent JSON parser is the only reliable way to parse and modify JSON data. — PerlDuck, Dec 19 '17 at 20:11
There is even [`jq`](https://stedolan.github.io/jq/), a tiny yet powerful command line tool to read and write JSON. — StephenKing, Dec 20 '17 at 05:27

score 5 · Answer 1 · answered Dec 19 '17 at 19:59

Perl with the JSON module:

cat servername.json | perl -MJSON -0 -ne '$j = decode_json($_); push @{$j->{run_list}}, q<role[My_New_Role]>; print encode_json($j)'

you can pretty-print it by replacing the print command with print to_json($j, {pretty => 1})

pault · Accepted Answer · 2017-12-19T19:07:38.580

3

I don't know if this is the best approach (vs sed, AWK, or Perl) but it is straightforward to do what you're asking using python's json library.

import json

# read the file as a dict using json.loads
d = json.loads(open('servername.json', 'r').read())

# add your new role to the end of the run_list
d['run_list'].append('role[My_New_Role]')

# write new json to file (specify a new file, or overwrite if you prefer)
open('new_servername.json', 'w').write(json.dumps(d, indent=2))

The output file looks like:

{
  "chef_environment": "test", 
  "name": "myserver123", 
  "run_list": [ 
    "role[base-pkg]", 
    "role[interesting_stuff]", 
    "role[user_apps]",
    "role[My_New_Role]"
  ]
}

It's pretty easy to modify this code into a script with the filename as an input so that it's easy to run multiple times.

edited Dec 19 '17 at 19:07

answered Dec 19 '17 at 19:00

pault

41,343
15
107
149

1

Versus _sed_ and _AWK_ this IS the best approach. Versus _Perl_ we could argue ;-) Python (afaik) already ships with a JSON module whereas Perl needs one from e.g. CPAN installed. – PerlDuck Dec 19 '17 at 20:40
1

@PerlDuck actually Perl ships with JSON::PP since 5.13.9. See [the `corelist` utility](http://perldoc.perl.org/corelist.html). :) – simbabque Dec 20 '17 at 00:16
@simbabque That's very good to know. Thank you. Sometimes (at work) I had to decide between Perl and Python for processing JSON and chose Python for its built-in support. Now I can pick Perl as well ;-) – PerlDuck Dec 20 '17 at 13:25
Is there anyway to force json.dumps to use Unix style LF only as a line break when running on Windows ? – eramm Dec 21 '17 at 15:13
@eramm This post should help: [How to explicitly set carriage return when doing json.dump?](https://stackoverflow.com/questions/41842875/how-to-explicitly-set-carriage-return-when-doing-json-dump) – pault Dec 21 '17 at 15:18
Yes I saw that previously but it doesn't seem to be working for me. d = json.loads(open('servername.json', 'r', newline='\n').read()) – eramm Dec 21 '17 at 15:37
You asked about `json.dumps` - that's called for the `write()`, not the `read()`: `open('new_servername.json', 'w', newline='\n').write(json.dumps(d, indent=2))` – pault Dec 21 '17 at 15:41
Facepalm ! I had previously tried putting it in the write() section thinking that's where it should go :-( works perfectly now. Thank You – eramm Dec 21 '17 at 16:13

Add a new role to a Chef .json file with Sed

2 Answers2

Linked