Edit YAML file with Bash

Question

I'm trying to edit the following YAML file

db:
  host: 'x.x.x.x.x'
  main:
    password: 'password_main'
  admin:
    password: 'password_admin'

To edit the host part, I got it working with

sed -i "/^\([[:space:]]*host: \).*/s//\1'$DNS_ENDPOINT'/" config.yml

But I can't find a way to update the password for main and admin (which are different values).

I tried to play around with \n and [[:space:]] and got different flavours of:

sed -i "/^\([[:space:]]*main:\n*[[:space:]]*password: \).*/s//\1'$DNS_ENDPOINT'/" config.yml

But never got it to work. Any help greatly appreciated!

Edit - Requirement: no external binaries/tools. Just good ol' bash.

You should consider using a YAML parser like `yq` Can you install such a tool? — Inian, Aug 25 '20 at 14:43
I would prefer to avoid doing that. Not a requirement, but almost. — Mornor, Aug 25 '20 at 14:44
Agreed, whatever options you are going to use with standard text processing tools aren't going to be reliable — Inian, Aug 25 '20 at 14:45
Okay, let's say it's a requirement. As far as I know `yq` is a wrapper around `jq`, so it should be possible to do that reliably. — Mornor, Aug 25 '20 at 14:47
So there are two versions of `yq` available. One using `jq` and other using a proper DSL, written in Go. The former -https://kislyuk.github.io/yq/, the latter - https://github.com/mikefarah/yq. Depends on which one you want to use — Inian, Aug 25 '20 at 14:49
Remember that sed != bash, so whatever approach you are taking, custom flags/regexes might work on one system and fail in another — Inian, Aug 25 '20 at 14:57
"_Requirement: no external binaries/tools. Just good ol' bash._" - but you are using `sed` yourself? — Ted Lyngmo, Aug 25 '20 at 15:52
`Just good ol' bash` would be a terrible choice. If you can't use `yq` then you should instead use one of the standard tools you can call from a shell, e.g. awk. — Ed Morton, Aug 25 '20 at 16:35

Ted Lyngmo · Answer 1 · 2022-03-22T19:45:14.303

Since you don't want to install yq you could use python that you most probably already have installed.

Here are the fundamentals:

#!/usr/bin/python

import yaml

with open("config.yml") as f:
    y = yaml.safe_load(f)
    y['db']['admin']['password'] = 'new_admin_pass'
    print(yaml.dump(y, default_flow_style=False, sort_keys=False))

Output:

db:
  host: x.x.x.x.x
  main:
    password: password_main
  admin:
    password: new_admin_pass

A similar piece of python code as a one-liner that you can put in a bash script would look something like this (and produce the same output):

python -c 'import yaml;f=open("config.yml");y=yaml.safe_load(f);y["db"]["admin"]["password"] = "new_admin_pass"; print(yaml.dump(y, default_flow_style=False, sort_keys=False))'

If you'd like to save the output to a file, you can provide an output stream as the second argument to dump():

#!/usr/bin/python

import yaml

with open("config.yml") as istream:
    ymldoc = yaml.safe_load(istream)
    ymldoc['db']['admin']['password'] = 'new_admin_pass'

with open("modified.yml", "w") as ostream:
    yaml.dump(ymldoc, ostream, default_flow_style=False, sort_keys=False)

If you'd like to overwrite the original file, I recommend writing to a temporary file first and only if that succeeds, use os.rename to move that file in place of the original one. That's to minimize the risk of creating a corrupt config.yml in case of problems.

@RavinderSingh13 You are welcome! It requires some editing to replace the hardcoded strings but it should a pretty good base to use when editing YAML files. — Ted Lyngmo, Aug 26 '20 at 11:49
Very good answer indeed, provide me with the robustness of Python. — Mornor, Aug 26 '20 at 19:29
This worked well for me but unfortunately it looks like it only print the updated value but doesn't save the updated file. How can i do this? — Jerry, Mar 22 '22 at 12:37
@Jerry I'm glad to hear that it helped. I updated the answer to show how you can save the updated information to file. — Ted Lyngmo, Mar 22 '22 at 15:03

0stone0 · Answer 2 · 2020-08-25T17:17:52.843

8

Note: Using a YAML parser like yq (or yq) will be a way more reliable solution.

However, I've used the following 'technique' to alter a 'pre-defined' line though the help of grep and sed like so;

/tmp/config.yml

db:
  host: 'x.x.x.x.x'
  main:
    password: 'password_main'
  admin:
    password: 'password_admin'

Get the line number where your 'old-password' is located:

grep -n 'password_admin' /tmp/config.yml | cut -d ':' -f1

6

Then, use sed to override that line with your new password:

sed -i '6s/.*/    password: \'new_admin_pass\'/' /tmp/config.yml

The new file now looks like this:

db:
  host: 'x.x.x.x.x'
  main:
    password: 'password_main'
  admin:
    password: 'new_admin_pass'

Note

Keep in mind that any special chars (&, \, /) in the password will cause sed to misbehave!
This could fail if the indent changes, since YAML cares about indentation. Just like I mentioned above, using a YAML parser will be a much more reliable solution!

edited Aug 25 '20 at 17:17

answered Aug 25 '20 at 14:56

0stone0

34,288
4
39
64

1

Thanks! Let's assume the indent does not change. Could something else cause this command to fail? – Mornor Aug 25 '20 at 15:07
1

@Mornor I dont think so, I've used this 'style' to alter YAML for quite some time now, never failed. However, my YAML file is quite static, I'm only using this technique to alter a password ;) – 0stone0 Aug 25 '20 at 15:12
4

It would fail if the new password contained backreference metacharacters like `&` or `\1` or if it contained delimiter chars `/` or escape chars `my\new\pass` or others. As long as your passwords are always strictly alphanumeric you should be OK but if they aren't then YMMV. You don't need grep+cut+sed though - a simple awk script will do the job. Not also that that's relying on you knowing the old admin password when all you really want is to change the admin (or main) password to the new value no matter what it was before. – Ed Morton Aug 25 '20 at 16:52
3

... I just realised it'd also fail if the `main:` password just happened to be the same as the `admin:` password or if that admin password just happened to also show up elsewhere in the file. – Ed Morton Aug 25 '20 at 17:01
This doesn't consider the tree structure of the yml files. A simple grep will find all the occurrences of the key and not the ones belonging to a particular node. I believe a correct implementation in pure bash would be complicated as it has to read the indentation and then parse the tree structure. – thanos.a Jul 14 '22 at 10:52

anubhava · Answer 3 · 2020-08-25T16:41:34.197

5

This is by no way as reliable as yq but you can use this awk if your yaml file structure is same as how it is shown in question:

pw='new_&pass'
awk -v pw="${pw//&/\\\\&}" '/^[[:blank:]]*main:/ {
   print
   if (getline > 0 && $1 == "password:")
      sub(/\047[^\047]*\047/, "\047" pw "\047")
} 1' file

db:
  host: 'x.x.x.x.x'
  main:
    password: 'new_&pass'
  admin:
    password: 'password_admin'

edited Aug 25 '20 at 16:41

answered Aug 25 '20 at 15:08

anubhava

761,203
64
569
643

Ed Morton · Accepted Answer · 2020-08-25T17:07:28.493

$ awk -v new="'sumthin'" 'prev=="main:"{sub(/\047.*/,""); $0=$0 new} {prev=$1} 1' file
db:
  host: 'x.x.x.x.x'
  main:
    password: 'sumthin'
  admin:
    password: 'password_admin'

or if your new text can contain escape sequences that you don't want expanded (e.g. \t or \n), as seems likely when setting a password, then:

new="'sumthin'" awk 'prev=="main:"{sub(/\047.*/,""); $0=$0 ENVIRON["new"]} {prev=$1} 1' file

See How do I use shell variables in an awk script? for why/how I use ENVIRON[] to access a shell variable rather than setting an awk variable in that second script.

score 2 · Answer 5 · answered Aug 25 '20 at 15:22

As mentioned by experts in other answers too, yq should be the proper way but in case someone doesn't have it then one could try following.

awk -v s1="'" -v new_pass="new_value_here" '
/main:/{
  main_found=1
  print
  next
}
main_found && /password/{
  next
}
/admin:/ && main_found{
  print "    password: " s1 new_pass s1 ORS $0
  main_found=""
  next
}
1
'  Input_file

NOTE: In case you want to save output into Input_file itself then append > temp && mv temp Input_file to above solution.

Edit YAML file with Bash

5 Answers5

Linked

Related