Read a file and edit the file at the same time (line by line) in bash

Question

Suppose I have a file temp.yaml.

# Comment shouldn't be deleted
passwords:
    key1: __password__
    key2: __password__
    key3: __password2__
    key4: __password__
# All comments should be preserved
passwords-new:
    key1: __newPassword__

In this file, I want to have each __password__ field with a different value. Essentially, all of the passwords in this file would have a different value.

I was thinking to read the file line by line and store/replace a password with a newly generated password. But not sure, how I can go through the file line by line and edit that particular line at the same time in bash.

Any other solution or better approach would also work.

Please show what you have tried and explain why it wasn't satisfactory. — Mad Physicist, Apr 11 '19 at 15:55
The general idea is that you don't "edit" lines as such. You read in a line from one file and write out to another. — Mad Physicist, Apr 11 '19 at 15:56
For now, I have used sed separately for searching 'key1: __password__' and replacing it with 'key1: xysdxy`. But in future, if we want to add a new key, we want to handle it automatically and do not want someone to have edit new sed statements for new keys. — Harshit, Apr 11 '19 at 16:06
Is there a reason you want to use bash instead of something like this: https://stackoverflow.com/questions/7255885/save-dump-a-yaml-file-with-comments-in-pyyaml — Tyler Marshall, Apr 11 '19 at 16:21
Almost our entire code base is in bash for that product. So we are looking for the simplest and easiest way to do it in bash. — Harshit, Apr 11 '19 at 16:36

score 0 · Answer 1 · answered Apr 11 '19 at 19:19

If the YAML file isn't too big, you could use Bash code to make the edits in memory and write the results back to the file. This, Shellcheck-clean, code demonstrates the idea:

#! /bin/bash -p

# Set passwords in a YAML file.  Update the file in place.
function set_yaml_passwords
{
    local -r yaml_file=$1

    # Create a regular expression to match key/password lines
    local -r key_rx='^([[:space:]]*key[[:digit:]]*:[[:space:]]*)'
    local -r pw_rx='__[[:alnum:]]*[Pp]assword[[:alnum:]]*__[[:space:]]*$'
    local -r key_pw_rx=${key_rx}${pw_rx}

    # Read the YAML file lines into an array, setting passwords as we go
    local yaml_lines=() is_updated=0
    local line keystring newpw
    while IFS= read -r line || [[ -n $line ]] ; do
        if [[ $line =~ $key_pw_rx ]] ; then
            keystring=${BASH_REMATCH[1]}
            newpw=$(pwgen 10)
            yaml_lines+=( "${keystring}${newpw}" )
            is_updated=1
        else
            yaml_lines+=( "$line" )
        fi
    done <"$yaml_file"

    # If any passwords have been set, update the YAML file
    if (( is_updated )) ; then
        printf '%s\n' "${yaml_lines[@]}" >"$yaml_file"
        printf "Updated '%s'\\n" "$yaml_file" >&2
    fi

    return 0
}

set_yaml_passwords 'temp.yaml'

This is an example of a YAML file produced by running the code above on the example YAML file given in the question:

# Comment shouldn't be deleted
passwords:
    key1: yai9cegiD4
    key2: Ahtah1eyoh
    key3: ya1teo1ooB
    key4: AhGae5EiLe
# All comments should be preserved
passwords-new:
    key1: oaKoh0teiK

Line-by-line processing in Bash code is very slow. The code above took almost 1 second to process a ten thousand line file (with a small number of key/password lines) on my (oldish) Linux system. On a modern system you'll run into time problems long before you run into memory problems.
I used pwgen 10 to generate new passwords. You may want to do something else.
There's no error checking in the demonstration code. Real code should check for missing file, unreadable file, unwritable file, wrong number of arguments to the function, etc.

Read a file and edit the file at the same time (line by line) in bash

1 Answers1