sed: change values of properties of an environment in a .yml file

Question

I have an .yml file that configures environment properties of an application, such like this:

env1:
  :prop1: "value1"
  :prop2: "value2"
        ...
  :propn: "valuen"

env2:
  :prop1: "value1"
  :prop2: "value2"
  :prop3: "value3"
        ...
  :propn: "valuen"

    ...

envn:
  :prop1: "value1"
  :prop2: "value2"
        ...
  :propn: "valuen"

I would like to produce a bash script with the following interface:

$ change_env.sh <environment> <property> <new value> <file.yml>

Example:

$ change_env.sh env2 prop3 "this value was changed" file.yml

The output will be:

env1:
  :prop1: "value1"
  :prop2: "value2"
        ...
  :propn: "valuen"

env2:
  :prop1: "value1"
  :prop2: "value2"
  :prop3: "this value was changed"
        ...
  :propn: "valuen"

    ...

envn:
  :prop1: "value1"
  :prop2: "value2"
        ...
  :propn: "valuen"

I found this post, however I could not do it work for my case. Replace an XML element's value? Sed regular expression?

I also tried this: (it fails because alters all properties)

sed 's/\(:pro3:\).*/\1 "new value"/'

Thanks in advance! -- Lourenco.

shellter · Accepted Answer · 2016-09-15T14:11:23.353

(very nice first post!)

Try this

cat change_env.sh

#!/bin/bash
# spec : change_env.sh <environment> <property> <new value> <file.yml>

case ${#} in [!4] ) 
    echo "usage: change_env.sh <environment> <property> <new value> <file.yml>" 1>&2
    exit 1 
   ;; 
esac

env="$1" prop="$2" new="$3" file="$4"
bakFile="${file}".bak
mv "$file" "$bakFile"
sed '/^'"${env}"'/,/^[   ]*$/{  # [ spaceChar tabChar ]
        /'"${prop}"'/s/\('"${prop}"'\)\(.*$\)/\1'"${new}"'/
    }' "$bakFile" > "$file"

edit

Note, if you expect input to contain white-space in the values you'll want to modify script to quote all variables ("$1","$2"...). (I have now done this, as it is a shell-scripting best practice).

The /env/,/^[{space,tab}]*$/ is a range address for sed. It reads a block of text that contains your environment settings. I'm assuming your sample input is correct and that each env is separated by a blank line. Hmm... this would include the last one in the file.

** edit**

Thanks to @posdef for pointing some problems with this answer. The code is updated to solve the particular case.

Even after the fix, I did notice that given an input like

   change_env.sh env2 prop2 "new value" file.yml

The relevant output was

     :prop2new value

So, without hardcoding extra : and space chars into the substitution, this means you'll need to be very verbose in how you call the <property> value AND the <new value> , i.e.

   change_env.sh env2 ":prop2: " "\"new value\"" file.yml
   # note extra cruft-^^-----^^^--^^---------^^--------------

relevant output

env2:
  :prop1: "value1"
  :prop2: "new value"
  :prop3: "value3"
    ...
  :propn: "valuen"

IHTH

I am not sure if it's just me but this script doesn't work properly for me: 1) the property name gets changed not the value; trying to change the value of a property called "differences" from 2 to 0 becomes `0: 2` instead. 2) the line `0: 2` is duplicated in the output file. — posdef, Sep 14 '16 at 14:53
@posdef : Thanks for pointing out that flaw. It is fixed now, but I can't spend time trying to make a solution for general cases, so the `` search target will have to be precise as well ``. (i.e. `":prop2: "` and `"\"new value\""` . Good luck to all. — shellter, Sep 15 '16 at 14:14
@shellter no worries, was just mentioning the potential flaw for others that might stumble upon this answer like I did :) — posdef, Sep 16 '16 at 08:33

score 3 · Answer 2 · answered Apr 19 '11 at 23:58

3

I'd use awk:

#!/bin/sh

if [ $# -ne 4 ]; then
    echo "usage: $0 env prop value file" >&2
    exit 1
fi

awk -F : -v env="$1" -v prop="$2" -v val=" \"$3\"" '
    BEGIN {OFS = FS}
    $1 == env {in_env = 1}
    NF == 0   {in_env = 0}
    in_env && $2 == prop {$3 = val}
    {print}
' "$4"

answered Apr 19 '11 at 23:58

glenn jackman

238,783
38
220
352

Nice to know that there is a simple way to do it with AWK. Thank you! – Lourenco Apr 25 '11 at 23:07
As far as I can tell this script doesn't work as intended, since it does not pay attention to the indentation of the items within an environment. In other words the condition `in_env && $2 == prop` doesn't ever hold since `prop` is typically in a new line and at that implies that it will not be $2 but instead $1 in a following line. I'll add an answer, based on yours with a small fix – posdef Sep 20 '16 at 10:04
No, I don't use indentation, I use colon as the field separator. And all the "prop"s have a *leading* colon, which makes the property become the second field. – glenn jackman Sep 20 '16 at 13:26
@glennjackman i see, must have missed the leading colons on the properties, not familiar with that notation. Apologies for the confusion – posdef Sep 22 '16 at 07:08

score 2 · Answer 3 · edited May 23 '17 at 12:33

This answer is based on Glenn Jackman's AWK script, which in my testing fails due to indentation issues inherent in the type of input OP (and yours truly) has.

Specifically, the condition of checking whether or not we are in the desired environment will happen on a different iteration than checking whether or not we have the desired property since these will typically be on different lines. Thus in_env && $2 == prop will never return true, considering that property : value pair will be read as $1 : $2 on a separate line.

Additionally, the comparison $2 == prop will suffer from the leading whitespace, which needs to be trimmed. I've added a couple of nice one-liners described here, for making the script more human-readable.

Lastly, the original script hard-coded double quotes around the new value, which is a problem if you are inserting numerical values.

I have modified the script in the following way which works well in my test case. I am providing here in case it is of use to others.

#!/bin/sh

if [ $# -ne 4 ]; then
    echo "usage: $0 env prop value file" >&2
    exit 1
fi

awk -F : -v env="$1" -v prop="$2" -v val=" $3" '
    function ltrim(s) { sub(/^[ \t\r\n]+/, "", s); return s }
    function rtrim(s) { sub(/[ \t\r\n]+$/, "", s); return s }
    function trim(s) { return rtrim(ltrim(s)); }
    BEGIN {OFS = FS}
    $1 == env {in_env = 1}
    NF == 0   {in_env = 0}
    in_env && trim($1) == prop {$2 = val}
    {print}  
' "$4"

bash-o-logist · Answer 4 · 2011-04-20T02:23:28.863

1

Bash script

#!/bin/bash
# tested with bash 4
if [ $# -ne 4 ];then
    echo "Usage: .... "
    exit
fi
env=$1
prop=$2
text="$3"
file=$4
while read -r line
do
    case "$line" in
        "$env"* )
        toggle=1
        ;;
    esac
    if [ "$toggle" = 1 ];then
        if [[ $line =~ "$prop" ]] ;then
            line="${line%%\"*}\"$text\""
            toggle=0
        fi
    fi
    echo "$line"
done < $file > t
mv t $file

edited Apr 20 '11 at 02:23

answered Apr 20 '11 at 01:41

bash-o-logist

6,665
1
17
14

what condition unsets the toggle var? – shellter Apr 20 '11 at 02:22
Cool! Firstly I thought use RE without loop, however it's really a way to do it. Thank you. – Lourenco Apr 25 '11 at 23:10

score 0 · Answer 5 · answered Apr 20 '11 at 00:07

Your going to need at least one lookahead assertion. There are many ways to do it, but you should use something that knows how to parse this.

s/(\s*$env:\s*(?:(?!\s*[^\W_]+:)[^\n]*\s*)*\s*:$prop:[^\S\n]*")[^\n]*(")/$1$replacement$2/g

Mildly tested in perl:

use strict;
use warnings;

my $str = '
env1:
  :prop1: "value1"
  :prop2: "value2"
        ...
  :propn: "valuen"

env2:
  :prop1: "value1"
        ...
  :prop2: "value2"
  :prop3: "value3"
  :propn: "valuen"
    ...

envn:
  :prop1: "value1"
        ...
  :prop3: "value3"
  :propn: "valuen"
';

my ($env, $prop, $replacement) = ('(?:env2|envn)', 'prop1', 'this changed');

if ( $str =~ s/
    (
      \s*$env:\s*
      (?: (?!\s*[^\W_]+:) [^\n]*\s* )*
      \s*:$prop:[^\S\n]*
      "
    ) [^\n]*
    ( " )
  /$1$replacement$2/xg )
{
     print "Found it!\n";
     print $str;
}

Output:

Found it!

env1:
  :prop1: "value1"
  :prop2: "value2"
        ...
  :propn: "valuen"

env2:
  :prop1: "this changed"
        ...
  :prop2: "value2"
  :prop3: "value3"
  :propn: "valuen"

    ...

envn:
  :prop1: "this changed"
        ...
  :prop3: "value3"
  :propn: "valuen"

*cons*: Big RE! Hard to maintain; needed parameters adaptation to work like the interface I mentioned. *pros*: possibility of choosing differents environments. It's also solve the problem. Thank you. — Lourenco, Apr 25 '11 at 23:13

sed: change values of properties of an environment in a .yml file

5 Answers5

Linked