Replacing values in yml file using yq - invalid key reference

Question

I am currently working with editing/modifying yml files. I am a bit new using yq and unsure if I am going overboard using the tool. I am trying to replace two values in the yaml file with two environment values(linux system) as shown below. I am getting an invalid key reference error just simply trying to read the fields of that field using yq. What will be the best approach for modifying the fields the yml file? Would it be better to use sed or awk?

Environment Variables

$APP_KEY="DSDedfwerer02"
$APP_NAME="Test1"

test.yml

common: &default_settings
  app_key: ""
  app_name: ""
  some_key1: "test"
  some_key2: "onetwo"
  some_key3: "filename"
  another_key_n: "somevalue"

Desired result: test.yml

common: &default_settings
  app_key: "DSDedfwerer02"
  app_name: "Test1"

Can you provide `yq` command which you have tried to use and gave you *invalid key reference error*? — Daweo, Jun 29 '22 at 08:27
Which yq are you using mikefarah/yq (Go) or kisulyuk/yq (Python). See https://stackoverflow.com/tags/yq/info — Inian, Jun 29 '22 at 10:15
Are `APP_KEY` and `APP_NAME` really **environment variable**, i.e. they are exported and so available to subshells? Or did you just mean to say "variables" rather than "environment variables" and they aren't actually exported and so shouldn't really be all upper case (see [correct-bash-and-shell-script-variable-capitalization](https://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization))? — Ed Morton, Jun 29 '22 at 14:25
@EdMorton correct they are environment variables - available to subshells. — MaryCoding, Jun 29 '22 at 14:36

Inian · Accepted Answer · 2022-06-29T17:08:21.340

2

With mikefarah/yq, it is relatively straightforward to just import the environment variables to the local context and update the required YAML elements

k="$APP_KEY" n="$APP_NAME" yq '.common |= ( . + {"app_key": strenv(k), "app_name": strenv(n)} ) | ..style="double"' yaml

Use the -i or --inplace flag to update the file inplace. Tested on version v4.25.3

edited Jun 29 '22 at 17:08

answered Jun 29 '22 at 15:07

Inian

80,270
14
142
161

Just seeing this now. This is great! So this shows an output, would it possible through yq making that modification and saving the file with those values? – MaryCoding Jun 29 '22 at 16:21
Thanks! One thing I see that this removes all the other variables that the yml may contains. Would it possible to keep those other variables that could be in that file\? – MaryCoding Jun 29 '22 at 16:38
Just updated that. – MaryCoding Jun 29 '22 at 16:42
Awesome. I am testing it and I am seeing that it removes certain parts. For example `common: &default_settings` and changes it to `common`. Do you get that same behavior? – MaryCoding Jun 29 '22 at 17:00

Ed Morton · Answer 2 · 2022-06-29T14:50:18.237

1

In case you don't get a good yq answer, here's how to do what you're asking for in any awk given the sample input you provided:

$ APP_KEY="DSDedfwerer02"
$ APP_NAME="Test1"
$ awk -v key="$APP_KEY" -v name="$APP_NAME" '
    BEGIN { map["app_key:"]=key; map["app_name:"]=name }
    $1 in map { sub(/".*/,""); $0=$0 "\"" map[$1] "\"" }
1' file
common: &default_settings
  app_key: "DSDedfwerer02"
  app_name: "Test1"

That will work whether your shell variables really are environment variables or not and assuming they don't contain any backslashes and assuming app_key: and app_name: only appear as the first strings in the input in the positions you want their values replaced.

If they truly are environment variables then you can do:

awk '
    BEGIN { map["app_key:"]=ENVIRON["APP_KEY"]; map["app_name:"]=ENVIRON["APP_NAME"] }
    $1 in map { sub(/".*/,""); $0=$0 "\"" map[$1] "\"" }
1' file

or if not:

APP_KEY="$APP_KEY" APP_NAME="$APP_NAME" \
awk '
    BEGIN { map["app_key:"]=ENVIRON["APP_KEY"]; map["app_name:"]=ENVIRON["APP_NAME"] }
    $1 in map { sub(/".*/,""); $0=$0 "\"" map[$1] "\"" }
1' file

which will work even if they contain backslashes.

edited Jun 29 '22 at 14:50

answered Jun 29 '22 at 14:32

Ed Morton

188,023
17
78
185

1

This it! awk is the much simpler approach. – MaryCoding Jun 29 '22 at 14:37
I havent test it yet, but will this be affected if they are strings that follow bellow `app_key` and `app_name`? – MaryCoding Jun 29 '22 at 14:39
Only if they're also named `app_key:` and `app_name:` and also occur as the first strings tin a line. – Ed Morton Jun 29 '22 at 14:40
Ok perfect, these key values are unique so that answers that. – MaryCoding Jun 29 '22 at 14:41
Another consultation, if the keys already contain a value such as `app_key:"test"` this tends to append to it like this `app_key:"test""new value"`. Is there a way to replace if a value is present? – MaryCoding Jun 29 '22 at 19:40
Sure but that's why you use a tool that understands the language the file is written in as opposed to trying to parse text based on provided example(s). Please ask a new question and this time really think about the various ways your input might be formatted other than what you've shown so far so we can best help you. Though it'd be easy, I don't want to solve this new problem only to then have another problem, then another - just show us everything at once in a new question. – Ed Morton Jun 29 '22 at 19:43

Replacing values in yml file using yq - invalid key reference

2 Answers2