Accessing environment variables in a YAML file for Ruby project (using ${ENVVAR} syntax)

Question

I am building an open source project using Ruby for testing HTTP services: https://github.com/Comcast/http-blackbox-test-tool

I want to be able to reference environment variables in my test-plan.yaml file. I could use ERB, however I don't want to support embedding any random Ruby code and ERB syntax is odd for non-rubyists, I just want to access environment variables using the commonly used Unix style ${ENV_VAR} syntax.

e.g.

order-lunch-app-health:
  request:
    url: ${ORDER_APP_URL}
    headers: 
      content-type: 'application/text'
    method: get
  expectedResponse:
    statusCode: 200
    maxRetryCount: 5

All examples I have found for Ruby use ERB. Does anyone have a suggestion on the best way to deal with this? I an open to using another tool to preprocess the YAML and then send that to the Ruby application.

See my updated answer - now also with what I believe is a better alternative, without going through Ruby's `%{var}` substitution. — DannyB, Apr 27 '19 at 15:25
**See also:** https://stackoverflow.com/questions/41620674/use-placeholders-in-yaml — dreftymac, Dec 30 '19 at 22:42

DannyB · Answer 1 · 2019-04-30T14:02:04.970

I believe something like this should work under most circumstances:

require 'yaml'

def load_yaml(file)
  content = File.read file
  content.gsub! /\${([^}]+)}/ do
    ENV[$1]
  end

  YAML.load content
end

p load_yaml 'sample.yml'

As opposed to my original answer, this is both simpler and handles undefined ENV variables well.

Try with this YAML:

# sample.yml
path: ${PATH}
home: ${HOME}
error: ${NO_SUCH_VAR}

Original Answer (left here for reference)

There are several ways to do it. If you want to allow your users to use the ${VAR} syntax, then perhaps one way would be to first convert these variables to Ruby string substitution format %{VAR} and then evaluate all environment variables together.

Here is a rough proof of concept:

require 'yaml'

# Transform environments to a hash of { symbol: value }
env_hash = ENV.to_h.transform_keys(&:to_sym)

# Load the file and convert ${ANYTHING} to %{ANYTHING}
content = File.read 'sample.yml'
content.gsub! /\${([^}]+)}/, "%{\\1}"

# Use Ruby string substitution to replace %{VARS}
content %= env_hash

# Done
yaml = YAML.load content
p yaml

Use it with this sample.yml for instance:

# sample.yml
path: ${PATH}
home: ${HOME}

There are many ways this can be improved upon of course.

do you mean `file` instead of `sample.yml` in the load_yaml function? — icarito, Apr 30 '19 at 05:40

score 0 · Answer 2 · answered Apr 27 '19 at 14:33

Preprocessing is easy, and I recommend you use a YAML loaderd/dumper based solution, as the replacement might require quotes around the replacement scalar. (E.g. you substitute the string true, if that were not quoted, the resulting YAML would be read as a boolean).

Assuming your "source" is in input.yaml and your env. variable ORDER_APP_URL set to https://some.site/and/url. And the following script in expand.py:

import sys
import os
from pathlib import Path
import ruamel.yaml

def substenv(d, env):
    if isinstance(d, dict):
        for k, v in d.items():
            if isinstance(v, str) and '${' in v:
                d[k] = v.replace('${', '{').format(**env)
            else:
                substenv(v, env)
    elif isinstance(d, list):
        for idx, item in enumerate(d):
            if isinstance(v, str) and '${' in v:
                d[idx] = item.replace('${', '{').format(**env)
            else:
                substenv(item, env)


yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(Path(sys.argv[1]))

substenv(data, os.environ)
yaml.dump(data, Path(sys.argv[2]))

You can then do:

python expand.py input.yaml output.yaml

which writes output.yaml:

order-lunch-app-health:
  request:
    url: https://some.site/and/url
    headers:
      content-type: 'application/text'
    method: get
  expectedResponse:
    statusCode: 200
    maxRetryCount: 5

Please note that the spurious quotes around 'application/text' are preserved, as would be any comments in the original file.

Quotes around the substituted URL are not necessary, but the would have been added if they were.

The substenv routine recursively traverses the loaded data, and substitutes even if the substitution is in mid-scalar, and if there are more than substitution in one scalar. You can "tighten" the test:

        if isinstance(v, str) and '${' in v:

if that would match too many strings loaded from YAML.

Accessing environment variables in a YAML file for Ruby project (using ${ENVVAR} syntax)

2 Answers2

Original Answer (left here for reference)