Go viper .yaml values environment variables override

Question

I'm trying to have application.yaml file in go application which contains ${RMQ_HOST} values which I want to override with environment variables.

In application.yaml I've got:

rmq:
  test:
    host: ${RMQ_HOST}
    port: ${RMQ_PORT}

And in my loader I have:

log.Println("Loading config...")
viper.SetConfigName("application")
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
viper.AutomaticEnv()
err := viper.ReadInConfig()

The problem I have is ${RMQ_HOST} won't get replaced by the values I've set in my environment variables, and tries to connect to the RabbitMQ with this string

amqp://test:test@${RMQ_HOST}:${RMQ_PORT}/test

instead of

amqp://test:test@test:test/test

why use a yaml file when you want to use environmetn variables? — Pizza lord - on strike, Oct 11 '18 at 14:22
Because yaml file values get mapped to environment variables avoiding hardcoded keys making it more extendable over multiple deployment environments. In other words, trying to achieve spring boot functionality. — Ajdin Halac, Oct 11 '18 at 14:30
then i might be the way you get the values using viper you will need to read to documentation to know which way works for you — Pizza lord - on strike, Oct 11 '18 at 14:33

Ajdin Halac · Answer 1 · 2021-01-07T22:18:47.513

Viper doesn't have the ability to keep placeholders for values in key/value pairs, so I've managed to solve my issue with this code snippet:

log.Println("Loading config...")
viper.SetConfigName("application")
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
err := viper.ReadInConfig()
if err != nil {
    panic("Couldn't load configuration, cannot start. Terminating. Error: " + err.Error())
}
log.Println("Config loaded successfully...")
log.Println("Getting environment variables...")
for _, k := range viper.AllKeys() {
    value := viper.GetString(k)
    if strings.HasPrefix(value, "${") && strings.HasSuffix(value, "}") {
        viper.Set(k, getEnvOrPanic(strings.TrimSuffix(strings.TrimPrefix(value,"${"), "}")))
    }
}

func getEnvOrPanic(env string) string {
    res := os.Getenv(env)
    if len(res) == 0 {
        panic("Mandatory env variable not found:" + env)
    }
    return res
}

This will overwrite all the placeholders found in the collection.

```len(env) == 0``` should be ```len(res) == 0``` – Jeongmyeong Lee Nov 19 '19 at 03:17 — Jeongmyeong Lee, Nov 19 '19 at 03:17

Ajdin Halac · Answer 2 · 2019-02-14T12:32:54.110

Update:

I extended the native yaml parser with this functionality and released it on github.

Usage:

type Config struct {
    Port     string   `yaml:"port"`
    RabbitMQ RabbitMQ `yaml:"rabbitmq"`
}

type RabbitMQ struct {
    Host     string `yaml:"host"`
    Port     string `yaml:"port"`
    Username string `yaml:"username"`
    Password string `yaml:"password"`
    Vhost    string `yaml:"vhost"`
}

func main() {
    var config Config
    file, err := ioutil.ReadFile("application.yaml")
    if err != nil {
        panic(err)
    }
    yaml.Unmarshal(file, &config)
    spew.Dump(config)
}

This is how application.yaml looks like:

port: ${SERVER_PORT}
rabbitmq:
  host: ${RMQ_HOST}
  port: ${RMQ_PORT}
  username: ${RMQ_USERNAME}
  password: ${RMQ_PASSWORD}
  vhost: test

vhost value will get parsed as usual, while everything surrounded with "${" and "}" will get replaced with environment variables.

It does not, if you need that feature we can arrange to add it into the repository with Backwards Compatibility. — Ajdin Halac, Aug 29 '22 at 17:51

score 2 · Answer 3 · answered Aug 23 '20 at 07:32

You can explicitly substitute the env variables before calling the "Unmarshal" method. Assuming the configuration is stored in "Config" variable, the following code snippet should work.

log.Println("Loading config...")
viper.SetConfigName("application")
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
viper.AutomaticEnv()
if err := viper.ReadInConfig(); err != nil {
    fmt.Fprintf("Error reading config file %s\n", err)
}

for _, k := range viper.AllKeys() {
    v := viper.GetString(k)
    viper.Set(k, os.ExpandEnv(v))
}

if err := viper.Unmarshal(&Config); err != nil {
    fmt.Fprintf("Unable to decode into struct %s\n", err)
}

iamdavidzeng · Answer 4 · 2020-08-28T11:06:25.077

I resolved similar question by Using regexp to replace ENV firstly, here is my solutions:

# config.yaml
DB_URI: ${DB_USER}

and main.go:

package main

import (
    "fmt"
    "os"
    "regexp"

    "github.com/spf13/viper"
)

type DBCfg struct {
    DBURI string `mapstructure:"DB_URI"`
}

func main() {
    viper.SetConfigName("config")
    viper.SetConfigType("yaml")
    viper.AddConfigPath(".")
    viper.AutomaticEnv()

    if err := viper.ReadInConfig(); err != nil {
        panic(fmt.Errorf("Failed to read config"))
    }

    for _, key := range viper.AllKeys() {
        value := viper.GetString(key)
        envOrRaw := replaceEnvInConfig([]byte(value))
        viper.Set(key, string(envOrRaw))
    }

    var config DBCfg
    if err := viper.Unmarshal(&config); err != nil {
        panic(fmt.Errorf("failed to load"))
    }

    fmt.Println(config)
}

func replaceEnvInConfig(body []byte) []byte {
    search := regexp.MustCompile(`\$\{([^{}]+)\}`)
    replacedBody := search.ReplaceAllFunc(body, func(b []byte) []byte {
        group1 := search.ReplaceAllString(string(b), `$1`)

        envValue := os.Getenv(group1)
        if len(envValue) > 0 {
            return []byte(envValue)
        }
        return []byte("")
    })

    return replacedBody
}

and my output:

>>> DB_USER=iamddjsaio go run main.go

{iamddjsaio}

Not a common solution, doesn't work with structured config values such as map — Tigran Babajanyan, Jun 09 '21 at 07:13
Seems like the solution only resolve the env value item, structured config values should parse to your customize struct by tag when use viper, instead of use regexp. — iamdavidzeng, Apr 04 '22 at 00:01

score 2 · Answer 5 · answered Nov 07 '22 at 08:03

Might not be the direct answer but the solution to the problem you're trying to solve.

Chances are you don't need to replace. If you leave the keys in yml empty and have viper.AutomaticEnv() on, then it will pick those values from environment variable. You'll need to add a replacer as well to match the keys with env names like:

viper.AutomaticEnv()
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
viper.SetConfigFile("config.yml")
config := &Config{}

Now, when you unmarhsal the config, you'll get the missing keys from env.

Example: RABBITMQ_PASSWORD env variable will be used to set rabbitmq.password if your yml looks like this:

rabbitmq
  password: ""

this is the most underrated answer I've seen. It should solve issues for many developers using yamls. The linux env vars cannot have `.` in the name and with this we can replace dots with underscore in the env variables and still override the yaml config. — sabertooth1990, Jan 18 '23 at 12:14

score 0 · Answer 6 · answered Aug 12 '21 at 19:44

I think an even better way to do this is to use Viper's concept of DecodeHooks when decoding configuration into a struct:

If you have a yaml configuration file like the following:

server:
  property: ${AN_ENV_VARIABLE}

Then inside the decode hook, grab the ENV variable as the decoding is happening:

decodeHook := func(f reflect.Type, t reflect.Type, data interface{}) (interface{}, error) {
    if f.Kind() == reflect.String {
        stringData := data.(string)
        if strings.HasPrefix(stringData, "${") && strings.HasSuffix(stringData, "}") {
            envVarValue := os.Getenv(strings.TrimPrefix(strings.TrimSuffix(stringData, "}"), "${"))
            if len(envVarValue) > 0 {
                return envVarValue, nil
            }
        }
    }
    return data, nil
}
err := viper.Unmarshal(c, viper.DecodeHook(decodeHook))

I think this could be further improved since you might have an incoming property that might need to be set to an int, for example. However, this general concept should work for most use cases.

Go viper .yaml values environment variables override

6 Answers6

Linked