Best way to read a (ba)sh config file in python 2 or 3

Question

I have the the following config file

# originally this was a (ba)sh config file, now read by python
TEST=123
  BAK=.bak
# comment for next line
TEST_2="cool spaced Value"
MY_VAR2="space value"  # With a Comment for the value

I have managed to read this with the following code

def parse_lines(self, lines):
  pattern = r'[ |\t]*([a-zA-Z_][a-zA-Z0-9_]*)=("([^\\"]|.*)"|([^# \t]*)).*[\r]*\n'
  prog = re.compile(pattern)
  hash={}

  for line in lines:
    result = prog.match(line)
    if not result is None:
      name = result.groups()[0]
      if result.groups()[2] is None:
        value= result.groups()[3]
      else:
        value= result.groups()[2]
    hash[name]=value
    return hash

 def read_shell_config(self, filename):
   with  open(filename) as f:
     lines = f.readlines()
     hash  = self.parse_lines(lines)
     return hash

Is there any better way (standard package?) to read a bash config file like the above with python?

seem to be very old config files, edited over the years, so that's what i found after some investigation. so yes. — Mandragor, Aug 17 '16 at 23:06
For this to work with any arbitrary config file, it would need to be a proper shell interpreter, with flow control and builtins and all the rest. — Charles Duffy, Aug 17 '16 at 23:08
basically somebody should have run already in such a problem, so does this already exist as "standard" package? — Mandragor, Aug 17 '16 at 23:09
For instance, assume you have someone running `FOO_CLASSPATH=$(find "$HOME/foo" -name '*.jar' -printf '%s:')` -- you'd need to implement `find` (with the local system's non-POSIX extensions), command substitution... it's outside feasibility. — Charles Duffy, Aug 17 '16 at 23:09
You might as well ask if anyone has written a Python interpreter in bash. These aren't config files as you're used to them, they're *arbitrary scripts*, just like a Django config file in the Python world can run any arbitrary Python code it wants, up to and including connecting to a database and pulling configuration from there. — Charles Duffy, Aug 17 '16 at 23:10
well the above code is working, but I believe there a better ways to do this. The bash configs, i have do not possess any (ba)sh "$VAR" logic , they are the values as simple as above. that's all. but about 2000 of them .... — Mandragor, Aug 17 '16 at 23:12
Sure -- you can make it work for a limited subset, but you won't find something that's actually guaranteed to behave identically (just like you could parse a *really simple* Django config file in bash, but not a complicated one). — Charles Duffy, Aug 17 '16 at 23:12
now, there *are* some ways you could actually have a real shell interpreter source a config file and then emit a list of defined variables in a format Python can easily parse (personally, I'd use a stream of NUL-delimited key/value pairs), but even then it'll only work with a subset of all possible config files -- anything that short-circuits evaluation or exits early will be a no-go. — Charles Duffy, Aug 17 '16 at 23:14

Charles Duffy · Answer 1 · 2016-08-17T23:48:12.347

Implementing a real shell interpreter in Python is beyond feasibility -- but you can cheat:

#!/usr/bin/env python
import subprocess
import os

script='''
set -a # export all variable definitions to the environment

# record prior names
declare -A prior_names=( )
while IFS= read -r varname; do
  prior_names[$varname]=1
done < <(compgen -v)

source "$1" >/dev/null

while IFS= read -r varname; do
  [[ ${prior_names[$varname]} ]] && continue # skip anything that was already present
  printf '%s\\0' "$varname" "${!varname}"
done < <(compgen -v)
'''

def getVars(configFile):
    p = subprocess.Popen(['bash', '-c', script, '_', configFile],
        stdout=subprocess.PIPE,
        env={'PATH': os.environ['PATH']})
    items = p.communicate()[0].split('\0')
    keys = [val for (idx, val) in enumerate(items) if (idx % 2 == 0)]
    vals = [val for (idx, val) in enumerate(items) if (idx % 2 == 1)]
    return dict(zip(keys, vals))

if __name__ == '__main__':
    import sys
    print getVars(sys.argv[1])

nickma · Answer 2 · 2019-09-05T08:31:46.557

a not quite as compatible version but beautiful option is using the python ConfigParser with the ExtendedInterpolation option.

To use it with a bash compatible file one added a [section] during file readout, as following (python3):

from configparser import ConfigParser, ExtendedInterpolation
def get_vars(config_file):
    with open(config_file, 'r') as f:
        config_string = '[DEFAULT]\n' + f.read()
    config = ConfigParser(interpolation=ExtendedInterpolation())
    config.read_string(config_string)

    return dict(config['DEFAULT'])

And the output of your proposed test file looks like this (like I said not perfectly compatible):

{'test': '123\nBAK=.bak', 'test_2': '"cool spaced Value"', 'my_var2': '"space value"  # With a Comment for the value'}

Best way to read a (ba)sh config file in python 2 or 3

2 Answers2