parsing linux iscsi multipath.conf into python nested dictionaries

Question

I writing a script that involves adding/removing multipath "objects" from the standard multipath.conf configuration file, example below:

# This is a basic configuration file with some examples, for device mapper
# multipath.

## Use user friendly names, instead of using WWIDs as names.

defaults {
                user_friendly_names yes
         }

##
devices {
        device {
                vendor "SolidFir"
                product "SSD SAN"
                path_grouping_policy multibus
                getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                path_selector "service-time 0"
                path_checker tur
                hardware_handler "0"
                failback immediate
                rr_weight uniform
                rr_min_io 1000
                rr_min_io_rq 1
                features "0"
                no_path_retry 24
                prio const
        }
}

multipaths {
        multipath {
                wwid 36f47acc1000000006167347a00000041
                alias dwqa-ora-fs
        }
        multipath {
                wwid 36f47acc1000000006167347a00000043
                alias dwqa-ora-grid
        }
        multipath {
                wwid 36f47acc1000000006167347a00000044
                alias dwqa-ora-dwqa1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000ae
                alias dwqa-ora-dwh2d10-1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000f9
                alias dwqa-ora-testdg-1
        }
}

So what I'm trying to do is read this file in and store it in a nested python dictionary (or list of nested dictionaries). We can ignore the comments lines (starting with #) for now. I have not come up with a clear/concise solution for this.

Here is my partial solution (doesn't give me the expected output yet, but it's close)

def nonblank_lines(f):
    for l in f:
        line = l.rstrip()
        if line:
            yield line

def __parse_conf__(self):
    conf = []
    with open(self.conf_file_path) as f:
        for line in nonblank_lines(f):
            if line.strip().endswith("{"): # opening bracket, start of new list of dictionaries
                current_dictionary_key = line.split()[0]
                current_dictionary = { current_dictionary_key : None }
                conf.append(current_dictionary)

            elif line.strip().endswith("}"): # closing bracket, end of new dictionary
                pass
                # do nothing...

            elif not line.strip().startswith("#"):
                if current_dictionary.values() == [None]:
                    # New dictionary... we should be appending to this one
                    current_dictionary[current_dictionary_key] = [{}]
                    current_dictionary = current_dictionary[current_dictionary_key][0]
                key = line.strip().split()[0]
                val = " ".join(line.strip().split()[1:])
                current_dictionary[key] = val

And this is the resulting dictionary (the list 'conf'):

[{'defaults': [{'user_friendly_names': 'yes'}]},
 {'devices': None},
 {'device': [{'failback': 'immediate',
              'features': '"0"',
              'getuid_callout': '"/lib/udev/scsi_id --whitelisted --device=/dev/%n"',
              'hardware_handler': '"0"',
              'no_path_retry': '24',
              'path_checker': 'tur',
              'path_grouping_policy': 'multibus',
              'path_selector': '"service-time 0"',
              'prio': 'const',
              'product': '"SSD SAN"',
              'rr_min_io': '1000',
              'rr_min_io_rq': '1',
              'rr_weight': 'uniform',
              'vendor': '"SolidFir"'}]},
 {'multipaths': None},
 {'multipath': [{'alias': 'dwqa-ora-fs',
                 'wwid': '36f47acc1000000006167347a00000041'}]},
 {'multipath': [{'alias': 'dwqa-ora-grid',
                 'wwid': '36f47acc1000000006167347a00000043'}]},
 {'multipath': [{'alias': 'dwqa-ora-dwqa1',
                 'wwid': '36f47acc1000000006167347a00000044'}]},
 {'multipath': [{'alias': 'dwqa-ora-dwh2d10-1',
                 'wwid': '36f47acc1000000006167347a000000ae'}]},
 {'multipath': [{'alias': 'dwqa-ora-testdg-1',
                 'wwid': '36f47acc1000000006167347a000000f9'}]},
 {'multipath': [{'alias': 'dwqa-ora-testdp10-1',
                 'wwid': '"SSolidFirSSD SAN 6167347a00000123f47acc0100000000"'}]}]

Obviously the "None"s should be replaced with nested dictionary below it, but I can't get this part to work.

Any suggestions? Or better ways to parse this file and store it in a python data structure?

Use Test Driven Development. Seriously, for a parser, when you need to cover various special and corner cases, you won't get far without automated tests that tell you that things still work after changes. That said, your question boils down to "Please fix/complete my code", which is off-topic here. — Ulrich Eckhardt, Jun 23 '16 at 20:10
I wasn't looking for a fix/complete my code answer (even though that's what I got). I was stuck and needed some suggestions, and the key here was to use recursion, that in it self would have been an acceptable answer. But yes, I will need to write some test cases to make sure this doesnt break with future updates -- thanks! — Polish Nightmare, Jun 28 '16 at 16:32

score 1 · Accepted Answer · edited May 23 '17 at 12:08

Try something like this:

def parse_conf(conf_lines):
    config = []

    # iterate on config lines
    for line in conf_lines:
        # remove left and right spaces
        line = line.rstrip().strip()

        if line.startswith('#'):
            # skip comment lines
            continue
        elif line.endswith('{'):
            # new dict (notice the recursion here)
            config.append({line.split()[0]: parse_conf(conf_lines)})
        else:
            # inside a dict
            if line.endswith('}'):
                # end of current dict
                break
            else:
                # parameter line
                line = line.split()
                if len(line) > 1:
                    config.append({line[0]: " ".join(line[1:])})
    return config

The function will get into the nested levels on the configuration file (thanks to recursion and the fact that the conf_lines object is an iterator) and make a list of dictionaries that contain other dictionaries. Unfortunately, you have to put every nested dictionary inside a list again, because in the example file you show how multipath can repeat, but in Python dictionaries a key must be unique. So you make a list.

You can test it with your example configuration file, like this:

with open('multipath.conf','r') as conf_file:
    config = parse_conf(conf_file)

    # show multipath config lines as an example
    for item in config:
        if 'multipaths' in item:
            for multipath in item['multipaths']:
                print multipath
                # or do something more useful

And the output would be:

{'multipath': [{'wwid': '36f47acc1000000006167347a00000041'}, {'alias': 'dwqa-ora-fs'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a00000043'}, {'alias': 'dwqa-ora-grid'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a00000044'}, {'alias': 'dwqa-ora-dwqa1'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a000000ae'}, {'alias': 'dwqa-ora-dwh2d10-1'}]}
{'multipath': [{'wwid': '36f47acc1000000006167347a000000f9'}, {'alias': 'dwqa-ora-testdg-1'}]}

recursion! that's what i needed! This is perfect, thank you! — Polish Nightmare, Jun 28 '16 at 16:31

handle · Answer 2 · 2016-06-24T11:59:34.503

If you don't use recursion, you will need some way of keeping track of your level. But even then it is difficult to have references to parents or siblings in order to add data (I failed). Here's another take based on Daniele Barresi's mention of recursion on the iterable input:

Data:

inp = """
# This is a basic configuration file with some examples, for device mapper
# multipath.

## Use user friendly names, instead of using WWIDs as names.

defaults {
                user_friendly_names yes
         }

##
devices {
        device {
                vendor "SolidFir"
                product "SSD SAN"
                path_grouping_policy multibus
                getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                path_selector "service-time 0"
                path_checker tur
                hardware_handler "0"
                failback immediate
                rr_weight uniform
                rr_min_io 1000
                rr_min_io_rq 1
                features "0"
                no_path_retry 24
                prio const
        }
}

multipaths {
        multipath {
                wwid 36f47acc1000000006167347a00000041
                alias dwqa-ora-fs
        }
        multipath {
                wwid 36f47acc1000000006167347a00000043
                alias dwqa-ora-grid
        }
        multipath {
                wwid 36f47acc1000000006167347a00000044
                alias dwqa-ora-dwqa1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000ae
                alias dwqa-ora-dwh2d10-1
        }
        multipath {
                wwid 36f47acc1000000006167347a000000f9
                alias dwqa-ora-testdg-1
        }
}
"""

Code:

import re
level = 0
def recurse( data ):
    """ """
    global level
    out = []
    level += 1
    for line in data:
        l = line.strip()
        if l and not l.startswith('#'):
            match = re.search(r"\s*(\w+)\s*(?:{|(?:\"?\s*([^\"]+)\"?)?)", l)
            if not match:
                if l == '}':
                    level -= 1
                    return out # recursion, up one level
            else:
                key, value = match.groups()
                if not value:
                    print( "  "*level, level, key )
                    value = recurse( data ) # recursion, down one level
                else:
                    print( "  "*level, level, key, value)
                out.append( [key,value] )
    return out  # once

result = recurse( iter(inp.split('\n')) )

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(result)

Resulting list with nested ["key", value] pairs:

[   ['defaults', [['user_friendly_names', 'yes']]],
    [   'devices',
        [   [   'device',
                [   ['vendor', 'SolidFir'],
                    ['product', 'SSD SAN'],
                    ['path_grouping_policy', 'multibus'],
                    [   'getuid_callout',
                        '/lib/udev/scsi_id --whitelisted --device=/dev/%n'],
                    ['path_selector', 'service-time 0'],
                    ['path_checker', 'tur'],
                    ['hardware_handler', '0'],
                    ['failback', 'immediate'],
                    ['rr_weight', 'uniform'],
                    ['rr_min_io', '1000'],
                    ['rr_min_io_rq', '1'],
                    ['features', '0'],
                    ['no_path_retry', '24'],
                    ['prio', 'const']]]]],
    [   'multipaths',
        [   [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a00000041'],
                    ['alias', 'dwqa-ora-fs']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a00000043'],
                    ['alias', 'dwqa-ora-grid']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a00000044'],
                    ['alias', 'dwqa-ora-dwqa1']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a000000ae'],
                    ['alias', 'dwqa-ora-dwh2d10-1']]],
            [   'multipath',
                [   ['wwid', '36f47acc1000000006167347a000000f9'],
                    ['alias', 'dwqa-ora-testdg-1']]]]]]

Alternatively, you could reformat your data into something a well-proven parser module would understand, e.g. `json`. — handle, Jun 24 '16 at 11:52
this is close, but the last values should be dicts not lists. thanks for the recursion idea though! — Polish Nightmare, Jun 28 '16 at 16:31
As I said, it's just another take on Daniele's iterator-recursion for you to build upon. Since your original all-dictionary request is not possible and your data seems record-like (same order), I figured you don't really need the keys at all, ergo no dictionary. Feel free to adapt. Main difference is the regular expression to match key and value of your data. — handle, Jun 29 '16 at 11:03

score 1 · Answer 3 · answered Jul 09 '21 at 13:14

Multipath conf is a bit of a pig to parse. This is what I use (originally based on the answer from daniele-barresi), the output is easier to work with than the other examples.

def get_multipath_conf():
    def parse_conf(conf_lines, parent=None):
        config = {}
        for line in conf_lines:
            line = line.split('#',1)[0].strip()
            if line.endswith('{'):
                key = line.split('{', 1)[0].strip()
                value = parse_conf(conf_lines, parent=key)
                if key+'s' == parent:
                    if type(config) is dict:
                        config = []
                    config.append(value)
                else:
                    config[key] = value
            else:
                # inside a dict
                if line.endswith('}'):
                    # end of current dict
                    break
                else:
                    # parameter line
                    line = line.split(' ',1)
                    if len(line) > 1:
                        key = line[0]
                        value = line[1].strip().strip("'").strip('"')
                        config[key] = value
        return config
    return parse_conf(open('/etc/multipath.conf','r'))

This is the output:

{'blacklist': {'devnode': '^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda|sdb)[0-9]*$'},
 'defaults': {'find_multipaths': 'yes',
              'max_polling_interval': '4',
              'polling_interval': '2',
              'reservation_key': '0x1'},
 'devices': [{'detect_checker': 'no',
              'hardware_handler': '1 alua',
              'no_path_retry': '5',
              'path_checker': 'tur',
              'prio': 'alua',
              'product': 'iSCSI Volume',
              'user_friendly_names': 'yes',
              'vendor': 'StorMagic'}]}

parsing linux iscsi multipath.conf into python nested dictionaries

3 Answers3