How to ConfigParse a file keeping multiple values for identical keys?

Question

I need to be able to use the ConfigParser to read multiple values for the same key. Example config file:

[test]
foo = value1
foo = value2
xxx = yyy

With the 'standard' use of ConfigParser there will be one key foo with the value value2. But I need the parser to read in both values.

Following an entry on duplicate key I have created the following example code:

from collections import OrderedDict
from ConfigParser import RawConfigParser

class OrderedMultisetDict(OrderedDict):
    def __setitem__(self, key, value):

        try:
            item = self.__getitem__(key)
        except KeyError:
            super(OrderedMultisetDict, self).__setitem__(key, value)
            return

        print "item: ", item, value
        if isinstance(value, list):
            item.extend(value)
        else:
            item.append(value)
        super(OrderedMultisetDict, self).__setitem__(key, item)


config = RawConfigParser(dict_type = OrderedDict)
config.read(["test.cfg"])
print config.get("test",  "foo")
print config.get("test",  "xxx")

config2 = RawConfigParser(dict_type = OrderedMultisetDict)
config2.read(["test.cfg"])
print config2.get("test",  "foo")
print config.get("test",  "xxx")

The first part (with config) reads in the config file us 'usual', leaving only value2 as the value for foo (overwriting/deleting the other value) and I get the following, expected output:

value2
yyy

The second part (config2) uses my approach to append multiple values to a list, but the output instead is

['value1', 'value2', 'value1\nvalue2']
['yyy', 'yyy']

How do I get rid of the repetitive values? I am expecting an output as follows:

['value1', 'value2']
yyy

or

['value1', 'value2']
['yyy']

(I don't mind if EVERY value is in a list...). Any suggestions welcome.

score 18 · Accepted Answer · edited Oct 25 '17 at 15:47

18

After a small modification, I was able to achieve what you want:

class MultiOrderedDict(OrderedDict):
    def __setitem__(self, key, value):
        if isinstance(value, list) and key in self:
            self[key].extend(value)
        else:
            super(MultiOrderedDict, self).__setitem__(key, value)
            # super().__setitem__(key, value) in Python 3

config = ConfigParser.RawConfigParser(dict_type=MultiOrderedDict)
config.read(['a.txt'])
print config.get("test",  "foo")
print config.get("test",  "xxx")

Outputs:

['value1', 'value2']
['yyy']

edited Oct 25 '17 at 15:47

bernie

9,820
5
62
92

answered Apr 06 '13 at 08:57

Nathan Villaescusa

17,331
4
53
56

I have no idea why `super(OrderedDict, self)` works but not `super(MultiOrderedDict, self)`. – Nathan Villaescusa Apr 06 '13 at 09:22
Yes, perfect! Except the `super` anomaly. Maybe it *is* the baseclass of `OrderedDict` then... – Alex Apr 06 '13 at 09:43
1

Is there any way to make this return a single value instead of a list when there is only one value? ['value1', 'value2'] and yyy instead of ['yyy'] – Vangelis Tasoulas Feb 20 '14 at 12:00
Do you have a way to make this work in python 3? I get an error: `configparser.DuplicateOptionError`, and I don't really know what I'm doing. – bgStack15 Jul 22 '16 at 13:00
3

@bgStack15 you should set `strict` to `False`, like this: `config = configparser.RawConfigParser(dict_type=MultiOrderedDict, strict=False)` – Roberto Oct 20 '16 at 12:52
2

This seems to return values with a newline character instead of an array for 2.6/2.7 – Sarkie Sep 04 '18 at 16:19
2

I am using 3.7 and I too am getting a string separated by `'\n'`, it appears to be something related to the `configparser.SectionProxy` being returned. – Melendowski Jul 27 '20 at 17:12

score 8 · Answer 2 · answered Aug 15 '17 at 09:11

The accepted answer breaks config.sections(), it returns always an empty list (tested with Python 3.5.3). Replacing super(OrderedDict, self).__setitem__(key, value) by super().__setitem__(key, value) fixes this, but now config.get(section, key) returns a concatenated string, no longer a list of strings.

My solution is:

class ConfigParserMultiValues(collections.OrderedDict):

    def __setitem__(self, key, value):
        if key in self and isinstance(value, list):
            self[key].extend(value)
        else:
            super().__setitem__(key, value)

    @staticmethod
    def getlist(value):
        return value.split(os.linesep)

    config = configparser.ConfigParser(strict=False, empty_lines_in_values=False, dict_type=ConfigParserMultiValues, converters={"list": ConfigParserMultiValues.getlist})
    ...
    values = config.getlist("Section", "key") # => ["value1", "value2"]

The config INI file accepts duplicate keys:

[Section]
    key = value1
    key = value2

I fixed the wrong super() call in the accepted answer and sections() now works. — bernie, Oct 25 '17 at 15:47

score 5 · Answer 3 · answered Apr 22 '20 at 13:21

in python 3.8 you need to also add strict=False:

class MultiOrderedDict(OrderedDict):
    def __setitem__(self, key, value):
        if isinstance(value, list) and key in self:
            self[key].extend(value)
        else:
            super().__setitem__(key, value)

config = ConfigParser.RawConfigParser(dict_type=MultiOrderedDict, strict=False)
config.read(['a.txt'])
print config.get("test",  "foo")
print config.get("test",  "xxx")

Thank you! This fixed my issues in Python 3.7.4 – nimig18 Jan 30 '21 at 21:44 — nimig18, Jan 30 '21 at 21:44

user2185338 · Answer 4 · 2016-07-10T08:16:35.393

More examples Multiple values in test.cfg.

[test]
foo = value1
foo = value2
 value3
xxx = yyy

<whitespace>value3 append value3 to foo list.

ConfigParser converts the list to a string.

/usr/lib/python2.7/ConfigParser.pyc in _read(self, fp, fpname)
    552             for name, val in options.items():
    553                 if isinstance(val, list):
--> 554                     options[name] = '\n'.join(val)
    555

value before converting is always list or dict (MultiOrderedDict).

Try this - with it, config.items works:

from collections import OrderedDict
import ConfigParser

class MultiOrderedDict(OrderedDict):
    def __setitem__(self, key, value):
        if key in self:
            if isinstance(value, list):
                self[key].extend(value)
                return
            elif isinstance(value,str):
                return # ignore conversion list to string (line 554)
        super(MultiOrderedDict, self).__setitem__(key, value)

config = ConfigParser.RawConfigParser(dict_type=MultiOrderedDict)
config.read(['test.cfg'])
print config.get("test",  "foo")
print config.get("test",  "xxx")
print config.items("test")

Outputs:

['value1', 'value2', 'value3']
['yyy']
[('foo', ['value1', 'value2', 'value3']), ('xxx', ['yyy'])]

Another implementation MultiOrderedDict

class MultiOrderedDict(OrderedDict):
    def __setitem__(self, key, value):
        if key in self:
            if isinstance(value, list):
                self[key].extend(value)
                return
            elif isinstance(value,str):
                if len(self[key])>1:
                    return
        super(MultiOrderedDict, self).__setitem__(key, value)

Outputs:

['value1', 'value2', 'value3']
yyy
[('foo', ['value1', 'value2', 'value3']), ('xxx', 'yyy')]

I found this comment unclear at first but now I understand that it is pointing out where in `ConfigParser` the list value gets concatenated into a string with newlines, and some workarounds that make `MultiOrderedDict` ignore the attempt to set the concatenated value. Unfortunately those workarounds have to make certain assumptions about how `Configparser` behaves; I think I prefer something like Stefan Bohlein's answer which just re-splits the value, but this explains why that splitting is required. — doshea, Sep 11 '22 at 04:18

score -2 · Answer 5 · edited May 23 '17 at 12:10

Just a little bit change to @abarnert's answer, otherwise it calles __setitem__ recursively, and won't stop for some reason.

ini file:

[section]
key1   = value1
key2[] = value21
key2[] = value22

Python:

class MultiOrderedDict(OrderedDict):
    LIST_SUFFIX = '[]'
    LIST_SUFFIX_LEN = len(LIST_SUFFIX)

    def __setitem__(self, key, value):
        if key.endswith(self.LIST_SUFFIX):
            values = super(OrderedDict, self).setdefault(key, [])
            if isinstance(value, list):
                values.extend(value)
            else:
                values.append(value)
        else:
            super(MultiOrderedDict, self).__setitem__(key, value)

    def __getitem__(self, key):
        value = super(MultiOrderedDict, self).__getitem__(key)
        if key.endswith(self.LIST_SUFFIX) and not isinstance(value, list):
            value = value.split('\n')
        return value

Test:

def test_ini(self):
    dir_path = os.path.dirname(os.path.realpath(__file__))
    config = RawConfigParser(dict_type=MultiOrderedDict, strict=False)
    config.readfp(codecs.open('{}/../config/sample.ini'.format(dir_path), encoding="utf_8_sig"))
    self.assertEquals(config.get("section1", "key1"), 'value1')
    self.assertEquals(config.get("section1", "key2[]"), ['value21', 'value22'])

How to ConfigParse a file keeping multiple values for identical keys?

5 Answers5

Linked

Related