How to convert a dict to list of dicts in Python?

Question

I have the following problem: I would like to provide a simple function which iterates over a list of parameter set and does the same action for all set. The idea is that the either scalar values, or any iterable is passed as kwarg to the function, which then generates each set of kwargs, and then calls the action on them.

def simple_function(**kwargs):
    list_of_kwargs = convert_kwargs(kwargs)
    return [my_action(**kwargs_set) for kwargs_set in list_of_kwargs]

So provided that, I find it hard to implement convert_kwargs to be efficient and generic. Some example test cases I need to fulfill:

class ConvertParamsTestCase(unittest.TestCase):

    def test_convert_kwargs_no_list(self):
        kwargs = {'arg1': 1, 'arg2': "string", 'arg3': None}
        self.assertDictEqual(kwargs, convert_kwargs(kwargs))

    def test_convert_kwargs_all_lists(self):
        kwargs = {'arg1': [1, 2], 'arg2': ["string", "str"], 'arg3': [None, None]}
        expected = [{'arg1': 1, 'arg2': "string", 'arg3': None}, {'arg1': 2, 'arg2': "str", 'arg3': None}]
        self.assertListEqual(expected, convert_kwargs(kwargs))

    def test_convert_kwargs_one_is_string(self):
        kwargs = {'arg1': [1, 2], 'arg2': "string", 'arg3': [None, None]}
        expected = [{'arg1': 1, 'arg2': "string", 'arg3': None}, {'arg1': 2, 'arg2': "string", 'arg3': None}]
        self.assertListEqual(expected, convert_kwargs(kwargs))

    def test_convert_kwargs_one_is_iterator(self):
        kwargs = {'arg1': range(1, 3), 'arg2': "string", 'arg3': [None, None]}
        expected = [{'arg1': 1, 'arg2': "string", 'arg3': None}, {'arg1': 2, 'arg2': "string", 'arg3': None}]
        self.assertListEqual(expected, convert_kwargs(kwargs))

I have checked this without success: list of dicts to/from dict of lists

My main problems are that:

strings are iterables
I cannot check the length of a generator only by evaluating it

I would greatly appreciate any ideas!

Patrick Artner · Accepted Answer · 2019-02-22T09:12:25.243

You can check for strings and do something different for them:

if isinstance(something,str): 
    do_thing_with_string(something) 
else: 
    do_thing_with_iterable(something)

and you are not supposed to be able to check the length of a generator:

def gen():
    i = 0
    while True: 
        yield i 
        i+=1

endless = gen()         
for _ in range(20):        
    print(next(endless))  # works ... while len(endless) won't

You can use this as starting block. The code get's a dictionary, looks up the longest list inside and creates a set of dictionaries from it. You would need to enhance it to work with lists of lists. If you give a string, it will occure in any of the lists:

def convert(paramdict):
    l = []
    # lists ordered by lenght, longest first, strings in the back
    params =  sorted(paramdict.items(), 
                     key=lambda x:len(x[1]) if isinstance(x[1],list) else -1,
                     reverse=True)

    # max number of inner dicts to create
    maxone = len(params[0][1]) if isinstance(params[0][1],list) else 1

    for n_th in range(maxone):
        l.append({})
        for k,value in params:
            if isinstance(value,list):
                if len(value) > n_th:
                    l[-1][k] = value[n_th]
            # add other things here: tuple, set, generator
            # it might be easiest to convert f.e. set/tuple to list
            # for generator you would need to takewhile() from it until
            # you got as much as you need
            else:
                l[-1][k] = value

    return l


print(convert( {'arg1': [1, 2],       'arg2': ["string", "str", "s"], 
                'arg3': [None, None], 'arg4': "some lengthy strings"} ) )

Output:

[{'arg2': 'string', 'arg1': 1, 'arg3': None, 'arg4': 'some lengthy strings'}, 
 {'arg2': 'str',    'arg1': 2, 'arg3': None, 'arg4': 'some lengthy strings'}, 
 {'arg2': 's',                               'arg4': 'some lengthy strings'}]

If a list is too short it will miss from the output dict. You will still have to handle ordering when asserting "equalness" in case this algo does create inner dictionaries in a different order.

Thanks, I need to support all types of iterables, especially Tuple and Range objects, so I need to work on this a bit. Fortunatelly I do not need to support list of lists as of now. — Hodossy Szabolcs, Feb 22 '19 at 09:04

score 1 · Answer 2 · answered Feb 22 '19 at 09:09

To resolve the problem in a generic way consider to conform all the argx value to a list type.

If you define a protocol where all arguments are lists of values:

kwargs = {
           'arg1': range(0,3), 'arg2': ["s1"]*3, 'arg3': [10, 20]
         }

giving:

[{'arg1': 0, 'arg2': 's1', 'arg3': 10},
 {'arg1': 1, 'arg2': 's1', 'arg3': 20},
 {'arg1': 2, 'arg2': 's1', 'arg3': None}]

this is a solution:

kwargs = {'arg1': range(0, 3), 'arg2': ["s1"]*3, 'arg3': [10, 20]}

max_args = max([len(kwargs[key]) for key in kwargs])

list_of_kwargs = [{k: kwargs[k][index] if index < len(
    kwargs[k]) else None for k in kwargs} for index in range(max_args)]

Note that for passing the same string value it is a matter of ["value"]*3.

How to convert a dict to list of dicts in Python?

2 Answers2