What is a Pythonic way of doing the following transformation on a list of dicts?

Question

I have a list of dicts like this:

l = [{'name': 'foo', 'values': [1,2,3,4]}, {'name': 'bar', 'values': [5,6,7,8]}]

and I would like to obtain an output of this form:

>>> [('foo', 'bar'), ([1,2,3,4], [5,6,7,8])]

But short of for-looping and appending I don't see a solution. Is there a smarter way than doing this?

names = []
values = []
for d in l:
    names.append(d['name'])
    values.append(d['values'])

Answer 1

Use generator expression:

l = [{'name': 'foo', 'values': [1,2,3,4]}, {'name': 'bar', 'values': [5,6,7,8]}]
v = [tuple(k["name"] for k in l), tuple(k["values"] for k in l)]
print(v)

Output:

[('foo', 'bar'), ([1, 2, 3, 4], [5, 6, 7, 8])]

Answer 2

I would use a list comprehension (much like eyllanesc's) if I was writing this code for public consumption. But just for fun, here's a one-liner that doesn't use any fors.

>>> l = [{'name': 'foo', 'values': [1,2,3,4]}, {'name': 'bar', 'values': [5,6,7,8]}]
>>> list(zip(*map(dict.values, l)))
[('foo', 'bar'), ([1, 2, 3, 4], [5, 6, 7, 8])]

(Note that this only reliably works if dictionaries preserve insertion order, which is not the case in all versions of Python. CPython 3.6 does it as an implementation detail, but it is only guaranteed behavior as of 3.7.)

Quick breakdown of the process:

• dict.values returns a dict_values object, which is an iterable containing all the values of the dict.
• map takes each dictionary in l and calls dict.values on it, returning an iterable of dict_values objects.
• zip(*thing) is a classic "transposition" recipe, which takes an iterable-of-iterables and effectively flips it diagonally. E.g. [[a,b],[c,d]] becomes [[a,c], [b,d]]. This puts all the names into one tuple, and all the values into another.
• list converts the zip object into a list.

Answer 3

You can use operator.itemgetter to guarantee ordering of values:

from operator import itemgetter

fields = ('name', 'values')
res = list(zip(*map(itemgetter(*fields), L)))

print(res)

[('foo', 'bar'), ([1, 2, 3, 4], [5, 6, 7, 8])]

If, assuming Python 3.6+, you cannot guarantee appropriate insertion-ordering of dictionaries within your input list, you will need to explicitly define an order as above.

Performance

While a list of "tuple comprehensions" works, it becomes unreadable and inefficient when querying more than a couple of fields:

from operator import itemgetter

n = 10**6
L = [{'name': 'foo', 'values': [1,2,3,4], 'name2': 'zoo', 'name3': 'xyz',
      'name4': 'def'}, {'name': 'bar', 'values': [5,6,7,8], 'name2': 'bart',
      'name3': 'abc', 'name4': 'ghi'}] * n

%timeit [tuple(k["name"] for k in L), tuple(k["values"] for k in L),\
         tuple(k["name2"] for k in L), tuple(k["name3"] for k in L),
         tuple(k["name4"] for k in L)]

%timeit fields = ('name', 'values', 'name2', 'name3' ,'name4');\
        list(zip(*map(itemgetter(*fields), L)))

1 loop, best of 3: 1.25 s per loop
1 loop, best of 3: 1.04 s per loop

Answer 4

This may not be exactly what you had in mind, but for tabular data like this I find that pandas is usually the best solution in the long run:

>>> import pandas as pd
>>> l = [{'name': 'foo', 'values': [1,2,3,4]}, {'name': 'bar', 'values': [5,6,7,8]}]
>>> df = pd.DataFrame(l)
  name        values
0  foo  [1, 2, 3, 4]
1  bar  [5, 6, 7, 8]

Usually you use the data frame directly for anything you would need to do, but you can also convert it to a list-based data structure:

>>> df['name'].tolist(), df['values'].tolist()
(['foo', 'bar'], [[1, 2, 3, 4], [5, 6, 7, 8]]) 

Answer 5

Not sure about performance, but here's another take using zip() and unpacking:

list(zip(*[tuple(i.values()) for i in l]))

# [('foo', 'bar'), ([1, 2, 3, 4], [5, 6, 7, 8])]

Edit: As @DeepSpace pointed out, it can be further reduced down to:

list(zip(*(i.values() for i in l)))

Here's a longer, but more explicit answer if you want to define the orders yourself:

list(zip(*(tuple(map(lambda k: i.get(k), ('name', 'values'))) for i in l)))

# [('foo', 'bar'), ([1, 2, 3, 4], [5, 6, 7, 8])]

Answer 6

use map for this

names = tuple(map(lambda d: d['name'], l))
values = tuple(map(lambda d: d['values'], l))
result = [names, values]

Answer 7

First : your code is fine, readable and efficient, which sounds Pythonic to me. Note that you probably don't want a list of tuples, though. Tuples are immutable, so you wouldn't be able to append another name to names.

With a single dict

If names are unique, you could convert your list of dicts to a large dict:

>>> l = [{'name': 'foo', 'values': [1,2,3,4]}, {'name': 'bar', 'values': [5,6,7,8]}]
>>> data = {d['name']:d['values'] for d in l}
>>> data
{'foo': [1, 2, 3, 4], 'bar': [5, 6, 7, 8]}

You can get the desired information directly:

>>> data.keys()
dict_keys(['foo', 'bar'])
>>> data.values()
dict_values([[1, 2, 3, 4], [5, 6, 7, 8]])

If you really want a list of lists:

>>> [list(data.keys()), list(data.values())]
[['foo', 'bar'], [[1, 2, 3, 4], [5, 6, 7, 8]]]

With pandas

If you're working with a large list of dicts, you might want to consider pandas.

You could initialize a DataFrame directly:

>>> import pandas as pd
>>> df = pd.DataFrame([{'name': 'foo', 'values': [1,2,3,4]}, {'name': 'bar', 'values': [5,6,7,8]}])
>>> df
  name        values
0  foo  [1, 2, 3, 4]
1  bar  [5, 6, 7, 8]

If you need the names as an iterable, you can get the corresponding column:

>>> df['name']
0    foo
1    bar
Name: name, dtype: object

If you really need a list of names:

>>> list(df['name'])
['foo', 'bar']

To get the names and values together:

>>> df.values.T
array([['foo', 'bar'],
       [list([1, 2, 3, 4]), list([5, 6, 7, 8])]], dtype=object)

Answer 8

Here's a recursive way of doing it:

def trans(l):
  if l:
    res = trans(l[1:])
    res[0], res[1] = (l[0]['name'],) + res[0], (l[0]['values'],) + res[1]
    return res
  return [(),()]

Answer 9

Just like this:

(lambda f:
    lambda l, r=[(), ()]: f(f, l, r)
)(lambda g, l, r:
    r if len(l) == 0  else g(g, l[1:], [r[0]+(l[0]['name'],), r[1]+(l[0]['values'],)])
)([
    {'name': 'foo', 'values': [1, 2, 3, 4]},
    {'name': 'bar', 'values': [5, 6, 7, 8]},
    {'name': 'baz', 'values': [9, 9, 9, 9]}
])

Result:

[('foo', 'bar', 'baz'), ([1, 2, 3, 4], [5, 6, 7, 8], [9, 9, 9, 9])]