Convert arbitrary-length nested lists to format for pandas dataframe?

Question

I have an optimization script which outputs data in a format similar to the fake lists below:

l1 = [1,2,3,4]
l2 = [[5,6,7,8],
      [9,10,11,12]]
l3 = [[13,14,15,16],
      [17,18,19,20]]

All lists are of the same length always (at least, the lists which contain values), but some are stored within a larger list container (l2 or l3 in this example). I ultimately want each individual list to be a separate column in a pandas dataframe (e.g., 1,2,3,4 is a column, 5,6,7,8 is a column, etc.). However, the number of lists within l2 or l3 will vary.

What is the best way to unpack these lists or otherwise get into a pandas dataframe?

What's throwing me off is doing this in a way which will always work regardless of the number of lists in l2 and l3.

score 2 · Accepted Answer · answered Apr 12 '23 at 20:36

You can use numpy to ravel your lists then reshape since one dimension is fixed:

import pandas as pd
import numpy as np

L = [l1, l2, l3]
N = 4
df = pd.DataFrame(np.concatenate([np.ravel(l) for l in L]).reshape(-1, N).T)

Output:

>>> df
   0  1   2   3   4
0  1  5   9  13  17
1  2  6  10  14  18
2  3  7  11  15  19
3  4  8  12  16  20

score 1 · Answer 2 · answered Apr 12 '23 at 20:25

Iterate over your lists and wrap the non-nested ones so that every list is a nested list of arbitrary length. After that you can concatenate them and transpose to get your final result:

from itertools import chain

arbitrary_lists = [l1, l2, l3]
df = pd.DataFrame(chain.from_iterable(
    [l] if not isinstance(l[0], list) else l for l in arbitrary_lists))
df = df.transpose()
df 

   0  1   2   3   4
0  1  5   9  13  17
1  2  6  10  14  18
2  3  7  11  15  19
3  4  8  12  16  20

Convert arbitrary-length nested lists to format for pandas dataframe?

2 Answers2