Create a 3D pandas dataframe

Question

I have this result which is a list of ndarrays. I'm trying to create a 3d pandas dataframe in a loop as follow:

col_names = list()
col_names += [('var%d(t)' % (j + 1)) for j in range(138)]
for b in range(0, B):
    result = get_new_results()
    df1 = pd.DataFrame(data=result, columns=col_names)
    df = pd.concat([df, df1], axis=0)

But this will add every single result as new rows at the end of the df. So if each result is 112 ndarrays of shape (138,), similar to a table of size 112x138, my df will have shape=(448, 138) after 4 iterations of b (448=112*4). And the row indexes will go from 0 to 111 and resets again from 0 to 112. The df.ndim is equal to 2. But what I need is to create a 3d dataframe as follow:

0  var1(t)     -0.863838, ... # 112 different values for var1(t)
   var2(t)      0.299368, ... # 112 different values for var2(t)
   ...           ...
   var137(t)    0.299368, ... # 112 different values for var137(t)

1  var1(t)      0.206053, ... # 112 different values for var1(t)
   var2(t)      1.063327, ... # 112 different values for var2(t)
   ...           ...
   var137(t)    0.299368, ... # 112 different values for var137(t)

2  var1(t)     -2.213588, ... # 112 different values for var1(t)
   var2(t)     -0.251905, ... # 112 different values for var2(t)
   ...           ...
   var137(t)    0.299368, ... # 112 different values for var137(t)
...

B  var1(t)    0.408204, ... # 112 different values for var1(t)
   var2(t)    1.266143, ... # 112 different values for var2(t)
   ...           ...
   var137(t)    0.299368, ... # 112 different values for var137(t)

I believe I can create it using pd.MultiIndex but I couldn't come up with a reasonable solution. Can someone please help me to fix it?

Possible duplicate of [Constructing 3D Pandas DataFrame](https://stackoverflow.com/questions/24290495/constructing-3d-pandas-dataframe) — G. Anderson, Oct 29 '18 at 15:32
@G.Anderson I've seen that post and I would say that's a different question. Of course that guy also tries to create a 3d dataframe, but there he already has arrays that he wants to somehow reshape it. In my case, I need to stack these dataframes in way that it auto generates the required indexes in 3d — Birish, Oct 29 '18 at 15:49

score 2 · Accepted Answer · answered Oct 31 '18 at 14:59

Here is how I managed to create the dataframe:

frames = []
col_names = list()
col_names += [('var%d(t)' % (j+1)) for j in range(138)]
for i in range(0, B):
    result = get_new_results()
    df_tmp = pd.DataFrame(data=results, columns=col_names)
    frames.append(df_tmp)
df = pd.concat(frames, axis=0, keys=range(0, 112))

Create a 3D pandas dataframe

1 Answers1