Python: Reshaping a dataframe with multiple indexes

Question

So imagine we have a DataFrame like this:

In[1]: operinc_df
Out[1] :

        ticker1     ticker2      ticker3
    0   0.343573    0.654719    0.246643
    1   0.186861    0.219793    0.761056
    2   0.417347    0.058368    0.684918
    3   0.803177    0.014781    0.896704
    4   0.294515    0.488001    0.291187
    5   0.402278    0.368005    0.821096
    6   0.985514    0.378000    0.929529
    7   1.168360    0.729640    0.347064
    8   0.025802    1.337121    0.638399
    9   0.019182    2.257563    0.041164

And we have also another DataFrame with the same number of rows and of columns (with the same name):

In[2]: opex_df
Out[2] :


     ticker1    ticker2      ticker3
0   1.450770    0.227986    2.243050
1   1.212298    0.406004    1.212320
2   0.918931    0.677043    0.361878
3   0.566981    1.155675    0.295542
4   0.600614    0.872015    1.129760
5   0.470118    0.730027    1.112045
6   1.489904    0.522885    0.475244
7   1.626853    0.142996    0.758590
8   0.290340    1.175891    0.591020
9   1.472838    0.107094    0.715764

What I cannot figure out is how could I create another DataFrame fundamentals made of operinc_df and opex_df such that it looks like the DataFrame below (possibly with two levels of indexes):

In[3]: fundamentals
Out[3] :




              operinc_df    opex_df 
ticker1   0    0.343573    1.450770
ticker1   1    0.186861    1.212298
.         .    .           .
.         .    .           .
.         .    .           .
ticker1   9    0.019182    1.472838
ticker2   0    0.654719    0.227986
ticker2   1    0.219793    0.406004
.         .    .           .
.         .    .           .
.         .    .           .
ticker2   9    2.257563    0.107094
ticker3   0    0.246643    2.243050
ticker3   1    0.761056    1.212320
.         .    .           .
.         .    .           .
.         .    .           .
ticker3   9    0.041164    0.715764

Reading Reshaping dataframes in pandas based on column labels and Create a pandas DataFrame from multiple dicts gave me some insights (because I was also trying to do it by converting the original DataFrames firstly to dicts, pack operinc_df and opex_df by keys with a dictionary comprehension, and then with pandas.DataFrame.from_dict() try to create fundamentals_df. Nevertheless, it did not work out for me so far.

Do you have any ideas on how I could do this correctly ? Thank you very much in advance.

score 4 · Accepted Answer · answered Sep 11 '18 at 19:25

You can concat the transposed dataframes,

new_df = pd.concat([operinc_df.T, opex_df.T], axis = 1, keys=['operinc_df', 'opex_df']).stack()


            operinc_df  opex_df
ticker1 0   0.343573    1.450770
        1   0.186861    1.212298
        2   0.417347    0.918931
        3   0.803177    0.566981
        4   0.294515    0.600614
        5   0.402278    0.470118
        6   0.985514    1.489904
        7   1.168360    1.626853
        8   0.025802    0.290340
        9   0.019182    1.472838
ticker2 0   0.654719    0.227986
        1   0.219793    0.406004
        2   0.058368    0.677043
        3   0.014781    1.155675
        4   0.488001    0.872015
        5   0.368005    0.730027
        6   0.378000    0.522885
        7   0.729640    0.142996
        8   1.337121    1.175891
        9   2.257563    0.107094
ticker3 0   0.246643    2.243050
        1   0.761056    1.212320
        2   0.684918    0.361878
        3   0.896704    0.295542
        4   0.291187    1.129760
        5   0.821096    1.112045
        6   0.929529    0.475244
        7   0.347064    0.758590
        8   0.638399    0.591020
        9   0.041164    0.715764

sacuL · Answer 2 · 2018-09-11T19:25:06.433

You can do:

fundamentals = (operinc_df.stack().rename('operinc_df').to_frame()
                .join(opex_df.stack().rename('opex_df'))
                .swaplevel().sort_index())

>>> fundamentals

           operinc_df   opex_df
ticker1 0    0.343573  1.450770
        1    0.186861  1.212298
        2    0.417347  0.918931
        3    0.803177  0.566981
        4    0.294515  0.600614
        5    0.402278  0.470118
        6    0.985514  1.489904
        7    1.168360  1.626853
        8    0.025802  0.290340
        9    0.019182  1.472838
ticker2 0    0.654719  0.227986
        1    0.219793  0.406004
        2    0.058368  0.677043
        3    0.014781  1.155675
        4    0.488001  0.872015
        5    0.368005  0.730027
        6    0.378000  0.522885
        7    0.729640  0.142996
        8    1.337121  1.175891
        9    2.257563  0.107094
ticker3 0    0.246643  2.243050
        1    0.761056  1.212320
        2    0.684918  0.361878
        3    0.896704  0.295542
        4    0.291187  1.129760
        5    0.821096  1.112045
        6    0.929529  0.475244
        7    0.347064  0.758590
        8    0.638399  0.591020
        9    0.041164  0.715764

Python: Reshaping a dataframe with multiple indexes

2 Answers2