Split a dataframe of dataframes and insert a column

Question

I have a large set of date-stamped files whose filenames are dates. Each file has a set of records, with the same structure, that is in a dataframe.

I would like to make these into a single dataframe, with the filename as a column in it.

The sample code is below.

import pandas as pd
import numpy as np
df1 = pd.DataFrame({
    'a': np.random.rand(6),
    'b': np.random.choice([4,9,np.nan], 6),
})

df2 = df1*3
df3 = np.sqrt(df2)
df4 = df1*df3
df5 = df2*0.5
df6 = df2*df2

df = pd.DataFrame({
    'fname': pd.date_range('1/1/2018', periods=6, freq='D'),
    'df': [df1, df2, df3, df4, df5, df6]
})

Output df expected is:

    a           b   fname
0   0.634776    NaN 2018-01-01
1   0.616349    NaN 2018-01-02
2   0.502559    NaN 2018-01-03
3   0.545378    4.0 2018-01-04
4   0.658668    4.0 2018-01-05
5   0.822417    9.0 2018-01-06
0   1.904328    NaN 2018-01-01
1   1.849046    NaN 2018-01-02
2   1.507678    NaN 2018-01-03
3   1.636133    2.0 2018-01-04
4   1.976004    2.0 2018-01-05
5   2.467252    3.0 2018-01-06
...and so on...

score 1 · Accepted Answer · answered Dec 03 '18 at 08:33

I think better is create dict, so then is possible use concat:

d = dict(zip(pd.date_range('1/1/2018', periods=6, freq='D'),
              [df1, df2, df3, df4, df5, df6]))

df = (pd.concat(d)
       .reset_index(level=1, drop=True)
       .rename_axis('fname')
       .reset_index()
       )

Or pass list of DataFrames to concat with parameter keys:

df = (pd.concat([df1, df2, df3, df4, df5, df6], 
               keys=pd.date_range('1/1/2018', periods=6, freq='D'))
       .reset_index(level=1, drop=True)
       .rename_axis('fname')
       .reset_index()
       )

print (df.head(10))
       fname         a     b
0 2018-01-01  0.075056   9.0
1 2018-01-01  0.646019   9.0
2 2018-01-01  0.656301   NaN
3 2018-01-01  0.420031   4.0
4 2018-01-01  0.239017   9.0
5 2018-01-01  0.921222   NaN
6 2018-01-02  0.225168  27.0
7 2018-01-02  1.938057  27.0
8 2018-01-02  1.968903   NaN
9 2018-01-02  1.260093  12.0

score 0 · Answer 2 · answered Dec 03 '18 at 08:38

To get the dataframes together i think you should use pd.Concat:https://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html

What code do you use to get read in the data? If it's something like this

file_name = os.listdir(path)
file_name = [file_name for file_name in glob.glob('* *')]
df = pd.DataFrame()
dfs = []
for file_name in file_name:
data = pd.read_csv(file_name, 'Sheet1')


data['DateTime'] = pd.to_datetime(file_name[:10])
dfs.append(data)

df = pd.concat(dfs, ignore_index=True)

Where file_name[:10] gets the date from the files based on character amount

Its similar to question i had a while ago: https://stackoverflow.com/questions/52969523/how-to-add-a-date-from-filename-to-a-time-column-to-make-datetime-column-python — Martijn van Amsterdam, Dec 03 '18 at 08:39

Split a dataframe of dataframes and insert a column

2 Answers2