0

I have a list of many DataFrames. Each DataFrame is a set of various measurements corresponding to a timestamp. Since many measurements can correspond to the same moment in time, there are many duplicate index entries in the time indices of the DataFrames.

I want to merge this list of DataFrames and obviously to keep the duplicate indices. How can this be done? I have checked this question but the solutions are applicable to the case of merging only two DataFrames, not a list of many DataFrames. The concat functionality apparently cannot handle duplicate indices.

BlandCorporation
  • 1,324
  • 1
  • 15
  • 33
  • 1
    I beilive [concat](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html) is the way to go. Consider creating a [MCVE] for concise answer. – harvpan Jun 01 '18 at 19:06
  • @HarvIpan Thanks, but [apparently](https://stackoverflow.com/questions/27719407/pandas-concat-valueerror-shape-of-passed-values-is-blah-indices-imply-blah2/27910308) `concat` cannot handle duplicate indices. – BlandCorporation Jun 01 '18 at 19:07
  • 1
    [Concat does work](https://stackoverflow.com/questions/24684441/pandas-concatenate-dataframe-and-keep-duplicate-indices) – harvpan Jun 01 '18 at 19:07
  • Possible duplicate of [Pandas: Concatenate dataframe and keep duplicate indices](https://stackoverflow.com/questions/24684441/pandas-concatenate-dataframe-and-keep-duplicate-indices) – harvpan Jun 02 '18 at 05:36

1 Answers1

0

See @HarvIpan comment: that is correct. You can concat a list a pandas dataframes:

import pandas as pd
df = pd.DataFrame({'a':[1,2,3],'b':['a','b','c']})
df.set_index('a', inplace=True)

df2 = pd.DataFrame({'a':[1,2,3],'b':['d','e','f']})
df2.set_index('a', inplace=True)

df3 = pd.DataFrame({'a':[1,2,3],'c':['g','e','h']})
df3.set_index('a', inplace=True)

list_of_dfs = [df,df2,df3]

pd.concat(list_of_dfs, sort=False)

    b   c
a       
1   a   NaN
2   b   NaN
3   c   NaN
1   d   NaN
2   e   NaN
3   f   NaN
1   NaN g
2   NaN e
3   NaN h
It_is_Chris
  • 13,504
  • 2
  • 23
  • 41