Concatenate list of dataframes

Question

This is my code snippet:

import os 
import pandas as pd

path = os.getcwd()
files = os.listdir(path)
df = []

for f in files:
    data = pd.read_csv(f, usecols = [0,1,2,3,4])
    df.append(data)

temp = pd.concat(df)

where df is a list of dataframes:

0
DataFrame
(1, 5)
1
DataFrame
(7, 5)
2
DataFrame
(5, 5)
3
DataFrame
(10, 5)
4
DataFrame
(1, 5)
5
DataFrame
(2, 5)

I'm trying to stack these dataframes below one another and get one single dataframe as output. I've tried a bunch of combinations from SO Q&A but none seems to work. I feel this is easy. What am I doing wrong?

Just for clarity, are you sure that the two have the same dimensions? — Rashid 'Lee' Ibrahim, May 09 '19 at 17:23
@Rashid'Lee'Ibrahim: What do you mean? We're stacking rows below each other. — srkdb, May 09 '19 at 17:23
if it's a list just pass `frame = pd.concat(df)` should do the trick. — Umar.H, May 09 '19 at 17:24

rahlf23 · Accepted Answer · 2019-05-09T17:27:58.547

3

You do not need a for loop or list comprehension for this task. Simply do:

pd.concat(df)

where df is the list of dataframes.

Here is an example:

import pandas as pd
import numpy as np

df1 = pd.DataFrame(np.random.randint(0,100,size=(1,5)), columns=list('ABCDE'))
df2 = pd.DataFrame(np.random.randint(0,100,size=(7,5)), columns=list('ABCDE'))
df3 = pd.DataFrame(np.random.randint(0,100,size=(5,5)), columns=list('ABCDE'))
df = [df1, df2, df3]

concatenated = pd.concat(df)

Yields (for example):

    A   B   C   D   E
0  10  48  49  84  86
0  29   5  44  20  80
1  80   7   5   9  81
2  35  32  15  42  33
3  59  79  74  80  66
4  48  91  44  33  73
5  52  98  94  44  86
6  70  16  73  25  71
0  52  20  75  34  90
1  92  88  26  35  26
2  54   3  49  70  46
3  24  12  71  69  57
4   3  71  93  58  74

And you can use .reset_index(drop=True) to reset the index if you desire.

edited May 09 '19 at 17:27

answered May 09 '19 at 17:24

rahlf23

8,869
4
24
54

Yes, I tried that. It gives me a 196x65 answer. The column dimension should still be 5. I've no clue what's happening here. – srkdb May 09 '19 at 17:27
It gives (1, 5) (7, 5) (5, 5) (10, 5) (1, 5) (2, 5) and so on. – srkdb May 09 '19 at 17:37
No. 21. The df list has 21 items. 196 comes from the sum total of all the rows in the 21 dataframes. – srkdb May 09 '19 at 17:47
I checked all the 21 dataframes. All are (x,5) dimensions. The sum of all x rows = 196. – srkdb May 09 '19 at 17:55
I figured it out. I was completely missing the obvious. My column names look very similar but have a tweak in each. – srkdb May 09 '19 at 18:04
Glad you figured it out! – rahlf23 May 09 '19 at 18:16
2

@db18 don't forget to green tick :) – Umar.H May 10 '19 at 02:32

Concatenate list of dataframes

1 Answers1