0

I am trying to load multiple CSVs into a single pandas dataframe. They are all in one file, and all have the same column structure. I have tried a few different methods from a few different threads, and all return the error 'ValueError: No objects to concatenate.' I'm sure the problem is something dumb like my file path? This is what I've tried:

temps = pd.concat(map(pd.read_csv, glob.glob(os.path.join('./Resources/temps', "*.csv"))))

Also this:

path = r'./Resources/temps'                     
temps_csvs = glob.glob(os.path.join(path, "*.csv"))     

df_for_each_csv = (pd.read_csv(f) for f in temps_csvs)
temps_df = pd.concat(df_for_each_csv, ignore_index = True)```

Thanks for any help!
Alawler12
  • 5
  • 3
  • you might be right; it's possible pandas is not seeing any file. what do you get for `df_for_each_csv`? – sammywemmy Feb 28 '21 at 00:21
  • first off make sure when you print `temps_csvs` that there's actually a list of files in there – Chris Feb 28 '21 at 00:30
  • @sammywemmy at 0x00000220B47BDC48> – Alawler12 Feb 28 '21 at 00:34
  • @Chris yeah i just fixed my file path and the list populates correctly, but now i get this error: ```ParserError: Error tokenizing data. C error: Expected 5 fields in line 1394, saw 6``` – Alawler12 Feb 28 '21 at 00:35

2 Answers2

1

It might not be as helpful as other answers, but when I tried running your code, it work perfectly fine. The only difference that conflicted was that I changed the path to be like this:

temps_csvs = glob.glob(os.path.join(os.getcwd(), "*.csv"))

df_for_each_csv = (pd.read_csv(f) for f in temps_csvs)
temps_df = pd.concat(df_for_each_csv, ignore_index = True)

and put the script in the same folder as to where the csv files are.

EDIT: I saw your comment about how you are having an error ParserError: Error tokenizing data. C error: Expected 5 fields in line 1394, saw 6

This means that the csv files don't have the same number of columns, here is a question that deals with a similar issue, maybe it will help : Reading a CSV file with irregular number of columns using Pandas

Cristian
  • 303
  • 2
  • 8
  • you're right! there were a few rouge data points chilling in a 6th column. ...i feel silly. thanks for your help! – Alawler12 Feb 28 '21 at 01:17
0

Change tuple to list on the third line. [pd.read_csv(f) for f in temps_csvs]

or add tuple to it: tuple(pd.read_csv(f) for f in temps_csvs)

Tuple Comprehension doesn't work this way. See Why is there no tuple comprehension in Python?

Huakun Shen
  • 312
  • 2
  • 9