I have some code that reads in multiple csv files into a pandas dataframe. The problem is that the first two lines of all the files need to be ignored and I cannot figure out how to do this.
import pandas as pd
import glob
import os
path = r'D:\E\Traficc\migration\Zambia-Mining\DATA\24monthimport' # use your path
all_files = glob.glob(os.path.join(path, "*.csv")) # advisable to use os.path.join as this makes concatenation OS independent
df_from_each_file = (pd.read_csv(f) for f in all_files)
data = pd.concat(df_from_each_file, ignore_index=True)
# doesn't create a list, nor does it append to one
print(data.tail())
I have tried to use next(df) but I am getting an error that df is not iterable.
Can I do all this within the existing 1 line loop or do I need to break it up? What can I use to accomplish this?
