0
input_filelist = os.listdir(path)
print(input_filelist)
merge_data = pd.concat(pd.read_csv(file).assign(sourcefilename = file) for file in input_filelist)
merge_data

I am trying to concatenate all the csv files of input_filelist into a same csv file. I do not know what is the correct process for that. This code is giving me the following error : UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 10: invalid start byte

df = pd.DataFrame()
for file in input_filelist:
     if file.endswith('.csv'):
        df = df.append(pd.read_csv(file), ignore_index=True) 
df.head()
df.to_csv('Consolidated.csv')

I have tried this code too, which is creating a new csv file, but it is totally blank. All the data is not being merged and showing.

Panagiotis Kanavos
  • 120,703
  • 13
  • 188
  • 236
Srijani
  • 1
  • 1
  • The error says that the file isn't UTF8. You'll have to find what the encoding is and pass it as an argument to `read_csv`. You could try with `encoding='latin_1'`. Whoever creates this file should switch to UTF8 – Panagiotis Kanavos Oct 31 '22 at 14:25

0 Answers0