Pandas Appending to CSV check if file has header before appending

Asked Feb 20 '18 at 21:53

Active Jan 14 '23 at 14:22

Viewed 2,122 times

I am trying to append several pandas dataframes to a csv file but I cannot know ahead of time which dataframe will be appended first as they are each generated on different worker machines. I can append each one to pre-made empty csv file by doing: df.to_csv('test.csv', mode='a', header=False) but then my csv has no header and I just has data.

If I make header=True then I have copies of the header every time I append which is redundant since they are all the same. Is there any direct way to overcome this? I suppose I can check the file each time I want to append to see if there is a header but that feels inefficient.

asked Feb 20 '18 at 21:53

guy

1,021
2
16
40

Can't you make an empty csv file with the universal header and always append data without header to the file? – TYZ Feb 20 '18 at 21:54
3

Always use `mode='a'` with `header=(not os.path.exists(filename))`? – JaminSore Feb 20 '18 at 21:56
@YilunZhang Would I have to do that manually each time? – guy Feb 20 '18 at 21:58
@JaminSore That's pretty clever, gonna give it a go. – guy Feb 20 '18 at 21:59
1

Check this old answer: https://stackoverflow.com/questions/17530542/how-to-add-pandas-data-to-an-existing-csv-file/30292938#30292938 – Yaniv Feb 20 '18 at 21:59

Pandas Appending to CSV check if file has header before appending

0 Answers0