How to export a dataframe to csv in chunks of 500 rows

Question

I have a large dataframe which i export to a CSV and upload to a third party product. The third party product can only accept uploads of max 500 rows of my data so I am wondering how to export a dataframe into smaller files.

at the moment my code reads:

df.to_csv("Export.csv",index=False)

But ideally would like code to export files so that it gives:

Export1.csv
Export2.csv
Export3.csv

etc until it is all completed and each with 500 rows (and then whatever is left over). Any help would be much appreciated!

Does this answer your question? [Writing large Pandas Dataframes to CSV file in chunks](https://stackoverflow.com/questions/38531195/writing-large-pandas-dataframes-to-csv-file-in-chunks) — Henry Yik, Jul 06 '20 at 03:11

vraj · Accepted Answer · 2020-07-06T03:32:50.873

5

you can use df.groupby function to split df in a chunk of 500. And save each chunk. You can see the reference here (https://stackoverflow.com/a/25703030/6996326)

import numpy as np
no_of_rows = 500
for k,g in df.groupby(np.arange(len(df))//no_of_rows):
    g.to_csv('Export{}.csv'.format(k+1), index=False)

edited Jul 06 '20 at 03:32

answered Jul 06 '20 at 03:19

vraj

108
3

How to export a dataframe to csv in chunks of 500 rows

1 Answers1