Python reading files in chunks

Asked Aug 05 '22 at 10:51

Active Aug 05 '22 at 10:55

Viewed 77 times

I have a large csv 20 gb file that i want to read to DataFrame. I am using the following code

import pyodbc
import sqlalchemy
import pandas
chunks in pd.read_csv("test.csv", chunksize = 1000):
       print(list(chunks))

i am able to execute it without any memory issue but i want to unite to chunks to a single unit so that i can transfer the file to Postgres. I have created the engine for Postgres using Sqlalchemy engine.

I am new to python. Please suggest how to loop and write the chunks to database. I read somewhere that generators are also a good option. TIA.

edited Aug 05 '22 at 10:55

asked Aug 05 '22 at 10:51

Will-Iam

2

not familiar with SQL myself, but does this post help? https://stackoverflow.com/q/70794118/17270192 – Omniswitcher Aug 05 '22 at 11:00
@Omniswitcher Thank you, this looks like a good approach. I ll try it out right away. – Will-Iam Aug 05 '22 at 11:04
I am quite curious, did it work out? – Omniswitcher Aug 19 '22 at 14:56
Does this answer your question? [Appending chunks of CSV to Database with Pandas](https://stackoverflow.com/questions/70794118/appending-chunks-of-csv-to-database-with-pandas) – Omniswitcher May 02 '23 at 11:03

Python reading files in chunks

0 Answers0