0

I want to read from a csv file using python. I am working on memory management, so I would like to ask: When I call read_csv of pandas module in python, the data from the file are loaded to the RAM memory and then read, or are they read straight from the disk? If I want to read the csv file in chunks will the data be loaded chunk by chunk in RAM ?

Nwlis
  • 63
  • 6
  • What do you mean by "read straight"? Where do you think the data is stored while the program is running if not in the ram? Regarding chunk reading, this is a duplicate. – kabanus Feb 19 '19 at 11:26
  • One possible duplicate is [How to read a 6 GB csv file with pandas](https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas) – kabanus Feb 19 '19 at 11:26
  • Basically my question is if breaking a file in to chunks, when travesring a chunk, all previous chunks are staying in RAM or deleted? – Nwlis Feb 19 '19 at 12:15
  • They could be, or not. Memory management is not up to you. But you definitely cannot access them anymore, so the OS is free to delete them (from the ram) at will. – kabanus Feb 19 '19 at 12:16

1 Answers1

0

See the documentation:

Also supports optionally iterating or breaking of the file into chunks.

and

chunksize : int, optional

Return TextFileReader object for iteration. See the IO Tools docs for more information on iterator and chunksize.

Selcuk
  • 57,004
  • 12
  • 102
  • 110