0

I'm reading a tutorial about python for very beginners and at some point the author define some ways to work with files. My doubt is related to memory management and file arrays.

#open a file for reading
file = open(filename, 'r')
#this turns the file into an array. 
lines = file.readlines() `

Python is smart enough to check the file size? What happens if the file has about 1 GB of data? Python will throw the entire file to the memory (array)? Or this is a lazy operation, just like C/C++ does!

Thanks in advance.

zhangxaochen
  • 32,744
  • 15
  • 77
  • 108
Guilherme
  • 443
  • 5
  • 22

2 Answers2

4

Quoting the Python Tutorial:

To read a file’s contents, call f.read(size), which reads some quantity of data and returns it as a string. size is an optional numeric argument. When size is omitted or negative, the entire contents of the file will be read and returned; it’s your problem if the file is twice as large as your machine’s memory. Otherwise, at most size bytes are read and returned. If the end of the file has been reached, f.read() will return an empty string ("").

It also applies for readlines(), so Python will "throw" the entire file to the memory.

Also, by quoting Python Docs:

file.readlines([sizehint]): Read until EOF using readline() and return a list containing the lines thus read. ...

Christian Tapia
  • 33,620
  • 7
  • 56
  • 73
0

Python is smart enough to check the file size?

Yes. There are some OS related function. You can get file size using them.

What happens if the file has about 1 GB of data? Python will throw the entire file to the memory (array)? Or this is a lazy operation, just like C/C++ does!

If you use fp.readlines, yes 1GB of data will be stored in memory. But unsurprisingly there is a such function which increase file pointer like C/C++. So you can read data chunk by chunk and reduce memory usage.

fp.read(n)

I think this is the C/C++ like file operation you are saying.

Kei Minagawa
  • 4,395
  • 3
  • 25
  • 43