0

I'm using Python 3 to download a file:

local_file = open(file_name, "w" + file_mode)
local_file.write(f.read())
local_file.close()

This code works, but it copies the whole file into memory first. This is a problem with very big files because my program becomes memory hungry. (Going from 17M memory to 240M memory for a 200 MB file)

I would like to know if there is a way in Python to download a small part of a file (packet), write it to file, erase it from memory, and keep repeating the process until the file is completely downloaded.

random
  • 9,774
  • 10
  • 66
  • 83
Avovk
  • 527
  • 1
  • 5
  • 15
  • 2
    possible duplicate of [Lazy Method for Reading Big File in Python?](http://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python) – chown Oct 30 '11 at 19:19

1 Answers1

3

Try using the method described here:

Lazy Method for Reading Big File in Python?

I am specifically referring to the accepted answer. Let me also copy it here to ensure complete clarity of response.

    def read_in_chunks(file_object, chunk_size=1024):
        """Lazy function (generator) to read a file piece by piece.
        Default chunk size: 1k."""
        while True:
            data = file_object.read(chunk_size)
            if not data:
                break
            yield data


    f = open('really_big_file.dat')
    for piece in read_in_chunks(f):
        process_data(piece)

This will likely be adaptable to your needs: it reads the file in smaller chunks, allowing for processing without filling your entire memory. Come back if you have any further questions.

Community
  • 1
  • 1
Brendon
  • 879
  • 1
  • 6
  • 18
  • What method are you even referring to? – random Oct 30 '11 at 19:34
  • The one that I linked to, which I will link here again: [link](http://stackoverflow.com/questions/519633/lazy-method-for-reading-big-file-in-python) – Brendon Oct 30 '11 at 19:37
  • You just linked to the question. Which of the answers is the method you're suggesting? – random Oct 30 '11 at 19:42
  • 1
    I assumed it was clear that I was referring to the accepted answer. I will edit my post to clarify further. – Brendon Oct 30 '11 at 19:45
  • 4
    The accepted answer isn't atomic and can change after you point vaguely in its direction – random Oct 30 '11 at 19:47