10

Possible Duplicate:
Quickly create large file on a windows system?

For testing purposes, I would like to be able to create and delete very large files (several GB's). I have no need to write anything specific to them. They could be random data. Is there a way to generate a large file by simply allocating the space on disk? If not, what is the quickest way to write such a file? To create the file in a matter of a few seconds is desirable.

I need to do this within a Python script and am doing so on Windows 7.

Community
  • 1
  • 1
Paul
  • 2,973
  • 6
  • 31
  • 40

1 Answers1

23

This should work on ntfs filesystems, since they support sparse files. It's almost instantaneous.

with open("file.to.create", "w") as file:
    file.truncate(10 ** 10)

The file will appearantly be filled with \x00 bytes, but those are in fact just created as needed when you read from the file. It uses almost no disk space (though it may look like it uses all 10 GB from the very beginning -- I've found no easy way to check the real file size in windows), and grows by allocating needed blocks when you write to it. AFAIK it's quite possible to create a sparse file that is much larger than the disk it resides on, though this may of course lead to trouble later on. :)

Beware: if you copy a sparse file, it may in the process expand to a non-sparse file (read "fake" \x00 bytes, write real \x00 bytes). This is a consequence of the fact that it looks just like an ordinary 10 GB file with null bytes in order to be "backward compatible" -- separate checks must be performed to reveal it as a sparse file. To successfully copy a sparse file and keep it a sparse file, two conditions must be met:

  • the tool used to copy it must be "aware" of sparse files, and
  • the filesystem it is copied to must support sparse files

For example, USB thumbdrives/pens are typically formatted with the old FAT filesystem by default, and it does not support sparse files. From testing, explorer of Windows XP appears not preserve sparse files when copying. This tip suggests Robocopy is up to the job, but I've not tested it.

Lauritz V. Thaulow
  • 49,139
  • 12
  • 73
  • 92
  • 1
    +1 for this very interesting tips. – Bite code Jun 27 '11 at 19:28
  • 2
    this seems to take a loooong time for me!!! – Shihab Shahriar Khan Mar 04 '17 at 18:27
  • Right now this solution does create a large file, but it's not a sparse file. Windows (via NTFS and ReFS) requires a special flag (`FSCTL_SET_SPARSE`) to allow a file to be created as sparse. At the moment, `file.truncate` does not set that flag. https://github.com/python/cpython/issues/84091 – tylerl Jul 18 '23 at 18:39