53

I want to create a file of particular size (say, 1GiB). The content is not important since I will fill stuff into it.

What I am doing is:

f = open("E:\\sample", "wb")
size = 1073741824 # bytes in 1 GiB
f.write("\0" * size)

But this takes too long to finish. It spends me roughly 1 minute. What can be done to improve this?

glglgl
  • 89,107
  • 13
  • 149
  • 217
onemach
  • 4,265
  • 6
  • 34
  • 52
  • 1 min sounds about as good as it gets to me. BTW, what exactly are you doing with this file? – David Robinson Jan 11 '12 at 08:14
  • 5
    1073741824 bytes != 1GB. Use an SSD instead of a mechanical HDD? Write to local disk rather than a network share? – johnsyweb Jan 11 '12 at 08:17
  • 4
    @onemach 1 GB = 10^9 B. 1 GiB = 2^30 B. – glglgl Jan 11 '12 at 09:38
  • 4
    [xkcd - Kilobyte](http://xkcd.com/394/) – Imran Jan 11 '12 at 16:47
  • 5
    @glglgl: Given how few people use gibibyte as a term, possibly due to how ridiculous it sounds (including many file system displays that summarize using base-2 GB instead of base-10 GB), getting pedantic about GiB is just that: Being pedantic. Hard drive manufacturers use base-10 to advertise larger sizes; almost no one else does. – ShadowRanger Oct 30 '15 at 17:14
  • @ShadowRanger Some call it pedantic, other call it concise. Maybe it sounds ridiculous, maybe we are just not used to it yet. – glglgl Oct 30 '15 at 20:00
  • It seems that your code is fine, it could not be done in a better way, IMHO. The bottleneck is your hard drive speed, with some profiling you can figure out that creating the 1 GiB string in memory is quite fast (~1 sec.) – Romain Jun 27 '19 at 19:56

2 Answers2

55

WARNING This solution gives the result that you might not expect. See UPD ...

1 Create new file.

2 seek to size-1 byte.

3 write 1 byte.

4 profit :)

f = open('newfile',"wb")
f.seek(1073741824-1)
f.write(b"\0")
f.close()
import os
os.stat("newfile").st_size

1073741824

UPD: Seek and truncate both create sparse files on my system (Linux + ReiserFS). They have size as needed but don't consume space on storage device in fact. So this can not be proper solution for fast space allocation. I have just created 100Gib file having only 25Gib free and still have 25Gib free in result.

Minor Update: Added b prefix to f.write("\0") for Py3 compatibility.

Community
  • 1
  • 1
Shamanu4
  • 5,296
  • 2
  • 27
  • 31
  • If your system supports it, the `truncate()` solution might - strictly seen - be better, as this solution here consumes 1 allocation block for this file although it would not be necessary. – glglgl Jan 11 '12 at 13:50
  • 2
    Doesn't that generate a sparse file, though? (This may be what was needed, but it's not quite the same) – Arafangion Jan 11 '12 at 13:51
  • 7
    In python 3 it's `f.write(b"\0")` – qed May 30 '14 at 14:32
  • Works on MacOS High Sierra from a file size enumeration point of view, even though it doesn't affect the overall system size. Great for writing tests for large files, which is exactly what I'm doing :) – brandonscript Feb 08 '18 at 16:49
45

The question has been answered before. Not sure whether the solution is cross platform, but it works in Windows (NTFS file system) flawlessly.

with open("file.to.create", "wb") as out:
    out.truncate(1024 * 1024 * 1024)

This answer uses seek and write:

with open("file.to.create", "wb") as out:
    out.seek((1024 * 1024 * 1024) - 1)
    out.write('\0')
Community
  • 1
  • 1
Imran
  • 87,203
  • 23
  • 98
  • 131
  • 8
    [Docs:](http://docs.python.org/library/stdtypes.html#file-objects) _Note that if a specified size exceeds the file’s current size, the result is platform-dependent: possibilities include that the file may remain unchanged, increase to the specified size as if zero-filled, or increase to the specified size with undefined new content._ – Janne Karila Jan 11 '12 at 08:38
  • @JanneKarila In this case `file.to.create` is an empty file, so the specified size will always exceed the file's current size? – qed May 30 '14 at 14:34
  • [New docs link](https://docs.python.org/3/library/io.html#io.IOBase.truncate): _This resizing can extend or reduce the current file size. In case of extension, the contents of the new file area depend on the platform (on most systems, additional bytes are zero-filled). The new file size is returned._ – Sreenikethan I Sep 03 '23 at 03:09