Stack Overflow
Stack Overflow

Questions tagged [sparse-file]

A sparse file is a file where only the non-zero data are stored in data blocks and all zero-data are generated by the filesystem when the file is read. As such, the sparse file uses disk space efficiently by storing only metadata about the zeros, rather than storing the zeros physically on storage. When the sparse file is later read, the filesystem generates the zeros dynamically.

62 questions
33
votes
2 answers

What is a sparse file and why do we need it?

What is a sparse file and why do we need it? The only thing that I am able to get is that it is a very large file and it is efficient(in gigabytes). How is it efficient ?
Luv33preet
  • 1,686
  • 7
  • 33
  • 66
23
votes
5 answers

Copying a 1TB sparse file

I got a sparse file of 1TB which stores actually 32MB data on Linux. Is it possible to "efficiently" make a package to store the sparse file? The package should be unpacked to be a 1TB sparse file on another computer. Ideally, the "package" should…
ericzma
  • 763
  • 3
  • 9
  • 23
17
votes
5 answers

How to make file sparse?

If I have a big file containing many zeros, how can i efficiently make it a sparse file? Is the only possibility to read the whole file (including all zeroes, which may patrially be stored sparse) and to rewrite it to a new file using seek to skip…
rurouni
  • 2,315
  • 1
  • 19
  • 27
14
votes
2 answers

How to create a sparse file on NTFS?

I'm testing a sparse file. But my test code doesn't work well. HANDLE h = CreateFileW(L"D:\\sparse.test", GENERIC_READ|GENERIC_WRITE, FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, 0, CREATE_ALWAYS, …
Benjamin
  • 10,085
  • 19
  • 80
  • 130
14
votes
7 answers

How does one reclaim zeroed blocks of a sparse file?

Consider a sparse file with 1s written to a portion of the file. I want to reclaim the actual space on disk for these 1s as I no longer need that portion of the sparse file. The portion of the file containing these 1s should become a "hole" as it…
z8000
  • 3,715
  • 3
  • 29
  • 37
13
votes
1 answer

sparse file usage in python

I'm creating sparse files in python as follows: >>> f = open('testfile', 'ab') >>> f.truncate(1024000) >>> f.close() when the file is done, it takes up 0 disk space, but its inode size is set to my truncated value (1000K): igor47@piglet:~/test$ ls…
Igor Serebryany
  • 3,307
  • 3
  • 29
  • 41
8
votes
2 answers

How do you programmatically create a completely empty sparse file on linux?

If you run dd with this: dd if=/dev/zero of=sparsefile bs=1 count=0 seek=1048576 You appear to get a completely unallocated sparse file (this is ext4) smark@we:/sp$ ls -ls sparsefile 0 -rw-rw-r-- 1 smark smark 1048576 Nov 24 16:19…
stu
  • 8,461
  • 18
  • 74
  • 112
8
votes
4 answers

How to create a 10MB file filled with "000000..." data in golang?

I am intended to use fdatasync in a system like log or diskqueue. The first thing is to create a 10MB file with "000000..." in file system like ext4. But I don't know how to do it properly.
hardPass
  • 19,033
  • 19
  • 40
  • 42
7
votes
2 answers

How to test if sparse file is supported

Given a file descriptor or file name, how do I know if I can write to an arbitrary location without waiting for the intervening part be explicitly zeroed out on disk?
Siyuan Ren
  • 7,573
  • 6
  • 47
  • 61
5
votes
1 answer

Sparse files: How to find contents

If I create a file, use lseek(2) to jump to a high position in the (empty) file, then write some valuable information there, I create a sparse file on Unix system (probably depending on the file system I use, but let's assume I'm using a typical…
Alfe
  • 56,346
  • 20
  • 107
  • 159
5
votes
2 answers

Sparse files are huge with io.Copy()

I want to copy files from one place to another and the problem is I deal with a lot of sparse files. Is there any (easy) way of copying sparse files without becoming huge at the destination? My basic code: out, err := os.Create(bricks[0] + "/" +…
Alex
  • 4,674
  • 5
  • 38
  • 59
4
votes
0 answers

Performance penalty for sparse files?

Sparse files can significantly reduce the storage requirements for a file that has large "empty" portions. But does the increased bookkeeping for these sparse regions materially impact access performance? This will, of course, depend on the file…
Tony the Pony
  • 40,327
  • 71
  • 187
  • 281
4
votes
1 answer

Why LMDB database taking more than actual data size?

I put around 11K key&values in LMDB database . LMDB database file size become 21Mb. For the same data the leveldb is taking 8Mb only (with snappy compression). LMDB env info…
Rajesh Gopu
  • 863
  • 9
  • 33
4
votes
2 answers

How to use ioctl with FS_IOC_FIEMAP

My problem is to deal with sparse file reads and understand where the extents of the file are to perform some logic around it. Since, there is no direct API call to figure these stuff out, I decided to use ioctl api to do this. I got the idea from…
Aila
  • 319
  • 1
  • 4
  • 11
3
votes
2 answers

Is there a method of quickly determining whether a chunk read from a (sparse) file is all zeros?

Is there a method of quickly determining whether a (4KB-16MB) chunk read from a file is all zeros? You can iterate over the chunk, checking each byte. There are obvious optimisations, but it remains O(N). My use case is for sparse files. I would…
fadedbee
  • 42,671
  • 44
  • 178
  • 308
1
2 3 4 5