8

If you run dd with this:

dd if=/dev/zero of=sparsefile bs=1 count=0 seek=1048576

You appear to get a completely unallocated sparse file (this is ext4)

smark@we:/sp$ ls -ls sparsefile 
0 -rw-rw-r-- 1 smark smark 1048576 Nov 24 16:19 sparsefile

fibmap agrees:

smark@we:/sp$ sudo hdparm --fibmap sparsefile 
sparsefile:
 filesystem blocksize 4096, begins at LBA 2048; assuming 512 byte sectors.
 byte_offset  begin_LBA    end_LBA    sectors

Without having to dig through the source of dd, I'm trying to figure out how to do that in C.

I tried fseeking and fwriting zero bytes, but it did nothing. Not sure what else to try, I figured somebody might know before I hunt down dd's innards.

EDIT: including my example...

FILE *f = fopen("/sp/sparse2", "wb");
fseek(f, 1048576, SEEK_CUR);
fwrite("x", 1, 0, f);
fclose(f);
stu
  • 8,461
  • 18
  • 74
  • 112
  • 1
    `dd` is open source. grab the source and look through it... – Marc B Nov 24 '15 at 21:25
  • you must have missed the part about "without having to dig through the source of dd". I figured somebody might just know. – stu Nov 24 '15 at 21:25
  • Maybe I don't understand the question but are you looking for `touch `? –  Nov 24 '15 at 21:26
  • @jshaw.fb: that just creates a 0byte file. sparse files occpuy as much space as you want, without actually using up disk space. it's like putting "reserved for..." on the disk blocks without actually using them. – Marc B Nov 24 '15 at 21:27
  • 1
    Normally you'd use `lseek` to a point beyond the current end of file, then write something. I'd be surprised if `fseek` didn't use `lseek`. Can you show the code that you're using? – Mark Plotnick Nov 24 '15 at 21:28
  • 9
    From running `strace`, I see that `dd` adds a call to `ftruncate`. That'll create a sparse file even if you don't do any `write`s. – Mark Plotnick Nov 24 '15 at 21:32
  • ahhhh.. good idea, I didn't think to try strace, and yes, ftruncate will do it. make it an answer and I'll accept, thanks. – stu Nov 24 '15 at 21:33
  • Your code tells fwrite to write zero bytes, so it doesn't write anything. Change the 0 to 1 and it'll work. – Mark Plotnick Nov 24 '15 at 21:35
  • if I change it to 1 it writes an entire block, that's what I'm trying to avoid. – stu Nov 24 '15 at 21:40
  • use which dd or whereis dd to find dd. strings /usr/bin/dd |sort -u |less, shows various function calls. and man will explain the functions used. – ChuckCottrill Nov 24 '15 at 23:37
  • Possible duplicate of [How to create a file with file holes?](https://stackoverflow.com/questions/5315428/how-to-create-a-file-with-file-holes) – Ciro Santilli OurBigBook.com Jun 05 '17 at 16:28

2 Answers2

10

When you write to a file using write or various library routines that ultimately call write, there's a file offset pointer associated with the file descriptor that determines where in the file the bytes will go. It's normally positioned at the end of the data that was processed by the most recent call to read or write. But you can use lseek to position the pointer anywhere within the file, and even beyond the current end of the file. When you write data at a point beyond the current EOF, the area that was skipped is conceptually filled with zeroes. Many systems will optimize things so that any whole filesystem blocks in that skipped area simply aren't allocated, producing a sparse file. Attempts to read such blocks will succeed, returning zeroes.

Writing block-sized areas full of zeroes to a file generally won't produce a sparse file, although it's possible for some filesystems to do this.

Another way to produce a sparse file, used by GNU dd, is to call ftruncate. The documentation says this:

The ftruncate() function causes the regular file referenced by fildes to have a size of length bytes.

If the file previously was larger than length, the extra data is discarded. If it was previously shorter than length, it is unspecified whether the file is changed or its size increased. If the file is extended, the extended area appears as if it were zero-filled.

Support for sparse files is filesystem-specific, although virtually all designed-for-UNIX local filesystems support them.

Community
  • 1
  • 1
Mark Plotnick
  • 9,598
  • 1
  • 24
  • 40
  • You just beat me to it! – Iharob Al Asimi Nov 24 '15 at 21:59
  • zfs does that by the way, writing chunks of contiguous zeroes to a file (if any compression scheme is on) will basically hole punch that area back into sparseness. – stu Nov 25 '15 at 15:04
  • @stu Thanks for that - I was hoping that people with expertise in various filesystem types would comment. I'll add the additional info to my answer after a few days. – Mark Plotnick Nov 25 '15 at 15:24
9

This is complementary to the answer by @MarkPlotnick, it's a sample simple implementation of the feature you requested using ftruncate():

#include <unistd.h>
#include <fcntl.h>

#include <sys/stat.h>

int
main(void)
{
    int file;
    int mode;

    mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH;
    file = open("sparsefile", O_WRONLY | O_CREAT, mode);
    if (file == -1)
        return -1;
    ftruncate(file, 0x100000);
    close(file);

    return 0;
}
Community
  • 1
  • 1
Iharob Al Asimi
  • 52,653
  • 6
  • 59
  • 97