10

I want to copy data from one stream to another. Now normally, I would do it this way:

n = fread(buffer, 1, bufsize, fin);
fwrite(buffer, 1, n, fout);

Is there a way to write the data directly from fin to fout, without going through a buffer, i.e. instead of fin->buffer->fout, I want to directly do fin->fout (no buffer).

Is it possible to do so in ANSI C? If not, is it possible to do it with POSIX functions? Or a Linux-specific solution?

sashoalm
  • 75,001
  • 122
  • 434
  • 781
Robby75
  • 3,285
  • 6
  • 33
  • 52
  • I don't think so, because AFAIK freopen() can only mangle either input or output streams, but seems not to be able to connect an input to an output. – Robby75 Jan 09 '14 at 08:18
  • It's Linux and /output will actually be a named pipe - but I don't know how that solves the problem. – Robby75 Jan 09 '14 at 08:27
  • Some process writes data to the stdin of my program. The job of my program is to set up a named pipe and put that data into the named pipe. – Robby75 Jan 09 '14 at 08:37
  • "Some process" does not know the name of the named pipe, the job of my program is to find a free name, set up the pipe, etc. – Robby75 Jan 09 '14 at 08:49
  • In fact my program does a little bit more than that - it also decides whether a pipe is used or another interface used. It's certainly not redundant ;-) – Robby75 Jan 09 '14 at 08:51
  • I prefer the old-school file-descriptors instead of FILE*, but I guess that's OK, so I'm fine with your edit. BTW, I looked up the source-code of "cat" and it also does it with a buffer - so I wonder whether it's even possible. – Robby75 Jan 09 '14 at 09:18
  • No, your question is good, really, I've wondered it, too. I think it's just not worth it, for I/O with harddisk certainly, but even with pipes, the overhead is probably negligible. I think if you really want to avoid copying, shared memory is the way to go. I deleted all my comments btw, so they don't clutter. – sashoalm Jan 09 '14 at 09:23
  • As for `cat` being implemented that way, you might want to look at http://stackoverflow.com/questions/15769970/linux-2-6-33-could-sendfile-be-used-to-implement-a-faster-cat – sashoalm Jan 09 '14 at 10:13

1 Answers1

9

2 possible Linux-only solutions are splice() and sendfile(). What they do is copy data without it ever leaving kernel space, thus making a potentially significant performance optimization.

Note that both have limitations:

  • sendfile() requires a socket for its output for Linux kernels before 2.6.33, after that, any file can be the output, and also it requires the input to support mmap() operations, meaning the input can't be stdin or a pipe.

  • splice() requires one of the input or output streams to be a pipe (not sure about both), and also for kernel versions 2.6.30.10 and older, it requires the file system for the stream that is not a pipe to support splicing.

Edit: Note that some filesystems might not support splicing for Linux 2.6.30.10 and below.

Community
  • 1
  • 1
sashoalm
  • 75,001
  • 122
  • 434
  • 781
  • sendfile() sounded promising, but I always get EINVAL, probably because according to the manpage: "an mmap(2)-like operation is not available for in_fd" - which sounds plausible for stdin. – Robby75 Jan 09 '14 at 14:12
  • splice() works on my desktop (Linux 3.11), but doesn't work on my actual target system, probably because of vhost-limitations (also gettting EINVAL, probably because according to the manpage: "Target filesystem doesn't support splicing". – Robby75 Jan 09 '14 at 14:23
  • I'll accept your answer, but you should remove sendfile() from it. – Robby75 Jan 09 '14 at 14:25
  • You can look here http://stackoverflow.com/questions/3638657/which-file-systems-support-splicing-via-linuxs-splice2 – sashoalm Jan 09 '14 at 14:25
  • OK, that clears it up, my cheap vhosted Linux 2.6 is not good enough for splice()... Thanks a lot for your support though. – Robby75 Jan 09 '14 at 14:28
  • I rewrote my answer to highlight the limitations for both functions. I've left `sendfile()`, because it might work in some cases where `splice()` does not, since it doesn't require a pipe. – sashoalm Jan 09 '14 at 14:58