1

Lets say I know that a file descriptor fd is open for reading in my process. I would like to pipe data from this fd into a fifo that is available for reading outside my of process, in a way that avoids calling poll or select on fd and manually reading/forwarding data. Can this be done?

iggy
  • 1,613
  • 1
  • 18
  • 35

1 Answers1

2

You mean ask the OS to do that behind the scenes on an ongoing basis from now on? Like an I/O redirection?

No, you can't do that.

You could spawn a thread that does nothing but read the file fd and write the pipe fd, though. To avoid the overhead of copying memory around in read(2) and write(2) system calls, you can use sendfile(out_fd, in_fd, NULL, 4096) to tell the kernel to copy a page from in_fd to out_fd. See the man page.

You might have better results with splice(2), since it's designed for use with files and pipes. sendfile(2) used to require out_fd to be a socket. (Designed for zero-copy sending static data on TCP sockets, e.g. from a web server.)

Linux does have asynchronous I/O, so you can queue up a read or write to happen in the background. That's not a good choice here, because you can't queue up a copy from one fd to another. (no async splice(2) or sendfile(2)). Even if there was, it would have a specific request size, not a fire-and-forget keep copying-forever. AFAIK, threads have become the preferred way to do async I/O, rather than the POSIX AIO facilities.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
  • thanks for the explanation, i guess `splice` in a separate thread is a good choice. `sendfile` has a limit on the number of bytes it can send. – iggy Nov 19 '15 at 14:05
  • @iggy: splice and sendfile both have a `size_t` length argument. IDK if it's better to call them with a small-ish size like 64kiB, or if it's good to use `SIZE_MAX` so your user-space loop only happens if/when the system call is interrupted (by a signal, for example). Actually transfering 2^64-1 bytes is unlikely to say the least, but you shouldn't assume that the call will never return. – Peter Cordes Nov 19 '15 at 14:15
  • quoting doc for sendfile `sendfile() will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on both 32-bit and 64-bit systems.)`, but thanks for suggesting splice, that would definitely work – iggy Nov 19 '15 at 14:28
  • @iggy: Which doc did you find that in? It's not in the current man page (Linux man-pages 3.74, sendfile man page last edited 2011-09-14). So IDK if it's an old limitation that's been removed, or if the documentation was previously incomplete. – Peter Cordes Nov 19 '15 at 14:40
  • you are right, this is probably not the most up-to date resource http://man7.org/linux/man-pages/man2/sendfile.2.html – iggy Nov 19 '15 at 14:56
  • @iggy: Look at the bottom. It is *more* up to date than what Ubuntu ships. So it's probably still a true fact that just wasn't in the man page before. – Peter Cordes Nov 19 '15 at 14:57