2

is there any system call that can copy data from a socket to a file? I looked into sendfile system call and I see that the input_fd has to be a file descriptor so I was wondering if there is any system call an application can leverage to do zero copy when receiving the data from a socket and storing it in a file?

user1870400
  • 6,028
  • 13
  • 54
  • 115

1 Answers1

2

(This looks like a duplicate of Understanding sendfile() and splice()) This question asker here wants to know if data read from a socket can be zero-copied to a file and the mention of io_uring strongly suggests the asker is specifically interested in Linux.

In short yes, it is possible to receive from a socket and output to a file without having to make unnecessary duplicate copies by using splice(2) on Linux but it's not trivial - the socket must be attached to one end of the pipe and the file's descriptor to the other end. Since the 5.7 Linux kernel io_uring also supports a splice operation so it too can do zero copy from a socket to a file via a pipe.

Anon
  • 6,306
  • 2
  • 38
  • 56
  • I am looking to see if there exists a system call that can do a zero copy from a socket to a local file? so there are two descriptors in my question 1) a socket descriptor 2) a file descriptor. To give a more concrete example, Imagine a web server that receives messages at particular socket and it needs to store them to a local file. can the web server do a zero copy? – user1870400 Oct 24 '20 at 09:39
  • It requires one the file descriptor to be a pipe. – user1870400 Oct 26 '20 at 18:31
  • yes sendfile cannot have socket descriptor as input_fd(input file descriptor). pipe takes two file descriptors as parameters (input_fd, output_fd) and one of them has to be pipe. In my case I want the input_fd to be a socket. so the question is can a pipe be the socket? and can it work as expected? I don't know the answer to that. That's why I posted my question here – user1870400 Oct 26 '20 at 19:15
  • can you explain how socket can be a pipe? an example may be along with how to do it using splice? – user1870400 Oct 27 '20 at 05:57
  • yeah but there must be a reason why sendfile must have restricted it. so I want to understand those – user1870400 Oct 27 '20 at 06:02
  • From a historic perspective `sendfile` was first. On Linux it was designed to optimise getting pages out of a populated page cache efficiently but is not generic hence the restrictions (this is alluded to in this [LKML sendfile thread](https://yarchive.net/comp/linux/sendfile.html)). `splice` came later and is more flexible (but more complicated) and was generic enough to entirely support `sendfile`. If you [follow the links in this answer](https://stackoverflow.com/a/8626372/2732969) on another question you will see an example of how you can read socket/pipe/write file with `splice`. – Anon Oct 27 '20 at 06:24
  • got it. do you know what is packet_mmap? online says it is a syscall but I cannot see it on my ubuntu and there is no man page either. – user1870400 Oct 27 '20 at 06:27
  • I think we might be getting off-topic but a quick google quickly led to https://stackoverflow.com/q/43193889/2732969 where the top answer even says "Many interfaces to the linux kernel are not well documented" :-). It looks like you're going down rabbit hole (so I hope you enjoy the journey :-) but so I'll bail out and leave you with a link to https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-future/ to distract you further... – Anon Oct 27 '20 at 06:34
  • yes. thanks for pointing out all the useful resources and your commentary(besides the sarcastic ones) does help. – user1870400 Oct 27 '20 at 06:58