4

Quoted from MSDN entry for TransmitFile:

The maximum number of bytes that can be transmitted using a single call to the TransmitFile function is 2,147,483,646, the maximum value for a 32-bit integer minus 1. The maximum number of bytes to send in a single call includes any data sent before or after the file data pointed to by the lpTransmitBuffers parameter plus the value specified in the nNumberOfBytesToWrite parameter for the length of file data to send. If an application needs to transmit a file larger than 2,147,483,646 bytes, then multiple calls to the TransmitFile function can be used with each call transferring no more than 2,147,483,646 bytes. Setting the nNumberOfBytesToWrite parameter to zero for a file larger than 2,147,483,646 bytes will also fail since in this case the TransmitFile function will use the size of the file as the value for the number of bytes to transmit.

Alright. Sending a file of size 2*2,147,483,646 bytes (~ 4 GiB) with TransmitFile would then have to be divided into two parts at minimum (e.g. 2 GiB + 2 GiB in two calls to TransmitFile). But how exactly would one go about doing that, while preferably also keeping the underlying TCP connection alive in between?

When the file is indeed <=2,147,483,646 bytes in size, one could just write:

HANDLE fh = CreateFile(filename, GENERIC_READ, FILE_SHARE_READ, NULL, 
 OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, NULL);   

TransmitFile(SOCK_STREAM_socket, fh, 0, 0, NULL, NULL, TF_DISCONNECT);

to let Windows handle all the lower-level stuff (caching, chunking the data up into pieces for efficient transmission etc. However, unlike the comparable Linux sendfile() syscall, there is no immediately obvious offset argument in the call (although the fifth argument, LPOVERLAPPED lpOverlapped probably is exactly what I'm looking for). I suppose I could hack something together, but I'm also looking for a graceful, good practice Win32 solution from someone who actually knows about this stuff.

You can use the lpOverlapped parameter to specify a 64-bit offset within the file at which to start the file data transfer by setting the Offset and OffsetHigh member of the OVERLAPPED structure. If lpOverlapped is a NULL pointer, the transmission of data always starts at the current byte offset in the file.

So, for lack of a minimal example readily available on the net, which calls are necessary to accomplish such a task?

elmov
  • 286
  • 1
  • 11
  • 1
    If you don't pass the `TF_DISCONNECT` flag presumably you can make another call to `TransmitFile()` to send the next 2GB, and so on. The `lpOverlapped` parameter is indeed how you provide the file offset. – Jonathan Potter Aug 14 '13 at 20:51
  • It simply reads the file at its current position. – Hans Passant Aug 14 '13 at 20:59
  • 2
    Have you tried the `Socket.SendFile()`? MSDN doesn't say anything about the file size, therefore, I assume (though never try it) it'll handle that circumstance. `socket.sendFile` uses the TransmitFile function to do the job. Also take a look at `SetFilePointerEx` to position the file before each call to `TransmitFile`. – ja_mesa Aug 14 '13 at 21:55
  • The _full_ description of lpOverlapped combined with the description of the return value seems to make it pretty clear how to proceed. As for best practices, most programmers who use functions like that on Windows are writing proprietary code, so don't publish. If you can't find good examples, then you're probably on your own. – Carey Gregory Aug 15 '13 at 04:30
  • Hmm. Thanks for the pointers. Weirdly enough, only the first call to `TransmitFile` seems to be actually incrementing the file pointer (retrieved by a call to `SetFilePointer(fh, 0, NULL, FILE_CURRENT)`), and by a seemingly unrelated amount too (32k even though I'm explicitly specifying 1024*1024 as `nNumberOfBytesToWrite`). According to [MSDN](http://msdn.microsoft.com/en-us/library/windows/desktop/aa364397(v=vs.85).aspx), each read and write operation should advance the file pointer by the number of bytes being read or written. Any subseq. calls are successful, but not incrementing the fp. – elmov Aug 15 '13 at 09:20
  • ^ to me, just positioning the file pointer manually after every call sounds like a bad idea (although it _does_ work, at least for my limited test cases). I'd prefer to do it in a way that avoids any of this (or the `lpOverLapped` method, for that matter). Could there be something wrong with the way I'm calling `CreateFile`? A misplaced flag perhaps? – elmov Aug 15 '13 at 09:34
  • (- and yeah, removing `TF_DISCONNECT` got rid of the TCP disconnecting problem, duh :D) – elmov Aug 15 '13 at 09:40

1 Answers1

3

Managed to figure it out based on the comments.

So, if LPOVERLAPPED lpOverlapped is a null pointer, the call starts transmission at the current file offset of the file (much like the Linux sendfile() syscall and its off_t *offset parameter). This offset (pointer) can be manipulated with SetFilePointerEx, so one could write:

#define TRANSMITFILE_MAX ((2<<30) - 1)

LARGE_INTEGER total_bytes;
memset(&total_bytes, 0, sizeof(total_bytes));

while (total_bytes < filesize) {
     DWORD bytes = MIN(filesize-total_bytes, TRANSMITFILE_MAX);
     if (!TransmitFile(SOCK_STREAM_socket, fh, bytes, 0, NULL, NULL, 0))
     { /* error handling */ }

     total_bytes.HighPart += bytes;
     SetFilePointerEx(fh, total_bytes, NULL, FILE_BEGIN);
}

closesocket(SOCK_STREAM_socket);

to accomplish the task. Not very elegant imo, but it works.

elmov
  • 286
  • 1
  • 11