0

I am developing an application which processes (video processing, etc) and sends large files (up to tens of gigabytes) over a network. I am sending the files using FTP. To improve the performance/memory consumption of the application, I would like optimize the buffers so that i dont send packets which are too large and get fragmented. The problem i have is that i dont have much RAM to hold the file data whilst sending. Basically, i read just enough bytes from disk, process it and immediately send to the destination. Currently i am looking to implement MTU path discovery.

I am familiar with the basic concept of how to do it. I would like to know if there is any .NET API in windows which keeps track of the MTU to a destination ?

I am guessing there is no such thing, but a friend of mine told me windows vista keeps track.

I am developing this application for windows XP, but i would like to learn if there is such a network tracking API in windows.

Andrew Keith
  • 7,515
  • 1
  • 25
  • 41

5 Answers5

2

winsock doesn't support reporting the discovered MTU, even though other TCP/IP stacks do (e.g. AIX through a socket option IP_GETPMTU). As winsock cannot report that, .NET can't provide an API (which would have to be on top of winsock).

I recommend sending data in chunks of 64kiB. This is the maximum IP packet size, and likely larger than the MTU, so the stack will send several full segments. The last fragment may be smaller, but then, the system may delay sending it (because it still needs to receive acknowledges for earlier data), so if you follow up quickly with the next send of 64kiB, the system will combine the chunks again into packets using the path mtu.

Martin v. Löwis
  • 124,830
  • 17
  • 198
  • 235
  • Ah nuts, IP_GETPMTU was what i was looking for. Anyway, I will set my app to use chunks of 64kiB. My app is a bit stingy on memory because its actually the second version of the software. The first version was written 10 years ago and is very prudent with memory. So naturally, i am trying to be just as cautious. – Andrew Keith Oct 22 '09 at 08:33
  • What would you possibly do with the return value from IP_GETPMTU? The chunk size that you use to write to the socket has very little to do with what goes out over the wire. I recommend attempt to write as much data as you've got and dealing with partial writes. When you can write, write more. Specifying a magic number like 64kB is silly; how does it make any difference? If it does, what happens when the Window size is greater than 64kB? Of course, with windows, you also have the option of just getting everything ready and doing a big asynchronous write. – janm Oct 22 '09 at 11:19
  • @janm: see the original question. OP can't write "as much data as you've got", as the data won't fit into memory simultaneously. – Martin v. Löwis Oct 22 '09 at 14:29
  • 1
    The "tens of gigabytes" information is new; note that I asked about the scale earlier. Of course you can send as "as much data as you've got"; that doesn't mean "all the data you need to send" it means "as much data as you've got". If you have a buffer with 1MB, attempt to send it. If the stack takes (say) 128kB, great, use select() or its equivalent as required to keep the socket filled. If the problem is "how to I send data bigger than my address space?" the answer is "stream it into the socket in as fast as the socket will take it", not "try to figure out what TCP is doing". – janm Oct 23 '09 at 00:22
0

This is not something you have to do yourself, provided you're using TCP. Also, there is no relation to the size of your send buffers. There can be multiple packets in transit (up to the TCP window size), but they will no longer be in the buffer you maintain.

Thorarin
  • 47,289
  • 11
  • 75
  • 111
  • I should have added more information. I am using FTP. My problem is that i dont have much RAM to hold the data. If i read too much and send too much, i dont get any benefit because the packets are fragmented. I want to only read the exact MTU and send only that amount so that i dont hold much data in RAM. – Andrew Keith Oct 22 '09 at 05:27
0

If you are using UDP, you might care about fragmentation, but if so, why are you implementing your own protocol to transfer large files? Use TCP instead and don't worry about it. If you are implementing your own protocol over UDP with congestion control and all the other things you need, then set the DF bit on the packets and deal with the responses.

If you are using TCP, you don't need to worry about fragmentation at all, the stack does that for you.

janm
  • 17,976
  • 1
  • 43
  • 61
  • I should have added more information. I am using FTP to send the files. My main issue is that i dont have much RAM to use so i have to budget the amount of data i am holding in RAM. – Andrew Keith Oct 22 '09 at 05:25
  • 1
    How much is "not much RAM"? Unless you care about kilobytes (as opposed to megabytes) you are optimising the wrong thing. Using TCP (as you are with FTP) you should just keep the sending socket buffer filled. The TCP/IP stack will deal with path MTU discovery and managing the window size. At the application level you don't even get to see packet boundaries, let along manage them. – janm Oct 22 '09 at 05:54
0

Use FtpWebRequest and FtpWebResponse. The matter is these classes don't use huge buffers but streams (which implemented using best sized buffer). The memory usage will be very minimal.

The packet size is not under your control but under network driver.

If you need best speed then implement data send/receive thru TcpClient class on both server and client sides.

UPD: Here is a sample on how files should be uploaded: http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx See class AsynchronousFtpUpLoader.

One more thing. Some time ago I was experimenting with MTU. All my changes decreased the data transfer speed. The windows network driver is smart enough to use best MTU.

Vasyl Boroviak
  • 5,959
  • 5
  • 51
  • 70
  • My FTP server is so custom, the bloody thing makes FtpWebRequest hang because of bad terminators. I wish i could replace the FTP server with a more standard server, but it runs within some embedded hardware, so i cant touch it. – Andrew Keith Oct 22 '09 at 08:34
  • Well. What can I say... Your life sucks. :) – Vasyl Boroviak Oct 22 '09 at 09:07
0

PMTU Discovery is used by you're TCP stack solely for optimizing packet size when sending. At you're level you only see the stream (ie. before packetization even takes place, let alone fragmentation). What you're problem really sounds like is you are trying to send data as you can, however you're tcp/ip connection is slower, so this is causing the space needed in ram to build up. Since you're using FTP, I would expect the FTP implementation you are using to already support and understand this??? Since you are asking for buffers, it sounds like a roll you're own based on a socket or something like that. If you are using the synchronous send methods, when the socket buffer is full, it should block you're sends until it has buffer space.

Can you maybe give more details on what API's you're using to send the file, Builtin vs. roll you're own? Whether it be socket, networkstream, etc?

Kevin Nisbet
  • 2,239
  • 16
  • 14