8

I have a program that attempts to write quite a large amount of text to a file on a remote server overseas, which has a slow network connection.

Using the following code, where outputFileContent is a StringBuilder:

using (var outfile = new StreamWriter(myRemoteFilePath))
{
    outfile.Write(outputFileContent.ToString());
}

it is taking a seriously long time to run (several minutes), whereas if I first write to a local file and then copy it across to the remote location, it is much quicker (20-30 secs):

string tempFilePath = Path.GetTempFileName();
using (var outfile = new StreamWriter(tempFilePath))
{
    outfile.Write(outputFileContent.ToString());
}

System.IO.File.Copy(tempFilePath, myRemoteFilePath, true)

Any idea why this is happening? My only guess is that it is something to do with buffering across the network, or perhaps because the stream writer doesn't know how big it needs to be ahead of time.

Soner Gönül
  • 97,193
  • 102
  • 206
  • 364
Stephen Holt
  • 2,360
  • 4
  • 26
  • 34
  • 2
    If you watch the network, e.g. with WireShark, you may find that the stream writes waits for an acknowledgement of each write while the copy is content to have a substantial amount of data in flight without being acknowledged. – HABO Jan 29 '13 at 16:32
  • You can try changing the buffer size using another StreamWriter constructor. Try a high value like 0x8000. – Guillaume Jan 29 '13 at 16:48

2 Answers2

16

If you create your StreamWriter using default buffer sizes then the underlying SMB protocol will be issuing write requests in chunks no larger than 4096 bytes, which means a large number of round-trips over the network. You could increase your StreamWriter's buffer size up to a maximum of 64k in order to reduce the number of round trips:

using (var outfile = new StreamWriter(myRemoteFilePath, false, Encoding.ASCII, 0x10000))

Increasing the buffer size beyond 64k will not help in any circumstance, as the underlying SMB protocol does not support buffer lengths beyond 64k. Note that a direct file copy still uses the SMB protocol, so from a network traffic perspective there's little difference between the operations except for the buffer sizes.

RogerN
  • 3,761
  • 11
  • 18
  • OK I'll give that a go. Is there a downside to setting a high buffer if in fact the file turns out to be quite small? – Stephen Holt Jan 29 '13 at 17:43
  • The only downside I'm aware of is that your application will use a little more RAM than it needs. I wouldn't worry about it, but if it bothers you then make the buffer size dynamic based on the content length. – RogerN Jan 29 '13 at 17:50
  • Packets in the network will be no bigger than the MTU, which is < 1500 bytes. This answer also doesn't explain the difference the OP observed compared to the file copy. – user207421 Jan 31 '13 at 05:11
  • 2
    The MTU won't have nearly as large an effect on total transfer time as the SMB protocol's buffer length, because packet fragmentation via the MTU does not result in additional round-trip communication in the form of SMB ACKs. I would argue that this answer *does* explain the difference compared to a standard file copy, and indeed in my own tests if I just increase the StreamWriter's buffer size then performance is identical to a standard file copy operation. – RogerN Jan 31 '13 at 13:46
  • I can confirm this speed increase (2 Mbps to 25 Mbps) with `FileStream.CopyToAsync`, for which I had set the buffer size to 4096 [as per this answer](https://stackoverflow.com/a/36925751/247702), so the answer is correct. It's a little late, but /cc @user207421 in case you still care :) – user247702 Jan 11 '19 at 15:05
-2

I'm not a security expert, but in my opinion, this is most related to permission issue.

Everytime you write somethin to a disk on remote server the permission is going to be checked against the user that execution that action. The permission to write, naturally.

In case instead of the copy, that control is executed only ones. So it's much faster.

In fact, to prove this you can try to copy several files over the network and after copy ZIP files (with the same amount of memory in zipped state) over network. In second case it will be much faster, as permission control is executed only once, for single ZIP file.

Hope this helps.

Tigran
  • 61,654
  • 8
  • 86
  • 123
  • 2
    The permission should be checked when the stream is created, but subsequent writes to an open stream shouldn't require reauthorization. The overhead of checking permission should be the same for copying a single file. – HABO Jan 29 '13 at 18:11