How do I download a file in vb .net with a download timeout?

Question

I want to download thousands of files from the web and save them locally. What is the most efficient way? It is important the failures timeout within 10 seconds.

Is there a better way to stream one stream into another? Maybe a smaller buffer, like 1024 bytes at a time, is more efficient for large files?

Dim w_req As System.Net.HttpWebRequest = CType(System.Net.HttpWebRequest.Create("http://blah.blah.blah/blah.html"), System.Net.HttpWebRequest)
w_req.Timeout = 10000
Dim w_res As System.Net.HttpWebResponse = CType(w_req.GetResponse(), System.Net.HttpWebResponse)
Dim br As New System.IO.BinaryReader(w_res.GetResponseStream())
Dim fs As New System.IO.FileStream(LocalFileName, IO.FileMode.CreateNew, IO.FileAccess.Write, IO.FileShare.Write)
Dim b() As Byte = br.ReadBytes(CInt(br.BaseStream.Length))
fs.Write(b, 0, b.Length)
fs.Close()
br.Close()

Why do you use streams when you could easily use WebClient.DownloadFile? — svick, Feb 04 '10 at 14:34

score 0 · Accepted Answer · edited May 23 '17 at 12:13

0

If you want to download multiple files, use multiple threads. That way, waiting for one server's response will not slow down your application.

The stream handling looks fine to me, but I'm not an expert in these matters. If you fetch very large files, it might reduce memory consumption to do it in chunks. In .net 4.0, you would be able to use Stream.CopyTo.

Related question: Best way to copy between two Stream instances - C#

edited May 23 '17 at 12:13

Community

1
1

answered Feb 04 '10 at 13:34

Heinzi

167,459
57
363
519

...but bear in mind, writing correct, robust threaded software is more often than not non-trivial... – Mitch Wheat Feb 04 '10 at 13:37
Of course. Let's assume that I know how to do the threads already. (Which I probably do wrong, too, but nevermind that.) Is the streaming done right? – Eyal Feb 04 '10 at 13:41

score 0 · Answer 2 · answered Dec 08 '22 at 12:17

There is a better way to stream one stream into another:

        Dim w_res As Net.HttpWebResponse = CType(w_req.GetResponse(), HttpWebResponse)
        Dim tmpFName As String = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString()) & ".html"

        Dim fs As New FileStream(tmpFName, FileMode.CreateNew, FileAccess.Write, FileShare.Write)
        w_res.GetResponseStream().CopyTo(fs)
        fs.Close()

How do I download a file in vb .net with a download timeout?

2 Answers2