I want to download large file from limited-transfer server using several threads with HttpWebRequest.AddRange(int, int). The problem is that all my threads but one get blocked on GetResponse(), probably becouse they want to reuse the connection of the first thread. I've set the KeepAlive property of request to false and I've increased the maximum limit of concurrent connections with ServicePointManager.DefaultConnectionLimit = 1000;
Funny thing, it sometimes works. It works most of the time if I use 2 threads, sometimes with 3 and I've never seen it working with 4 or more threads.
This is part of my code, the not-really-important parts are removed, still the snippet given is a continous block in my code and I haven't removed anything in between despite the comments.
for (int i=0; i < threadpool; i++)
{
pool[i] = new Thread(new ParameterizedThreadStart((object id) =>
{
int myId = (int)(id);
Console.WriteLine("StartThread " + myId);
byte[] buffer = new byte[chunksize];
while (currentChunk < chunks)
{
int myJob = -1;
HttpWebResponse response = null;
lock (currentChunkLock)
{
myJob = currentChunk++;
}
Console.WriteLine(myId + " GOT JOB " + myJob);
HttpWebRequest request = MakeRangedRequest(url, myJob * chunksize, chunksize);
Console.WriteLine(myId + " MADE REQUEST " + myJob);
response = (HttpWebResponse)request.GetResponse();
//They get all stuck here
Console.WriteLine(myId + " GOT RESPONSE " + myJob);
int totalCount = 0;
int targetCount = (myJob + 1 == chunks) ? lastChunkSize : chunksize;
Thread.Sleep(1);
while (totalCount < targetCount)
{
//The only thread that passes is stuck here, it won't allow other threads to continue.
int left = targetCount-totalCount;
int count = response.GetResponseStream().Read(buffer, totalCount, left > 1024 ? 1024 : left);
totalCount += count;
totalBytesSoFar += count;
Console.WriteLine("READING " + myId + "/" + totalCount);
Thread.Sleep(1);
}
response.Close();
Console.WriteLine(myId + " READ BUFFER " + myJob);
lock (file)
{
file.Seek(myJob * chunksize, SeekOrigin.Begin);
file.Write(buffer, 0, chunksize);
file.Flush();
}
Console.WriteLine(myId + " WRITE FILE " + myJob);
Console.WriteLine("Current chunk: " + myJob + "/" + chunks + "\r");
Console.WriteLine("Thread " + myId);
Thread.Sleep(1);
}
Console.WriteLine("Thread " + myId + " out.");
lock (threadsDonePool)
{
threadsDone++;
if (threadsDone == threadpool)
{
file.Close();
Console.WriteLine("File Closed.");
}
}
}));
pool[i].Start(i);
}
And here is the MakeRangedRequest function:
static HttpWebRequest MakeRangedRequest(string url, int from, int size)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AddRange(from, from + size);
request.KeepAlive = false;
return request;
}
Am I forced to do this all using the TCP classes? It would be great to stick to the HttpWebRequest
