0

I am downloading about 1000000 xml files from the internet.

I have tried many ways for this, but when I download fast, the files are empty. Otherwise it is very slow ... How can I do better, Best regards...

WebClient client = new WebClient();
NetworkCredential myCreds = new NetworkCredential("user", "pass");
client.Credentials = myCreds;


         foreach (var id in ids)
                {
                    try
                    {
                        string url = "http://....../fileid=" + id;

                        client.Encoding = Encoding.UTF8;
         //  client.DownloadFileAsync(new Uri(url), "C:\\Data\\"+id+".xml");

                        string result= client.DownloadString(url);
                        WriteFile(id, result);
                    }
                    catch (Exception)   {   }
                }


     private void WriteFile(decimal id, string text)
            {
                path ="C:\\Data\\"+id+".xml";
                using (StreamWriter sw = File.CreateText(path))
                {
                    sw.WriteLine(text);
                }
            }
Şaban Selvitop
  • 1
  • You must be getting an exception when you are not getting results. When running async how can you tell when each file completely get downloaded or when all the file are completed? – jdweng Aug 11 '18 at 17:23
  • Though it took a very long time, we have already downloaded all the data. So we have information about the data. Failure to find some files due to the fall or webclient object time out due to the need to catch the error we did. The problem now is that when downloading async, the overwrite of the data or the data is empty, it takes too long when we download it as sync ... – Şaban Selvitop Aug 13 '18 at 05:39
  • The download time should not be any different in Sync and Async. With the Async mode, you will get the data in pieces and need to wait for the entire file to be received. For timeout see following p[osting :https://stackoverflow.com/questions/1789627/how-to-change-the-timeout-on-a-net-webclient-object – jdweng Aug 13 '18 at 09:32

0 Answers0