How can I make this image crawler more efficient?

Question

I am making a simple application that have to fetch all images in subdirectories from a site and recreate the file and folder structure locally. Here is what i have so far:

string folder = "c:\someLocalFolder\";

// To keep track of how many files that are processed
int i = 0;

// Folder x
for (int x = 2; x <= 7; x++)
{
   // Folder y
   for (int y = 0; y < 80; y++)
   {
       //File z
       for (int z = 0; z <= 70; z++)
       {
           // File, increment
           i++;

           string destFolderPath = Path.Combine(folder, x.ToString(), y.ToString());
           string filePath = Path.Combine(destFolderPath, z.ToString() + ".png");

           if (!File.Exists(filePath))
           {
               var url = string.Format("http://www.somesite.com/images/{0}/{1}/{2}.png", x, y, z);
               if (!Directory.Exists(destFolderPath))
                   // Folder doesnt exist, create
                   Directory.CreateDirectory(destFolderPath);
               var webClient = new WebClient();
               webClient.DownloadFileCompleted += (o, e) =>
               {
                   // less than 1 byte recieved, delete
                   if( (new FileInfo(filePath).Length) < 1 ) 
                   {
                       File.Delete(filePath);
                   }
                   // File processed
                   i--;
                   Console.WriteLine(i);
               };
               webClient.DownloadFileAsync(new Uri(url), filePath);
           }
           else
           {
               // File processed
               i--;
               Console.WriteLine(i);
           }
       }
   }
}

So as you can se currently i am iterating and creating the folder structure, then i am download file asynchronously and then check if the file is lesser than 1 byte in file size and deleting it if it is.

I think that i am doing this in an very cumbersome way, it is not very fast, and it makes a whole lot of files only do delete the once that does not meet the requirements.

Is there any faster way of determining if the file exists on the web server, and based on that download if exist, and the way i create the folder structure, is this an appropriate way how i do it to accomplish what i want?

score 1 · Answer 1 · edited May 23 '17 at 11:59

Is there any faster way of determining if the file exists on the web server

You can send a HEAD request to the webserver.

If the webserver supports the method, check the returned status code.

When status code is 200, it means the file does exist.
When status code is 404, it means the file doesn't exist.

On the other hand, if the webserver doesn't support this method, you fallback to your original code.

See this SO question for details: How to send a HEAD request with WebClient in C#?

the way i create the folder structure, is this an appropriate way

There is an invariant in the //File z for loop:

string destFolderPath = Path.Combine(folder, x.ToString(), y.ToString());

Try this instead:

// Folder x
for (int x = 2; x <= 7; x++) {
   string xAsString = x.ToString();

   // Folder y
   for (int y = 0; y < 80; y++) {
       string destFolderPath = Path.Combine(folder, xAsString, y.ToString());

       //File z
       for (int z = 0; z <= 70; z++) {
           // File, increment
           i++;
           string filePath = Path.Combine(destFolderPath, z.ToString() + ".png");

           // ...
       }
   }
}

How can I make this image crawler more efficient?

1 Answers1