Microsoft Face API: Not working for bulk images

Question

I am trying to fetch faces from images inside a folder. If there are relatively less number of images, it works but if there are huge number of images(around 1500) it raises an error after processing around 10 images.
"MakeAnalysisRequest" is the function that sends the request to Microsoft FaceAPI. This function also add "faceId" to the list because I need this list only after processing of all the images is finished. That's why there is this piece of code in the function

foreach (FileInfo file in files)
{
    tasks.Add(MakeAnalysisRequest(file.FullName, file.Name, delay, subscriptionKey, endPointURL));
}

//Waiting for "MakeAnalysisRequest" to finish.
Task.WaitAll(tasks.ToArray());

If I process around 100 images, everything works fine. However if I try to run the process for around 1500 images it fails(raises "There was an error while saving the file!"). The Exception only says "Aggregate Exception". Handling "Aggregate" exception also do not reveal anything in inner exception. Here is the complete code(Variables are not included. Only relevant code for readability):

public void ProcessImages(string imageFolderPath)
{
    try
    {
        DirectoryInfo dInfo = new DirectoryInfo(imageFolderPath);
        //Only jpeg|jpg files are needed for the process. Filtering the images in folder. 
        string[] extensions = new[] { ".jpg", ".jpeg" };
        FileInfo[] files = dInfo.GetFiles().Where(f => extensions.Contains(f.Extension.ToLower())).ToArray();
        var tasks = new List<Task>();

        foreach (FileInfo file in files)
        {
            tasks.Add(MakeAnalysisRequest(file.FullName, file.Name, delay, subscriptionKey, endPointURL));
        }

        //Waiting for "MakeAnalysisRequest" to finish.
        Task.WaitAll(tasks.ToArray());

        //faceIds are generated in MakeAnalysisRequest function.
        CreateFaceGroups(faceIds);
    }
    catch (Exception ex)
    {
        Console.WriteLine("There was an error while saving the file! ");
    }
}

static async Task MakeAnalysisRequest(string imageFilePath, string fileName, int delay, string subscriptionKey, string endPointURL)
{
    //To control the numberof requests sent per second. Client does not allow more than 10 per second.
    await Task.Delay(delay);

    HttpClient client = new HttpClient();

    HttpResponseMessage response;

    // Request body. Posts a locally stored JPEG image.
    byte[] byteData = GetImageAsByteArray(imageFilePath);

    using (ByteArrayContent content = new ByteArrayContent(byteData))
    {
        content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");

        // Execute the REST API call.
        response = await client.PostAsync(endPointURL, content);

        // Get the JSON response.
        string contentString = await response.Content.ReadAsStringAsync();
        if (!String.IsNullOrEmpty(contentString))
        {
            try
            {
                //Converting the response to DTO class object.
                List<RootObject> objRoot = JsonConvert.DeserializeObject<List<RootObject>>(contentString);
                foreach (var obj in objRoot)
                {
                    if (obj != null)
                    {
                        objFaceIdImage.Add(new FaceIdImageLink { faceId = obj.faceId, imageName = fileName });
                        faceIds.Add(obj.faceId);
                        //Based on the coordinates creating face image by cropping the source image.
                        CropImage(fileName, imageFilePath, obj.faceRectangle.left, obj.faceRectangle.top, obj.faceRectangle.width, obj.faceRectangle.height, obj.faceId);
                    }
                }
            }
            catch (Exception ex)
            {
                var a = ex;
            }
        }
        else
            ReStart("No faces found in the images!");
    }
}

Can't find out where the issue is!

Answer 1

The problem is that your code to limit your processing to 10 requests per second doesn't work. In fact all it does is ensure that every request is delayed by delay and then they all fire at once. A safer solution would be to do the processing in batches and only delay if a batch was done in less than a second. For example (untested code) and using this handy batching extension method from this answer:

public void ProcessImages(string imageFolderPath)
{
    FileInfo[] files = ...

    foreach (var batch in files.Batch(10))
    {
        var stopWatch = new System.Diagnostics.Stopwatch();
        stopWatch.Start();

        //Process the files as before, but now only 10 at a time
        ProcessBatch(batch);

        stopWatch.Stop();

        //If less than 1 second has passed, let's hang around until it has
        if(stopWatch.ElapsedMilliseconds < 1000)
        {
            //Probably don't use Thread.Sleep here, but you get the idea
            Thread.Sleep(1000 - (int)stopWatch.ElapsedMilliseconds);
        }
    }
}

private void ProcessBatch(IEnumerable<FileInfo> batch)
{
    //Process the batch as before
    var tasks = new List<Task>();

    foreach (FileInfo file in batch)
    {
        //Remove the 'delay' parameter as it's no longer needed
        tasks.Add(MakeAnalysisRequest(file.FullName, file.Name, 
            subscriptionKey, endPointURL));
    }

    //Waiting for "MakeAnalysisRequest" to finish.
    Task.WaitAll(tasks.ToArray());
}