How to check asynchronously if an image in HTML exists?

Question

I'm writing a program that uses an API which crawls the web to fetch pieces of information. I am then showing the results on an HTML Document (Excel was an option, but due to some restrictions and features I opted for an Html Document)

Long story short, the request result often have links to an image and I need to show them to the user. The problem is that sometimes some images don't exist anymore.

The basic is as follow:

string carouselItems = "";

for (int i = 0; i < imgs.Count; i++)
{

    if (i == 0)
        carouselItems += $"" +
            $"<div class=\"carousel-item active\">" +
            $"  <img src = \"" + imgs[i] + "\" class=\"d-block w-100\" alt=\"Bild "
                + (i + 1).ToString() + " von " + imgs.Count.ToString()
                + "\" onerror=\"this.style.display = 'none'\">" +
            $"  <div class=\"carousel-caption d-none d-md-block\">" +
            $"      <p>Bild " + (i + 1).ToString() + "</p>" +
            $"  </div>" +
            $"</div>";
    else
        carouselItems += $"" +
            $"<div class=\"carousel-item\">" +
            $"  <img src = \"" + imgs[i] + "\" class=\"d-block w-100\" alt=\"Bild "
                + (i + 1).ToString() + " von " + imgs.Count.ToString()
                + "\" onerror=\"this.style.display = 'none'\">" +
            $"  <div class=\"carousel-caption d-none d-md-block\">" +
            $"      <p>Bild " + (i + 1).ToString() + "</p>" +
            $"  </div>" +
            $"</div>";
}

I thought to make an HttpRequest to the link and to hold on only to those images that returned a content type of "image", because sometime some links redirect me to a website (probably because the image doesn't exist anymore), like this:

for (int i = 0; i < line.Bild.Count; i++)
{
    var client = new RestClient(line.Bild[i]);
    var request = new RestRequest(Method.GET);
    client.FollowRedirects = true;
    request.AddHeader("Accept", "image/*");
    var res = client.Execute(request);
    if (res.ContentType.Contains("image/")) images.Add(line.Bild[i]);
}

This approach is fine, but "slow" whenever I have hundreds of images to check. What I would like to achieve is some form of parallelization for which all images are checked at the same time and I am returned with a list of unbroken images. I am not so good with asynchronous coding that's why I need some help.

I will be happy to read your answers! BR

Answer 1

You should start by implementing an asynchronous method that checks the existence of individual image URLs. This method should have a return value of Task<bool> instead of bool, since it is asynchronous. Below is a basic implementation, that you may want to customize to your needs:

private HttpClient _httpClient = new HttpClient();

private async Task<bool> CheckImageAsync(string imageUrl)
{
    try
    {
        byte[] imageBytes = await _httpClient.GetByteArrayAsync(imageUrl);
        using (var ms = new MemoryStream(imageBytes))
        {
            var image = Image.FromStream(ms);
        }
        return true;
    }
    catch (Exception)
    {
        return false;
    }
}

Then inside your main method that does the fetch/parse/extract work, you can use the above method to create one task for each unverified image URL in the raw HTML. All these tasks will start running immediately and will be running concurrently. You must wait all of them to complete before continuing. You can wait them either synchronously or asynchronously. Below is an example with synchronous waiting, that you may be more familiar with. It uses LINQ and value tuples to combine each URL with a bool result.

var unverifiedImageUrls = new List<string>();
// Here put the code that extracts the images from the raw HTML,
// and puts them into the list.

Task<(string Url, bool Verified)>[] verifyTasks = unverifiedImageUrls
    .Select(async url => (Url: url, Verified: await CheckImageAsync(url)))
    .ToArray();

string[] verifiedImageUrls = Task.WhenAll(verifyTasks).Result
    .Where(result => result.Verified)
    .Select(result => result.Url)
    .ToArray();

// Here use the resulting array to generate your HTML result

If you prefer to make everything asynchronous, you should remove the .Result and add an await before the Task.WhenAll. But this change will require changes to other parts of your program as well.

Answer 2

You could save a lot of time just by switching GET to HEAD, which tells to receive metadata only, not the actual file.

using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace ConsoleApp3
{
    class Program
    {
        static HttpClient httpClient = new HttpClient();

        static async Task<bool> IsImageExists(string url)
        {
            var result = await httpClient.SendAsync(new HttpRequestMessage()
            {
                RequestUri = new Uri(url),
                Method = HttpMethod.Head,
            });

            return result.IsSuccessStatusCode;
        }

        static async Task Main(string[] args)
        {
            bool exists = await IsImageExists("https://upload.wikimedia.org/wikipedia/en/9/95/Test_image.jpg");

            if (!exists)
            {
                // do something else to confirm that image does not exist, e.g. your regular GET request
            }
        }
    }
}