Httpclient calculating multiple url remote file size

Question

I want to calculate the total file size of 1000+ url. While the task is running I want to show the current number and total. But I'm getting error ssl if I will add the info which you can see in my code.

this is the error I'm getting:

    "Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)"

Could you please suggest what will be the best way to show the counts?

 #region GET REMOTE FILE SIZE
public static async Task<string> GetRemoteFileSize(this List<string> urls) {
    long sum = 0;
    long localSum = 0;
    var tasks = new List<Task<long?>>();
    for(int i = 0; i < urls.Count; i++) {
        var url = urls[i];
        tasks.Add(GetTotalBytes(url));
        //if i will add here the counts, it's not getting the actual count
        Console.Title = $"{i} of {urls.Count}";
    }
    
    for (int i = 0; i < tasks.Count; i++) {
        Task<long?> task = tasks[i];
        localSum += Convert.ToInt32(await task.ConfigureAwait(false));
        Console.Title = $"{i} of {urls.Count}"; //if I will add this, I'm getting the SSL error even if I already added this " ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }," to my httpclient handler
    }
    await Task.WhenAll(tasks.ToArray()).ConfigureAwait(false);
    return Interlocked.Add(ref sum, localSum).FormatFileSize();
}
private static async Task<long?> GetTotalBytes(string url) {

    Uri uri = new Uri(url, UriKind.Absolute);
    uri.SetServiceManagerConnection();

    using var request = new HttpRequestMessage(HttpMethod.Head, uri);
    request.AddAllCommonHeaders(uri);
    using var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
    HttpStatusCode statusCode = response.StatusCode;

    if (!response.IsSuccessStatusCode) {
        throw new InvalidDataException($"Error occured: {statusCode} {(int)statusCode}");
    }

    return response.Content.Headers.ContentLength.GetValueOrDefault();
}
#endregion GET REMOTE FILE SIZE

// the class I used

static string FormatBytes(this long bytes) {
        var unit = 1024;
        if (bytes < unit) { return $"{bytes} B"; }
        var exp = (int)(Math.Log(bytes) / Math.Log(unit));
        return $"{bytes / Math.Pow(unit, exp):F2} {("KMGTPE")[exp - 1]}B";
    }
    public static string FormatFileSize(this long bytes) => bytes.FormatBytes();

 public static void SetServiceManagerConnection(this Uri uri) {
        var sp = ServicePointManager.FindServicePoint(uri);
        sp.ConnectionLimit = 20; //default 2 //The number of connections per end point.
        sp.UseNagleAlgorithm = false; //Nagle’s algorithm is a means of improving the efficiency of TCP/IP networks by reducing the number of packets that need to be sent over the network
        sp.Expect100Continue = false; //save bandwidth before sending huge object for post and put request to ensure remote end point is up and running.
        sp.ConnectionLeaseTimeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds;//60 * 1000; // 1 minute
    }

UPDATE: this one is works:

  static HttpClient Client { get; set; } = new(new HttpClientHandler { 
        Proxy = null, 
        UseProxy = false,
        SslProtocols = (SslProtocols)(SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13)
    });

So, looks like the remote end cut you off (you got 0 bytes back, when you were expecting an SSL handshake). I don't think there's much we can add to that — canton7, Jan 06 '21 at 11:12
it's only showing the error. If i include the counting. Do you mind help me where is the best I can monitor the counting? — mark12345, Jan 06 '21 at 11:13
Can you create two [mcve] (emphasis on **minimal**) -- one which produces the error, and one which doesn't? — canton7, Jan 06 '21 at 11:15
there's no error and I can get the total length. I only getting the error if I put the console.title to monitor the counting — mark12345, Jan 06 '21 at 11:18

eriksv88 · Accepted Answer · 2021-01-06T11:33:09.137

0

Updated: This error is typical when there is a TLS handshake error with HTTPS. You should specify SecurityProtocol

System.Net.ServicePointManager.SecurityProtocol = 
SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;

Ref: https://learn.microsoft.com/en-us/dotnet/api/system.net.securityprotocoltype

edited Jan 06 '21 at 11:33

answered Jan 06 '21 at 11:16

eriksv88

3,482
3
31
50

score 0 · Answer 2 · answered Jan 06 '21 at 11:39

0

As mentioned by others, you are getting TLS errors.

Once you have solved that, a faster method for the operation you are doing will be HTTP HEAD, this returns to you just the headers, where you can see the size.

Be aware: not all servers allow HEAD, specifically AWS for example.

Answers to this question show how to do that with HttpClient: HTTP HEAD request with HttpClient in .NET 4.5 and C#

answered Jan 06 '21 at 11:39

Charlieface

52,284
6
19
43

I see. maybe it's not allowed HEAD. but if I use GET it will takes more time to calculate the total length – mark12345 Jan 06 '21 at 11:41

Httpclient calculating multiple url remote file size

2 Answers2