Azure NotificationHubClient throws SocketException when used on Azure function intensively

Question

We are using an Azure function to send push notifications via Azure NotificationHubClient. This function is called a lot at peak times on live environment and as a result we got a lot of SocketExceptions thrown from the NotificationHubClient.SendNotificationAsync() method.

Exception details:

Microsoft.Azure.NotificationHubs.Messaging.MessagingException: Unexpected exception encountered TimeStamp:2020-03-27T04:14:35.4655705Z ---> System.Net.Http.HttpRequestException: Cannot assign requested address ---> System.Net.Sockets.SocketException: Cannot assign requested address
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.Azure.NotificationHubs.NotificationHubClient.SendRequestAsync(HttpRequestMessage request, String trackingId, HttpStatusCode[] successfulResponseStatuses, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.NotificationHubs.ExceptionsUtility.HandleUnexpectedException(Exception ex, String trackingId)
   at Microsoft.Azure.NotificationHubs.ExceptionsUtility.TranslateToMessagingException(Exception ex, Int32 timeoutInMilliseconds, String trackingId)
   at Microsoft.Azure.NotificationHubs.NotificationHubClient.SendRequestAsync(HttpRequestMessage request, String trackingId, HttpStatusCode[] successfulResponseStatuses, CancellationToken cancellationToken)
   at Microsoft.Azure.NotificationHubs.NotificationHubClient.SendNotificationImplAsync(Notification notification, String tagExpression, String deviceHandle, CancellationToken cancellationToken)

We create the NotificationHubClient like:

new NotificationHubClient(connectionString, hubName, new NotificationHubClientSettings { OperationTimeout = TimeSpan.FromSeconds(10) });

I've observed that the Microsoft implementation behind is instantiating a new HttpClient on each new NotificationHubClient instantiation - which would probably be the problem(the well known socket exhaustion), but I have no ideea how this could be tackled.

Has anyone confronted with the same problem and managed to fix it some way? Thanks

Hey there, @vtrenea. Would you mind opening a support ticket through the Azure portal? The engineering team would like to investigate your situation further. — MarStr, Mar 27 '20 at 17:31
we are having very similar issues, however our exception is quite different. 'Unable to connect to the remote server.TrackingId:92cf019d-f3fc-4721-ac99-4e375f65f68b,TimeStamp:2020-04-08T06:56:24.9336294Z' but it is related to high volumes of pushes being sent, when we reboot the app, it immediately starts working again. — Zapnologica, Apr 09 '20 at 07:12

score 4 · Answer 1 · answered Apr 16 '20 at 06:34

4

In the end we managed to overcome the problem by injecting

    IHttpMessageHandlerFactory _httpMessageHandlerFactory;

in our class and pass it to the notificationHubClient in the constructor:

    return new NotificationHubClient(connectionString, hubName, 
    new NotificationHubClientSettings
    {
       OperationTimeout = TimeSpan.FromSeconds(10),
       MessageHandler = _httpMessageHandlerFactory.CreateHandler()
    });

answered Apr 16 '20 at 06:34

vtrenea

171
4

Can you elaborate on how you did this? We cannot get this working? We running in a windows service and want to use the same instance of 'httpclient' – Zapnologica Jul 14 '20 at 08:46

score 0 · Answer 2 · answered Mar 27 '20 at 13:42

0

There is a RESTful API for Notification Hubs. You can see it here. If you wanted more control, you could write your own wrapper around the methods you need, making sure to set the HttpClient instance as a static member variable within your Azure Function.

answered Mar 27 '20 at 13:42

Rob Reagan

7,313
3
20
49

Could one not just use a singleton / static instance of your 'NotificationHubClient'. Would this be safe to do? Have the authors of the library designed it in a way that I can be reused over and over. – Zapnologica Apr 09 '20 at 07:13
Turns out you cannot. The Nuget library recreates the `HttpClient` object every time.. – Zapnologica Jul 14 '20 at 08:46

Azure NotificationHubClient throws SocketException when used on Azure function intensively

2 Answers2

Linked