1

I would like to connect 100000 SignalR users to IIS (Windows Server 2016). Everything works fine until around 16000 connections. Then I start to receive this error:

Only one usage of each socket address (protocol/network address/port) is normally permitted

Here is my client code - it's a loop that creates all SignalR objects and it connects to the server:

private static void Run()
    {            
        for (int i = 0; i < 100000 ; i++)
        {
            Guid g = Guid.Empty;
            string lG = g.ToString();
            string lGS = lG.Substring(0, lG.Length - i.ToString().Length);
            string lTenantIdentfier = lGS + i.ToString();


            bool lConnected = false;
            SignalRClient sc = new SignalRClient(lTenantIdentfier);
            while (!lConnected)
            {
                sc.Stop();
                lConnected = sc.Connect();
                if (lConnected)
                    break;
                Console.WriteLine("[" + lTenantIdentfier + "] - Repeating connection...");
                System.Threading.Thread.Sleep(5000);
            }

            if ((i % 1000) == 0)
                Thread.Sleep(5000);              

        }

        Console.ReadKey();            
    }

Connect function:

public bool Connect()
    {
        try
        {
            if (connection != null)
            {
                Stop();
            }
            if (connection == null)
            {
                connection = new HubConnection("my_url");                 
                
                connection.Headers.Add("TenantIdentifier", TenantIdentifier);

                HubProxy = connection.CreateHubProxy("notificationHub");
                

                GetNotification = HubProxy.On("NotifyChange", () =>
                {
                    Console.ForegroundColor = ConsoleColor.Yellow;
                    Console.WriteLine("[" + TenantIdentifier + "] " + DateTime.Now);
                });

            }

            connection.Start().Wait();

            if (connection.State == ConnectionState.Connected)
            {
                Console.ForegroundColor = ConsoleColor.Green;
                Console.WriteLine("[" + TenantIdentifier + "] - Connected");            
                connection.Closed -= Connection_Closed;
                connection.Closed += Connection_Closed;
                return true;
            }
        }
        catch (Exception ex)
        {
            //Show exception
        }

        return false;
    }

I added to machine.config this under <system.web>:

 <processModel autoConfig="false" maxIoThreads="999999" maxWorkerThreads="999999" memoryLimit="999999" minIoThreads="999999" minWorkerThreads="999999" />

I added to regedit:

MaxUserPort 65535 TcpTimedWaitDelay 30

I suppose that IIS has no more free sockets or ports. Maybe I'm wrong - can I tweak it somehow?

Michal_Drwal
  • 526
  • 1
  • 9
  • 26
  • 1
    There are a lot of technical reasons why this would not work well, if they are constantly connecting and disconnecting good luck – TheGeneral Oct 20 '21 at 10:22
  • @CamiloTerevinto this is not my case. I unlocked the 5000 limit. – Michal_Drwal Oct 20 '21 at 10:27
  • So you're saying that the duplicate doesn't help because you cannot figure out how to change "5000" for "100000"? Also, regardless of IIS limits, you will face hardware limits long before that number. This simply doesn't make sense and cannot be done in a manner that it would perform consistently well. – Camilo Terevinto Oct 20 '21 at 10:32
  • @CamiloTerevinto What I'm saying is that I changed "5000" for "100000" before I asked this question. – Michal_Drwal Oct 20 '21 at 10:39
  • 1
    @CamiloTerevinto it's unlikely a duplicate of that question, because here OP basically exhausted all ports on the system, which is different from what is described in "duplicate" – Evk Oct 20 '21 at 10:40
  • @Evk I agree. But shouldn't I be able to connect around 60k users? Because the port limit is 65535. – Michal_Drwal Oct 20 '21 at 10:42
  • Are you connecting from the same machine your server runs at? Does that error happen on client or server side? – Evk Oct 20 '21 at 10:55
  • @Evk These are different machines. Error is show on client but it's from the server. – Michal_Drwal Oct 20 '21 at 11:01
  • Your 16,000 finding as the max number of connections to a single Windows server jives with my experience. I believe it's limited by the non-paged memory pool where socket objects are allocated from, the size of which is proportional to installed RAM, so it's loosely related to that. – 500 - Internal Server Error Oct 20 '21 at 11:24
  • 1
    What happens if you apply MaxUserPort and TcpTimedWaitDelay on _client_? – Evk Oct 20 '21 at 11:37
  • @Evk Yes, I changed MaxUserPort on client and now it goes above 16k and keeps going. I checked netstat on IIS and there were my clients' ports starting from 49165. So it's the client that runned out of ports. Thanks. – Michal_Drwal Oct 20 '21 at 12:28

2 Answers2

2

As discussed in comments, the reason seems to be that client runs out of free ports. That MaxUserPort has default value of 5000 on windows, so only ports from 1024 to 4999 are available for client to connect to server. Server itself can't run out of free ports this way, because all connections go to the same port (take usual webserver for example, all clients connect to port 80 or port 443 etc, same story here).

Then you try to connect from client in a loop, using new local port for each connection. I suspect that you can connect more than (4999-1024) times because you do not explicitly close signalR connections, but you also do not store them in some kind of a global list, so they are eligible for garbage collection, which calls finalizer, which in turn closes the connection. Or connection is closed by other means, but not fast enough so at some point you exhaust the local ports on client, but the number you have likely does not represent number of concurrent connections.

Setting MaxUserPort to 65535 and TcpTimedWaitDelay 30 on client should allow you to make more connections from this client.

Evk
  • 98,527
  • 8
  • 141
  • 191
0

Evk is correct but here is something interesting I remembered reading when looking to into user connection volume.

From the original SignalR leader, Damien Edwards, back in 2014... Yes this is possible. This would have been under the ASP.NET version 2.x. But if you read through a few replies Damien say that .NET CORE version should go much higher.

https://twitter.com/DamianEdwards/status/486642486350061568

New @SignalR concurrent connection record on 10GB svr: 150,000 connections (WebSockets), w/ experimental buffer pooling changes.

Frank M
  • 1,379
  • 9
  • 18