Managed Sockets not behaving as (I) expected

Question

So I've been having a tough time finding documentation on exactly how sockets should behave when you have 2, both bound to the same endpoint, but one of them is also connected to a remote endpoint.

• The sockets are UDP IPv4
• Running in .net core 2.2/3 on linux x64

What I have been able to gather from various sources, is that the connected socket should always and only receive datagrams from the endpoint it is connected to and the "unconnected" socket will receive everything else.

I vaguely remember reading that the kernel socket implementation assigns "points" to each socket when a dgram arrives, and the socket with the higher score (most specific route) gets the data. If two socket get the same score, the dgrams are "load balanced" between the sockets.

I put together a small test:

class Program
{
    static void Main(string[] args)
    {
        var localEp = new IPEndPoint(IPAddress.Loopback, 1114);
        var remoteEp = new IPEndPoint(IPAddress.Loopback, 1115);

        //Socket bound to local EP, not connected should receive from everyone
        var notConnected = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
        notConnected.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
        notConnected.Bind(localEp);

        //Socket bound and connected should receive from only it's remote EP
        var connected = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
        connected.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
        connected.Bind(localEp);
        connected.Connect(remoteEp);

        var notConnectedTask = Task.Run(() => Receive(notConnected, "Not Connected"));
        var connectedTask = Task.Run(() => Receive(connected, "Connected"));

        //Remote socket to send to connected socket
        var remote1 = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
        remote1.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
        remote1.Bind(remoteEp);
        remote1.Connect(localEp);

        //Remote socket to send to notConnected socket
        var remote2 = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
        remote2.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
        remote2.Bind(new IPEndPoint(IPAddress.Loopback, 1116));
        remote2.Connect(localEp);

        for (int i = 0; i < 10; i++)
        {
            //This should be received by connected socket only
            remote1.Send(Encoding.Default.GetBytes($"message {i} to connected socket"));

            //This should be received by unconnected socket only
            remote2.Send(Encoding.Default.GetBytes($"message {i} to notConnected socket"));
        }

        remote1.Send(Encoding.Default.GetBytes("end"));
        remote2.Send(Encoding.Default.GetBytes("end"));

        Task.WaitAll(notConnectedTask, connectedTask);
    }

    public static void Receive(Socket sock, string name)
    {
        EndPoint ep = new IPEndPoint(IPAddress.Any, 0);
        var buf = new byte[1024];

        Console.WriteLine($"{name} is listening...");

        while (true)
        {
            var rcvd = sock.ReceiveFrom(buf, ref ep);

            var msg = Encoding.Default.GetString(buf.Take(rcvd).ToArray());

            Console.WriteLine($"{name} => {msg}");

            if (msg.SequenceEqual("end"))
                return;
        }
    }
}

To My surprise and chagrin, the result was nothing close to what I expected:

Connected is listening...
Not Connected is listening...
Connected => message 0 to connected socket
Connected => message 0 to notConnected socket
Connected => message 1 to connected socket
Connected => message 1 to notConnected socket
Connected => message 2 to connected socket
Connected => message 2 to notConnected socket
Connected => message 3 to connected socket
Connected => message 3 to notConnected socket
Connected => message 4 to connected socket
Connected => message 4 to notConnected socket
Connected => message 5 to connected socket
Connected => message 5 to notConnected socket
Connected => message 6 to connected socket
Connected => message 6 to notConnected socket
Connected => message 7 to connected socket
Connected => message 7 to notConnected socket
Connected => message 8 to connected socket
Connected => message 8 to notConnected socket
Connected => message 9 to connected socket
Connected => message 9 to notConnected socket
Connected => end

Not only did the notConnected socket not receive anything, but the connected socket got everything...

So neither of my expectations seem true. No load balancing, and no point system.

I had posted a comment on SO asking this question and got a reply almost confirming my expectation:

I think if one is connected to a remote endpoint, then all datagrams originating from that remote endpoint will end up at the connected socket. The unconnected one would only catch datagrams from other remote endpoints.

And I also have an e-mail from the OpenSSL mailing group mostly confirming it as well...

I suppose the above test should answer my question definitively, but it just seems so wrong!

Perhaps I made a mistake in the code, or I'm just missing something. I'd appreciate a bit of guidance.

Am I completely wrong about how sockets work?

EDIT

So I just re-ran my test, exactly as above, and the result is almost the same, only the socket receiving the data is the "notConnected" socket.

Binding the notConnected socket after the connected socket also has no effect.

Answer 1

I vaguely remember reading that the kernel socket implementation assigns "points" to each socket when a dgram arrives, and the socket with the higher score (most specific route) gets the data. If two socket get the same score, the dgrams are "load balanced" between the sockets.

I would be curious to know where you read that. Because it's not consistent with anything I've ever read or heard about reusing socket addresses.

Rather, my understanding has always been that if you reuse a socket address, the behavior is undefined/non-deterministic. For example:

Once the second socket has successfully bound, the behavior for all sockets bound to that port is indeterminate.

When I run your test code, I get behavior opposite from which you report. In particular the "Not Connected" socket is the one that receives all of the traffic.

When I modify the code so that both sockets call Connect(), one each to each of the remote endpoint addresses, only one socket winds up getting any datagrams. This is also consistent with my understanding and the previous test. In particular, Connect() on a connectionless-protocol socket operates at the socket level, filtering out any datagrams the socket receives before the application sees them.

So, on my computer the "Not Connected" socket is the one that's getting all the traffic, and if I tell it to connect to one of the remote endpoints that is sending datagrams, then while it still receives all of the traffic, my application sees only those datagrams it asked for with the Connect() call. The other datagrams are discarded.

(As an aside: in my view, "connecting" a socket that is using a connectionless protocol should be considered simply a convenience, and should not be viewed as actually connecting the socket. The same socket can still send datagrams to other remote endpoints, via SendTo(), and the socket is still receiving traffic from other remote endpoints, your program is just not getting to see that traffic.)

For reused socket addresses, I have also seen in the past, traffic delivered randomly. I.e. sometimes one socket gets the traffic, sometimes the other one does. The fact that there is at least some socket that is consistently receiving the traffic is an improvement over that!

But nonetheless, I don't believe you should have any reason to ever expect SocketOptionName.ReuseAddress to work reliably. It's not documented to do so, and in my experience it does not. Both the results you report, as well as the different results I obtained with the same code, are entirely consistent with the "non-deterministic" nature of reusing socket addresses.

If you have seen anything that claims that reusing socket addresses can and/or should produce some deterministic result, I would say that reference is simply incorrect.

Answer 2

So reading over the linux man pages for SO_REUSEPORT, I came across this:

For UDP sockets, the use of this option can provide better distribution of incoming datagrams to multiple processes (or threads) as compared to the traditional technique of having multiple processes compete to receive datagrams on the same socket.

So it would seem that SO_REUSEPORT is what I need, rather than SO_REUSEADDRESS. Which is unfortunate because SO_REUSEPORT is not available on windows...

Also, to confirm Peter Duniho's answer, straight from the horse's mouth:

The motivating case for so_reuseport in UDP would be something like a DNS server. An alternative would be to recv on the same socket from multiple threads. As in the case of TCP, the load across these threads tends to be disproportionate and we also see a lot of contection on the socket lock. Note that SO_REUSEADDR already allows multiple UDP sockets to bind to the same port, however there is no provision to prevent hijacking and nothing to distribute packets across all the sockets sharing the same bound port.

To sum it up, multiple UDP sockets bound to the same endpoint with SO_REUSEADDRESS set will have undefined behavior. That is to say there is no way to tell where the data will end up.

Multiple UDP sockets bound with SO_REUSEPORT will see the dgrams distributed in a sort of "load balanced" way.

As I still don't know how one connected/bound socket and one bound socket will behave with SO_REUSEPORT, I will test my scenario above with SO_REUSEPORT and update this answer.

So this commit to the linux kernel does in fact implement the "socket scoring" system I thought I had read about. Specifically, static int compute_score seems to take a "udp table" and compute the socket with the highest score for a given datagram. This should guarantee that a connected socket will receive dgrams from it's endpoint, even when another socket is also bound to the same local endpoint.

This is a gist I created to test this case. It works as I had hoped, with the connected socket always receiving dgrams from it's remote endpoint.