MacOS SO_REUSEADDR/SO_REUSEPORT not consistent with Linux?

Question

Consider this code:

#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <string.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

#define SERVADDR "::1"
#define PORT 12345

int main() {
    int sd = -1;

    if ((sd = socket(AF_INET6, SOCK_STREAM, 0)) < 0) {
        fprintf(stderr, "socket() failed: %d", errno);
        exit(1);
    }

    int flag = 1;
    if(setsockopt(sd, SOL_SOCKET, SO_REUSEADDR, &flag, sizeof(flag)) == -1) {
        fprintf(stderr, "Setsockopt %d, SO_REUSEADDR failed with errno %d\n", sd, errno);
        exit(2);
    }
    if(setsockopt(sd, SOL_SOCKET, SO_REUSEPORT, &flag, sizeof(flag)) == -1) {
        fprintf(stderr, "Setsockopt %d, SO_REUSEPORT failed with errno %d\n", sd, errno);
        exit(3);
    }

    struct sockaddr_in6 addr;
    memset(&addr, 0, sizeof(addr));
    addr.sin6_family = AF_INET6;
    addr.sin6_port = htons(23456);

    if(bind(sd, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
        fprintf(stderr, "Bind %d failed with errno %d: %s\n", sd, errno, strerror(errno));
        exit(4);
    }

    struct sockaddr_in6 server_addr;
    memset(&server_addr, 0, sizeof(server_addr));

    server_addr.sin6_family = AF_INET6;
    inet_pton(AF_INET6, SERVADDR, &server_addr.sin6_addr);
    server_addr.sin6_port = htons(PORT);

    if (connect(sd, (struct sockaddr *)&server_addr, sizeof(server_addr)) == -1) {
        fprintf(stderr, "Connect %d failed with errno %d: %s\n", sd, errno, strerror(errno));
        exit(5);
    }

    printf("Seems like it worked this time!\n");
    close(sd);
}

Pretty simple:

• create socket
• set SO_REUSEADDR
• set SO_REUSEPORT
• bind to local port 23456
• connect to ::1 on port 12345

Weirdly enough on MacOS running this in a row causes this:

$ for i in {1..5}; do ./ipv6; done
Seems like it worked this time!
Connect 3 failed with errno 48: Address already in use
Connect 3 failed with errno 48: Address already in use
Connect 3 failed with errno 48: Address already in use
Connect 3 failed with errno 48: Address already in use
$

While running this on Linux seems to work just fine:

$ for i in {1..5}; do ./ipv6; done
Seems like it worked this time!
Seems like it worked this time!
Seems like it worked this time!
Seems like it worked this time!
Seems like it worked this time!
$

I have a listener on port 12345:

$ nc -6 -l -v -p12345 -k

This is NOT limited to IPv6, tried the same thing with IPv4 - same behavior.

Can anybody explain it?

I previously thought it was failing in bind() but it's in connect().

Edit #1

According to How do SO_REUSEADDR and SO_REUSEPORT differ?, this applies to BSD:

So if you bind two sockets of the same protocol to the same source address and port and try to connect them both to the same destination address and port, connect() will actually fail with the error EADDRINUSE for the second socket you try to connect, which means that a socket with an identical tuple of five values is already connected.

So that makes sense why that doesn't work. What doesn't make sense if how is it possible this actually works on Linux?

I'd ideally of course have this work on MacOS but I currently feel like it might not be possible - I'd however still like to understand how Linux does it.

Answer 1

Yes, the Linux implementation is different to the most other OS. You can find an exhaustive explanation here. To quote the specific part:

Linux 3.9 added the option SO_REUSEPORT to Linux as well. This option behaves exactly like the option in BSD and allows binding to exactly the same address and port number as long as all sockets have this option set prior to binding them.

Yet, there are still two differences to SO_REUSEPORT on other systems:

1. To prevent "port hijacking", there is one special limitation: All sockets that want to share the same address and port combination must belong to processes that share the same effective user ID! So one user cannot "steal" ports of another user. This is some special magic to somewhat compensate for the missing SO_EXCLBIND/SO_EXCLUSIVEADDRUSE flags.
2. Additionally the kernel performs some "special magic" for SO_REUSEPORT sockets that isn't found in other operating systems: For UDP sockets, it tries to distribute datagrams evenly, for TCP listening sockets, it tries to distribute incoming connect requests (those accepted by calling accept()) evenly across all the sockets that share the same address and port combination. Thus an application can easily open the same port in multiple child processes and then use SO_REUSEPORT to get a very inexpensive load balancing.