How can a socket connect when bound to a specific ethernet interface that's also being used by VPN / utun interface?

Question

I'm trying to write a function that can connect to a server using a specific network interface so that it's consistently routed through that interface's gateway. This is on a macOS system that has one or more VPN connections.

Here's a proof-of-concept test function I've written:

void connectionTest(const char *hostname, int portNumber, const char *interface) {
    struct hostent *serverHostname = gethostbyname(hostname);
    
    if (serverHostname == NULL) {
        printf("error: no such host\n");
        return;
    }

    int socketDesc = socket(AF_INET, SOCK_STREAM, 0);
    int interfaceIndex = if_nametoindex(interface);
    
    if (interfaceIndex == 0) {
        printf("Error: no such interface\n");
        close(socketDesc);
        return;
    }
    
    // Set the socket to specifically use the specified interface:
    setsockopt(socketDesc, IPPROTO_IP, IP_BOUND_IF, &interfaceIndex, sizeof(interfaceIndex));

    struct sockaddr_in servAddr;
    bzero((char *)&servAddr, sizeof(servAddr));
    servAddr.sin_family = AF_INET;
    bcopy((char *)serverHostname->h_addr, (char *)&servAddr.sin_addr.s_addr, serverHostname->h_length);
    servAddr.sin_port = htons(portNumber);

    if (connect(socketDesc, (struct sockaddr *)&servAddr, sizeof(servAddr)) < 0) {
        printf("connect failed, errno: %d", errno);
        close(socketDesc);
        return;
    }
    
    printf("connection succeeded\n");
    close(socketDesc);
}

This function will successfully connect so long as the interface is one of the utun interfaces created by the VPNs, or a physical interface that is not used by the VPNs. But if I try to use the physical interface that is used by the VPNs, the function fails with errno 51: Network is unreachable.

For a more specific example, consider a system with the following network interfaces:

en0: Ethernet connection
en1: Wi-Fi connection
utun10: VPN connection 1, connected via en0
utun11: VPN connection 2, also connected via en0

If I call my function with something like:

connectionTest("api.ipify.org", 80, "en1");
connectionTest("api.ipify.org", 80, "utun10");
connectionTest("api.ipify.org", 80, "utun11");

... it will succeed. However, this is what produces the "network unreachable" error:

connectionTest("api.ipify.org", 80, "en0");

Is there some way to have the function work in the case of en0? (Preferably without changing the system's routing table just for this one connection?)

Edit:

It looks like the system doesn't know how to route packets through en0 when the VPN is up, unless it has a non-default route for en0.

I tried using the route command to check which route in the table would be used for a specific interface, and I get the following:

$ route get -ifscope en0 1.1.1.1
route: writing to routing socket: not in table

Only -ifscope en0 produces that error. However, the route table indicates there is a default route for en0. Here is the routing table when only ethernet and the VPN are connected (so no Wi-Fi or second VPN):

$ netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
0/1                10.16.0.1          UGSc          165        0  utun10
default            192.168.20.1       UGSc            0        0     en0
10.16/16           10.16.0.8          UGSc            3        0  utun10
10.16.0.8          10.16.0.8          UH              2        0  utun10
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              7  7108160     lo0
128.0/1            10.16.0.1          UGSc           40        0  utun10
169.254            link#8             UCS             1        0     en0      !
192.168.20         link#8             UCS             9        0     en0      !
192.168.20.1/32    link#8             UCS             2        0     en0      !
224.0.0/4          link#22            UmCS            0        0  utun10
224.0.0/4          link#8             UmCSI           1        0     en0      !
224.0.0.251        1:0:5e:0:0:fb      UHmLWI          0        0     en0
255.255.255.255/32 link#22            UCS             0        0  utun10
255.255.255.255/32 link#8             UCSI            0        0     en0      !

There's clearly a default route listed for en0 pointing to its gateway, 192.168.20.1. Why isn't the packet being routed? If I create a static route for 1.1.1.1/32 or even 1/8 it will work. But so long as en0 only has a default route, it won't work. It's like the default route has been disabled somehow.

Edit 2:

If I add a new route to the table using:

$ route add -ifscope en0 0/0 192.168.20.1

so that the routing table now includes the following entry:

Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.20.1       UGScI           1        0     en0

alongside all of the above entries, so there are now two default entries, then the connection works. Why is it necessary for there to be an interface-specific default route in order for this to work?

That means there is no route to that network through that interface. That message could be either from your host or a router through which the packet travels on the way to the destination. — Ron Maupin, Jun 24 '22 at 20:23
@RonMaupin The odd thing is that there is a route for that interface, or at least one I can see in the route table. It has a default / `0/0` route pointing to its gateway. The VPN app creates higher priority routes (specifically `0/1` and `128/1`) for the VPN gateway so that traffic is generally routed through it, but I don't understand why that would prevent sockets specifically set to use `en0` from having an outbound route. — Bri Bri, Jun 24 '22 at 20:53
As I explained, the message can come from any router through which the packet passes if the router does not have a route to the destination network. A router with no route to a destination network will drop the packet and send an ICMP error with that message. You need to look at the packet with the ICMP error to see the source address of the error. — Ron Maupin, Jun 24 '22 at 21:48
On a side note, this code is leaking the socket. You need to `close()` it when you are done using it. — Remy Lebeau, Jun 24 '22 at 22:02
@RonMaupin It's not the router or local network, it's something on the system running this code. When it's not connectsd to the VPN and the `utun` interfaces don't exist, the connection succeeds when its interface is set to `en0`. Additionally, if I create a route in the route table specifically for the IP address of the hostname passed into this function that points to `en0`, even when the VPN is up the connection works and the function succeeds. — Bri Bri, Jun 25 '22 at 01:06
And you verified that by the source address on the ICMP error message? You should edit the question to include it. — Ron Maupin, Jun 25 '22 at 01:36
@BriBri `shutdown()` disconnects the communication channels inside the socket (ie, sends a `FIN` to the peer and prevents further packets from being exchanged). But it does not release the socket resource itself from memory. You need to use `close()` for that. — Remy Lebeau, Jun 25 '22 at 02:30
@RonMaupin Admittedly I don't have much experience working with ICMP error messages, but as near as I can tell there is no ICMP error message being generated when the socket attempts to connect. At least none that I can see when using `tcpdump` to monitor them -- though I could be using it incorrectly. My suspicion is that it's failing entirely within the system running my code, and no packet ever is sent out during this connection attempt. — Bri Bri, Jun 25 '22 at 05:39
A connection starts with a TCP handshake (SYN, SYN/ACK, and ACK) to open the connection before sending any data. The error is an ICMP error message, either from your IP process, or from an external router. The error is sent in response to the initial packet with the SYN segment. You need to track down the source address of the ICMP error to know for sure where to place the blame. Speculation and guessing is no way to troubleshoot, and the entire purpose of the ICMP error is to tell you specifically where the lack of the network route exists. — Ron Maupin, Jun 25 '22 at 06:48
@RonMaupin Do you know how I get the source address of this ICMP error message? I've been scouring around the internet trying to find information on how to do this, but I've barely found anything useful. The only thing I've found so far is using `IP_RECVERR`, and according to this: https://stackoverflow.com/questions/29931804/osx-equivalent-for-ip-recverr it doesn't exist in macOS. — Bri Bri, Jun 25 '22 at 13:38
@BriBri you are still leaking the socket if any error occurs in your system calls. You need to `close()` the socket before exiting your function, regardless of its outcome. — Remy Lebeau, Jun 25 '22 at 14:54
@BriBri you are not going to be able to get the source address of the ICMP error in your code. Use a packet sniffer instead, such as Wireshark. — Remy Lebeau, Jun 25 '22 at 14:56
@RemyLebeau I tried using Wireshark and, just like with tcpdump, no entry for the connection appears in its log when the connection fails. There are log entries for the connection when it succeeds. (i.e. making the connection using `utun10`, or using `en0` when the VPN is not connected and `utun10` does not exist) — Bri Bri, Jun 25 '22 at 16:19
@BriBri makes sense, since you are not getting a completed connection. But you should be able to see the raw network packets. — Remy Lebeau, Jun 25 '22 at 17:32
@RemyLebeau Would you advise on what I should be looking for? I'm not very familiar with using Wireshark / tcpdump and analyzing raw packets. Should I be looking for something that mentions the IP address of the host my app is trying to connect to? — Bri Bri, Jun 25 '22 at 20:34
@RonMaupin @RemyLebeau I've confirmed that the ICMP error is coming from the OS, and no packet ever leaves the network interface. It doesn't know how to route any traffic through my ethernet interface while the VPN is up. It's like `en0`'s default route has been disabled. I've added more info to my original post, including the system routing table. — Bri Bri, Jun 26 '22 at 05:14
In that routing table, any destination network in the `0.0.0.0/1` range is routed through the `10.16.0.1` gateway. Remember that a default route is the route of last resort. The longest match in a routing table wins, and the length of the default route is `0` (shortest possible, chosen last), but the length of the `0/1` route is `1`, which is longer than the `0` of the default route, so it is chosen over the default route for any destination network that falls in its range. — Ron Maupin, Jun 26 '22 at 05:55
Based on the routing table you added, it would seem that the ICMP error is coming from the router on the other end of `utun10`, not your local OS. You are trying to reach something on the public Internet with an address that falls in the `0.0.0.0/1` range, and such an address will be sent to the `10.16.0.1` gateway. What you are doing is to source your packets from the `en0` interface, but the routing table determines to which gateway packets destined for a different network will be sent. — Ron Maupin, Jun 26 '22 at 06:08

Ron Maupin · Accepted Answer · 2022-06-26T09:53:29.247

Once you added the routing table to your question, your problem became obvious.

It is the routing table that determines to which gateway a packet is sent. The routing table tells the sending host to which gateway the packet is sent. It does that by comparing the destination address to the routes in the routing table. The most-specific (longest match) route is used. A default route is the least-specific (shortest match) route, and it is used as the route of last resort when there are no more-specific routes in the routing table.

Based on the routing table you provided, any packet with a destination address from 1.0.0.0 to 126.255.255.255 (0.0.0.0/8 and 127.0.0.0/8 are exceptions as unusable ranges) will match the 0/1 routing table entry rather than the default route (0/0), and any packet with a destination address from 128.0.0.0 to 223.255.255.255 (224.0.0.0/4 is multicast, and 240.0.0.0/4 is unusable) will match the 128/1 routing table entry rather than the default route (0/0), because the route length of 1 is more specific than the default route length of 0. That means any packets destined to an address in those ranges (combined, all addresses destined for a different network) will be sent to the gateway (10.16.0.1) referenced by the routing table entries for the 0/1 and 128/1 routes.

To solve your problem, you need to remove the 0/1 and 128/1 routing table entries and replace them with one or more entries that are restricted to the networks which the tunnel can reach. With that, the entries not matching the tunnel route(s) or other more specific routing table entries will use the default route.

What you've written is of course all correct, but it still doesn't fully answer my question, because in my situation the packets being sent are bound to a specific network interface. Why is it that a packet bound to `en0` can't be routed when the `0/1` and `128/1` routes are in the table, given that those more specific routes associated with `utun10`? Shouldn't it fall back on the default route in this case? Further, if I create a new interface-specific default route for `en0` (see the edit I just made to my question), it starts to work. Why is that necessary? — Bri Bri, Jun 26 '22 at 13:39
That means you are sourcing the packet from `eno` (making its address the source address on the packets), but they get properly routed according to the routing table to the `10.16.0.1` gateway, which is the source of the ICMP error message. You are no longer discussing programming, but are getting into the standard behavior of the OS, more proper asked on [apple.se] or [su]. — Ron Maupin, Jun 26 '22 at 18:40
Okay, I think I understand now, and yes, you are correct, this is veering away from a programming question so I'll mark this as correct and consider the matter settled! — Bri Bri, Jun 26 '22 at 19:17

How can a socket connect when bound to a specific ethernet interface that's also being used by VPN / utun interface?

1 Answers1