3

I'm ultimately trying to test a UDP client, and want to make sure that it works when sending data not through the loopback interface, to avoid any subtle issues this introduces, such as differences in checksum validation (Bad UDP checksum has no effect: why?).

However, even when sending data to the result of socket.gethostbyname(socket.gethostname()), which is not 127.0.0.1, then according to Wireshark, the data seems to go via the loopback interface.

The below program sends and receives b'somedata' successfully, and has the below capture from Wireshark.

import asyncio
import socket

async def server():
    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
        sock.setblocking(False)
        sock.bind(('', 4567))
        data = await loop.sock_recv(sock, 512)
        print('Received', data)

async def main():
    local_ip = socket.gethostbyname(socket.gethostname())
    print('Local IP', local_ip)  # Outputs 192.168.0.34

    asyncio.ensure_future(server())
    await asyncio.sleep(0)

    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
        sock.setblocking(False)
        sock.connect((local_ip, 4567))
        await loop.sock_sendall(sock, b'somedata')
        await asyncio.sleep(1)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

enter image description here

How can I send data from a client running locally, to a server running locally, but avoiding the loopback interface and actually sending data out into the network?

Ideally answers would be applicable to both Linux and macOS.

Michal Charemza
  • 25,940
  • 14
  • 98
  • 165
  • May I ask what you are trying to achieve? Are you testing _your_ code, or are you testing the network interface? If you want to test how your code handles bad packets, then maybe you should create a method that works on a socket, so you could mock the socket object in a unit test. If you are creating a production test / integration test / system test, then the test setup should mimic the real (expected) setup as close as possible. Ideally you would have a real server and client separated from each other. Maybe you can achieve this using virtual machines, or using two USB-to-Ethernet adapters? – wovano Apr 29 '19 at 14:03
  • @wovano I'm testing my code, _and_ how it interacts with the rest of the system, avoiding assumptions that mocking introduces, so it's more of an integration-style test... "then the test setup should mimic the real (expected) setup as close as possible." Agreed, and seeing how far I can get towards that without the overhead of multiple machines/adapters. – Michal Charemza Apr 29 '19 at 21:42
  • I certainly understand that you want to avoid that overhead. But I don't think you can test how your application interacts with the rest of the system without having the rest of the system ;-) To realistically test the network interface, you'll need at least two network interfaces to guarantee that the packets are being transmitted over the line. And even then the behavior without switch can be very different than with a switch (e.g. due to packet reordering, IP fragmentation, etc.). And a Linux system might have subtle differences than a Windows system, for example. – wovano Apr 29 '19 at 22:22
  • What you _can_ do is test the normal (expected) behavior, and all error conditions you can think of (such as a bad UDP checksum). If the interface boundaries of your application are chosen correctly, you'll probably be able to test most of the situations. Can you describe _what_ you want to test, i.e. which test cases? That might help to provide better solutions. And what assumptions do you think mocking will introduce? – wovano Apr 29 '19 at 22:23
  • "Can you describe what you want to test". For example, how a packet arriving that has a bad UDP checksum affects the application. Is it ignored? Processed as usual? An exception raised? "what assumptions do you think mocking will introduce": that the lower level behaviour actually acts like the mock. Understood that different systems have subtle differences, and this is exactly why I want the tests to use the lower-level behaviour of the system to ensure that fewer assumptions are made about it, and that the tests passing means there is a good chance that the application works. – Michal Charemza Apr 30 '19 at 10:26
  • What do you expect that your application does with a corrupt packet (bad UDP checksum)? Did you write any code that verifies the checksum? I think it is handled by the Ethernet stack of your operating system, so your application will notice the effect of that: either a missing packet (if the checksum is verified and the packet is dropped), or a corrupt packet (if the checksum is not verified by the O.S.). You could write tests for both situations. Likewise, you could test for out-of-order packets, etc.. These scenarious are very difficult to reproduce in a real test scenario anyway. – wovano Apr 30 '19 at 10:35
  • @wovano "What do you expect that your application does with a corrupt packet (bad UDP checksum)". Right now, I would like it to be ignored and the application not affected. – Michal Charemza Apr 30 '19 at 10:47

2 Answers2

1

To 'convince' the networking stack to physically transmit the frame using the Ethernet (or WiFi) card rather than the loopback, use a broadcast address.

I have successfully sent and received an UDP packet this way on my Linux system. I verified it with tcpdump. It shows one packet on Ethernet interface and no activity on loopback.

I have used literal broadcast address. The socket module documentation mentions also the string '<broadcast>' as a special case address. I did not tried it.

with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
    sock.setblocking(False)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
    sock.connect(('192.168.0.255', 4567))
    await loop.sock_sendall(sock, b'somedata')
    await asyncio.sleep(1)

Notes:

  1. other hosts on the same network will receive the UDP packet as well.
  2. make sure the firewall/packet filter (e.g. Linux iptables/nftables) will not block the packet.
  3. regarding the setsockopt: Python socket.error: [Errno 13] Permission denied
VPfB
  • 14,927
  • 6
  • 41
  • 75
  • Interesting... never considered this. For my case however, while making the tests _more_ like the real case in terms of going over the network, it’s also _less_ like the real case since broadcast isn’t used – Michal Charemza Apr 25 '19 at 11:03
  • I have found a related topic: https://serverfault.com/q/483209 I tried 2 IP addresses on a single Ethernet card and was able to transmit the datagram from addr1 to addr2, but it was not received back from the switch. – VPfB Apr 26 '19 at 10:20
0

That's probably because your hostname is pointing to the loopback address hence socket.gethostbyname(socket.gethostname()) will yield 127.0.0.1

What you need to do is cancel that pointing from the hostname to the loopback address:

  • in Linux edit the /etc/hosts and comment out the line 127.0.0.1 YOUR_HOSTNAME
  • in Windows you should have c:\windows\system32\drivers\etc\hosts which looks similar to the Linux one

After this if you call the socket.gethostbyname(socket.gethostname()) it will yield your DHCP assigned IP.

Though even in this case, calling from yourip to yourip might result in the network driver to route the package through the loopback interface. Alternative would be to use the public IP, outside your network router. You could use an external service as described in this answer

andreihondrari
  • 5,743
  • 5
  • 30
  • 59
  • Ah I don't think that's what's happening here `socket.gethostbyname(socket.gethostname())` does not yield 127.0.0.1. I have edited the question to make this more explicit. – Michal Charemza Apr 22 '19 at 09:13
  • The Wireshark dump shows conclusively that the packets are not addressed to or from 127.0.0.1. – user207421 Apr 24 '19 at 06:55
  • @user207421 I think I might have missed that detail the fist time. Nevertheless it would still go through the `lo` interface even like this. – andreihondrari Apr 24 '19 at 08:15