4

I'm working with a piece of hardware that generates a data stream packaged in UDP packets. These are sent out on a dedicated 40Gb Ethernet link to another piece of receiver hardware. There are no hubs or switches involved, just a single sender and a single receiver.

We have recently disconnected the receiver hardware and plugged that end into a commodity Linux workstation in order to receive the data stream with software. I can crank up the sender hardware and with Wireshark running on the receiving workstation I can see exactly the data I am supposed to see. However, if I crank up a separate application on the receiver that does UDP recvfrom(), it can't see the UDP packets that Wireshark can see, it just gets recvfrom() timeouts. I bind to the exact same IP address and port that Wireshark reports as the destination for the packets. The IP address is a 192.168 private subnet, and the port is in the 4000's, so it is "reserved" but not "well-known". Setting the IP address to INADDR_ANY doesn't make any difference. Whether I have Wireshark running when I try to do the UDP receive doesn't make any difference.

The software that is trying to receive works if the sender is another software application (admittedly sending to some other NIC on the workstation). I check the return status for every call (socket(), setsockopt(), bind(), recvfrom()) and they all claim success.

The NIC that is receiving the data is a Mellanox ConnectX-4 (I believe) that is set to 40Gb Ethernet mode. The MTU is set to 9000, although the packets in the data stream are much smaller (100-200 bytes).

What conditions would make a packet stream visible to Wireshark but not to other application software? Might there be a NIC setting that is not right? Could there be flags in the Ethernet/IP/UDP headers that are questionable and don't make it up the stack? Where should I look next?

MarkT
  • 71
  • 3
  • Dig through all of the collapsed header fields on one of these packets in Wireshark - if there's something invalid, I would expect to see an indication of this somewhere in there. Another possibility is that the NIC isn't configured with an address/netmask that is in the same range as the device's destination address - in that case, binding to the address will choose a different interface. – jasonharper Nov 04 '21 at 04:06
  • Opening up all the headers didn't show any red flags (but I'm no expert). The IP address I am using is the one that ip addr shows for the NIC that Wireshark is listening to. However, I noticed something I hadn't thought to look at before. The MAC address that the sender is targeting as destination IS NOT the MAC address of the NIC. Does it make sense that Wireshark can still see the traffic? It certainly makes sense that the network stack wouldn't accept the packets. I'm guessing the sender is not ARPing but is using a hard-coded MAC address. – MarkT Nov 04 '21 at 13:54
  • 1
    Yes, if the sending device hard-codes a MAC address rather than using ARP, that would certainly explain the problem - and in that case, it's fixable via MAC spoofing. (Wireshark doesn't care, it would be of rather limited use if it could only see traffic directed to your machine.) – jasonharper Nov 04 '21 at 14:10
  • 1
    The wrong MAC address was the key problem (though there were additional problems that were relatively easy to fix). Turns out it was possible to give the sending hardware a different MAC address to use. I'll submit an answer to sum this up. – MarkT Nov 04 '21 at 18:58
  • Were you setting the device to promiscuous mode? That's very likely the reason why Wireshark "sees" the data and your process doesn't – Braiam Nov 04 '21 at 19:47

1 Answers1

3

As noted in the comments above, the fundamental problem was that the sending hardware was using a destination MAC address that was not the one for the receiving NIC. Turns out it, as well as the IP address, were hard-coded in the sending hardware. And the hardware was not capable of using ARP to resolve. Discovering this involved cross-checking the output of "ip addr show" with what Wireshark was reporting. Fortunately it was possible to program a new MAC address for the sender to use. The lesson here, if any, is assume nothing, check everything.

MarkT
  • 71
  • 3