Why am I receiving packer bigger than with raw packet

Question

I am trying to transfers a packet from an interface to another by using raw packets (just for playing). First I focused on received packets.

On my machine (archlinux, that has 192.168.30.3 as IP) I created this code:

#include <stdio.h>
#include <net/ethernet.h>       /* the L2 protocols */
#include <netinet/ip.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>

int main()
{
    int packet_socket = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_IP));


    /* test reception */
    char packet[4096];
    struct sockaddr rcvaddr;
    struct in_addr addr;
    addr.s_addr = inet_addr("192.168.30.3");    //my ip

    // use nc to send a use packet
    while (1) {
        int len = sizeof(rcvaddr);
        int len_packet =
            recvfrom(packet_socket, packet, 4096, 0, &rcvaddr, &len);

        // check if the packet is for us
        struct iphdr *iph =
            (struct iphdr *) (packet + sizeof(struct ethhdr));
        if (iph->daddr != inet_addr("192.168.30.3"))
            continue;

        // check if tcp
        if (iph->protocol != IPPROTO_TCP)
            continue;

        printf("Total packet length: %d\n",
               sizeof(struct ethhdr) + ntohs(iph->tot_len));
    }
}

Then I run it as root and also execute nc -lp 12345 -n > /dev/null.

On another machine (debian, 192.168.30.4) I run dd if=/dev/urandom | nc 192.168.30.3 12345 which makes my previous program prints the length of the received packets.

From it, I see there is packets that are greater that the size of MTU (which is 1500 on the two machines). For instance I can read "Total packet length: 16962" from my program. ( Also observed by linux raw ethernet socket receive more bytes than MTU).

I know about IP fragmentation so I first thought about IP reassembling. However I read in man 7 raw: "Note that packet sockets don't reassemble IP fragments, unlike raw sockets." Because I used packet sockets (AF_PACKET) I should not have packet reassembling and then keep the MTU size right?

I also did sudo ethtool -K ens3 tx off sg off tso off and test with the value 0,1,2 and 3 in /proc/sys/net/ipv4/ip_no_pmtu_disc on both machines.

Do you think 192.168.30.4 is sending more the MTU? Or does my machine performs some reassembling despite what is written in the manual?

ethtool -k ens3 gives:

On 192.168.30.4:

seb@SERVER:~$ sudo ethtool -k ens3 
Features for ens3:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

On 192.168.30.3:

[seb@archlinux ~]$ sudo ethtool -k ens3
Features for ens3:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

Also, note that the two machines are qemu machine run by GNS3 with the following net options: -net none -device e1000,mac=0c:7e:08:49:13:00,netdev=gns3-0 -netdev socket,id=gns3-0,udp=127.0.0.1:20049,localaddr=127.0.0.1:20048

Unfortunately turning off lro gives the same results. I just edit my question to add the information you want. Since the two machines are VMs, I will try on real machines (maybe the hypervisor creates a large buffer) — sebastien dontneedtoknowthat, Aug 22 '21 at 22:56
My Guess: you are ignoring the options. Start reading these RFCs... — wildplasser, Aug 22 '21 at 23:07
@wildplasser: Can you elaborate? what are the 'options' your are talking about? what RFC? — sebastien dontneedtoknowthat, Aug 22 '21 at 23:19
@stackinside : I disabled generic-receive-offload. This gives better results. I am doing more experiments before confirming it. — sebastien dontneedtoknowthat, Aug 22 '21 at 23:21
@stackinside: I confirm that disabling GRO solves my problem (maybe not the good thing to do for performance reasons but it answer my question). Your first comment obviously gives me the good way to think, so yes you can provide the LRO/GRO suggestion as an answer. — sebastien dontneedtoknowthat, Aug 23 '21 at 08:21

score 1 · Accepted Answer · answered Aug 23 '21 at 08:52

Since the observed total packet length is way greater than that of a typical jumbo frame (MTU 9k), it's apparent that the receiver side employs either Large Receive Offload (LRO) or Generic Receive Offload (GRO) thus reassembling smaller packets into larger ones on the network interface driver level. This might explain why the packet socket in question sees already reassembled (large) packets.

In this specific case, ethtool -k output indicates clearly that LRO is always disabled whilst GRO is indeed active and can be adjusted. As per the discussion in comments, disabling GRO indeed bears fruit.

Why am I receiving packer bigger than with raw packet

1 Answers1

Linked