0

For a networking project, I'm using UDP Multicast to build an overlay network with my own implementation of IP.

I use the following to parse and build my Header first, then append the payload:

def __init__(buffer_size_bytes):
    self.__buffer = bytearray(buffer_size_bytes)

def read_sock(self, listening_socket):
    n_bytes, addr = listening_socket.recvfrom_into(self.__buffer, Packet.HEADER_SIZE)
    packet = Packet.parse_header(self.__buffer)

    if packet.payload_length is not 0:
        packet.payload = parse_payload(packet.payload_length, listening_socket)

    self.__router.add_to_route_queue(packet, listening_socket.locator)

def parse_payload(to_read, socket):
    payload = bytearray(to_read)
    view = memoryview(payload)
    while to_read:
        n_bytes, addr = socket.recvfrom_into(view, to_read)
        view = view[n_bytes:]
        to_read -= n_bytes

    return payload

The header seems to be parsed correctly, but the payload gets corrupted every time. I can't figure out what I'm doing wrong when parsing the payload, and I can confirm I'm sending a bytearray from the other side.

For example, when I send a packet with the payload "Hello World" encoded in utf-8, I receive the following:

b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00'

The Packet.parse_header method:

def parse_header(cls, packet_bytes):
    values = struct.unpack(cls.ILNPv6_HEADER_FORMAT, packet_bytes[:cls.HEADER_SIZE])

    flow_label = values[0] & 1048575
    traffic_class = (values[0] >> 20 & 255)
    version = values[0] >> 28
    payload_length = values[1]
    next_header = values[2]
    hop_limit = values[3]
    src = (values[4], values[5])
    dest = (values[6], values[7])

    return Packet(src, dest, next_header, hop_limit, version, traffic_class, flow_label, payload_length)

For reference, the entire sent packet looks like this:

b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01Hello World'

On receiving the first packet, the socket.recvfrom_into blocks when reading for the payload, and doesn't return until I send another message. It then seems to discard the payload of the previous message and use the second packet received as the payload...

Jordan Mackie
  • 2,264
  • 4
  • 25
  • 45
  • 1
    I don't think anyone is going to be able to repro this. Can you show the actual `bytes` before and after corruption? – tripleee Feb 04 '19 at 14:32
  • 1
    Are you sure the header is parsed correctly? The `\x0b` in the string is 11 which is also `len("Hello World")`. – David Feb 04 '19 at 15:08
  • @tripleee I should've looked at the full sent bytes not just the payload! What I'm receiving as the payload on the other end appears to be the start of the actual packet rather than the payload – Jordan Mackie Feb 04 '19 at 15:08
  • @David The header is parsed correctly, but when I read from the socket, it doesn't seem to be "getting rid of" the already read data (the header), so I'm reading it again as the payload. – Jordan Mackie Feb 04 '19 at 15:10
  • @JordanMackie And do you really read from the socket twice? And do you get a fresh destination buffer for the parsing of the payload? – David Feb 04 '19 at 15:13
  • @David Definitely read from the socket twice, definitely get a fresh destination buffer for the parsing of the payload. – Jordan Mackie Feb 04 '19 at 15:17
  • Can you please update the question and provide the code which parses the header? – David Feb 04 '19 at 15:18
  • @David added the header parsing code. It just struct.unpacks from the given byte array. Size of 40 bytes. Also added all of the bytes that are sent. – Jordan Mackie Feb 04 '19 at 15:22
  • Your `parse_payload` takes three args but you call it with two? (At least in the example.) – David Feb 04 '19 at 15:26
  • @David my bad, that was a leftover of me trying something else before asking SO! – Jordan Mackie Feb 04 '19 at 15:32

1 Answers1

0

Found my explanation here.

So the key thing was that I'm using UDP. And UDP sockets discard anything that doesn't fit in the buffer you give it.

TCP sockets however behave more like the bytestream I was expecting.

Fun!

Jordan Mackie
  • 2,264
  • 4
  • 25
  • 45