Read data from a UDP socket with an unknown length

Question

How could I socket.ReadFromUDP without knowing the length beforehand? Right now I have something like this.

buff := make([]byte, <unknownLength>)
for {
    socket.ReadFromUDP(buff)
    fmt.Println("Bytes read:", buff.Len())
}

I tried this, but it always read 1024 bytes.

buff := make([]byte, 1024)
for {
    bytesRead, _, _ := socket.ReadFromUDP(buff)
    fmt.Println("Bytes read:", bytesRead)
}

The packets I'm receiving from the server vary from a couple of bytes to hundres of bytes.

So I've seen in node.js, people catch an `onmessage` event, which contains the packet sent from a server. Then, you can do a `message.length` and see how many bytes were received. — 425nesp, Jul 11 '14 at 19:36
sorry, totally missed that this was a UDP question. That requires a little more explanation, since UDP isn't a stream. — JimB, Jul 11 '14 at 20:07

score 12 · Accepted Answer · edited May 23 '17 at 10:27

12

A UDP socket receives messages in discrete datagrams (hence the name SOCK_DGRAM). Though Go has a relatively clean networking API, you still can't get away without knowing a little bit about the underlying sockets (How large should my recv buffer be when calling recv in the socket library).

A UDP socket will Read up to the size of the receiving buffer and will discard the rest, so you can't continue to read like you would on a TCP stream. A single receive on a UDP socket consumes a single datagram from the network.

Though the theoretical max payload of a UDP packet is 65,507, in practice you are probably not going to get messages over 1400 bytes because of path MTU limits. If you set your buffer considerably larger than this, to something like 4096 bytes you will be very safe, but you could use a 64k buffer to be certain.

Though it's not feasible in Go to try and determine the pending packet size, you can tell if you lost data for what it's worth:

oob := make([]byte, 1024)
n, on, flags, addr, err := c.ReadMsgUDP(buff, oob)
if flags & syscall.MSG_TRUNC != 0 {
    fmt.Println("truncated read")
}

edited May 23 '17 at 10:27

Community

1
1

answered Jul 11 '14 at 20:13

JimB

104,193
13
262
255

In environments that expose direct access to the underlying socket API, like C/C++, there are ways to query the socket for how much pending bytes it has, then you can allocate a buffer accordingly before reading into it. But it does not appear that Go has such a query, so what Jim said is about the closest you can get. Over-allocate, and pay attention to the return value of ReadFromUDP() as it tells you how many bytes where actually read. – Remy Lebeau Jul 11 '14 at 23:29
@RemyLebeau: good points. You can theoretically use `syscall.Syscall(syscall.SYS_IOCTL, fd.Fd(),...`, (though `FIONREAD` isn't in the syscall package) but since Go doesn't have an equivalent to select/epoll to check for pending reads, it's not of much use when you'll often be calling Read without any packets in the buffer. – JimB Jul 12 '14 at 01:39
`syscall.MSG_TRUNC` does not appear to exist on windows. Any other ideas there? – captncraig Nov 06 '15 at 16:12
1

@captncraig: though I mentioned it for completeness, I don't think there's actually much value in checking for a short read here, since UDP is lossy to begin with and you could have just as well lost a packet fragment. Just use a buffer large enough to read your expected datagram size. (You could also use a slightly larger buffer than needed, and return an error if your buffer is filled, which would indicate a bad message and probably discarded data) – JimB Nov 06 '15 at 16:28

score 4 · Answer 2 · answered Jul 12 '14 at 03:36

The basic technique is to use a buffer one larger than the largest expected datagram. Then if you get a datagram that size you know it doesn't fit your application protocol and should be discarded, probably after logging it in case it's a programming error at the sender, or your application protocol is incompletely specified.

Read data from a UDP socket with an unknown length

2 Answers2