2

I am writing an application which sends and receives packages using UDP. However, the documentation of recv_from states:

If a message is too long to fit in the supplied buffer, excess bytes may be discarded.

Is there any way to receive all bytes and write them into a vector? Do I really have to allocate an array with the maximum packet length (which, as far as I know, is 65,507 bytes for IPv4) in order to be sure to receive all data? That seems a bit much for me.

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
Lukor
  • 1,485
  • 2
  • 14
  • 25
  • This is a problem for C as well: http://www.microhowto.info/howto/listen_for_and_receive_udp_datagrams_in_c.html – Neikos Jan 17 '18 at 08:44
  • Temporarily getting 65k of uninitialized memory from stack shouldn't be a big deal. – ArtemGr Jan 17 '18 at 09:01
  • @ArtemGr it probably isn't, but why would I allocate that much if I could just grow only if I need the space? because most packets are probably only a few bytes long – Lukor Jan 17 '18 at 12:27
  • 1
    @Lukor Because extra system calls are much more expensive than getting some space from the stack. – ArtemGr Jan 17 '18 at 12:34
  • @ArtemGr okay, I guess that makes sense, thank you – Lukor Jan 17 '18 at 12:36

1 Answers1

2

Check out the next method in the docs, UdpSocket::peek_from (emphasis mine):

Receives a single datagram message on the socket, without removing it from the queue.

You can use this method to read a known fixed amount of data, such as a header which contains the length of the entire packet. You can use crates like byteorder to decode the appropriate part of the header, use that to allocate exactly the right amount of space, then call recv_from.

This does require that the protocol you are implementing always provides that total size information at a known location.

Now, is this a good idea?

As ArtemGr states:

Because extra system calls are much more expensive than getting some space from the stack.

And from the linked question:

Obviously at some point you will start wondering if doubling the number of system calls to save memory is worth it. I think it isn't.

With the recent Spectre / Meltdown events, now's a pretty good time to be be reminded to avoid extra syscalls.

You could, as suggested, just allocate a "big enough" array ahead of time. You'll need to track how many bytes you've actually read vs allocated though. I recommend something like arrayvec to make it easier.

You could instead implement a pool of pre-allocated buffers on the heap. When you read from the socket, you use a buffer or create a new one. When you are done with the buffer, you put it back in the pool for reuse. That way, you incur the memory allocation once and are only passing around small Vecs on the stack.

See also:

Shepmaster
  • 388,571
  • 95
  • 1,107
  • 1,366
  • I would use the word *pool* instead of *magazine* here; is there a specific connotation to magazine I am not aware of that would make it a better fit? Were you thinking about a machine gun maybe? – Matthieu M. Jan 17 '18 at 14:59
  • @MatthieuM. changed. Magazine is kind of an obscure term I have stuck in my head for whatever reason, but I didn't make it up. See [*Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources*](http://www.parrot.org/sites/www.parrot.org/files/vmem.pdf) and [*Solaris Internals: Core Kernel Components*](https://books.google.com/books?id=r_cecYD4AKkC&pg=PA226&lpg=PA226&dq=memory+magazine+layer&source=bl&ots=oEpjeYtAAQ&sig=4NV4OoA5kr9skdyppUSFgNFiB9w&hl=en&sa=X&ved=0ahUKEwiw07KNot_YAhVEUKwKHSr7C18Q6AEIVTAK#v=onepage&q=memory%20magazine%20layer&f=false). – Shepmaster Jan 17 '18 at 15:04
  • The "leading 4 byes containing the length of the entire packet"... There is no such information in the leading 4 bytes. – doliphin Dec 11 '22 at 19:01
  • @doliphin yes, which is why the full quote says "**such as** the leading 4 bytes". This is completely up to whatever protocol you are implementing. Some protocols provide information in a fixed size header, others have magic sequences that you read until, etc. – Shepmaster Dec 21 '22 at 18:42
  • @Shepmaster but in this case, OP is writing an application that 'sends and receives' using `std::net::UdpSocket`. `send` and `send_to` just send whatever bytes you specify. It is not even guaranteed that there will be 4 bytes available to be received! As someone who was also looking for an answer to this question, your response mislead me and resulted in me implementing buggy code, I think it would be helpful if you specified that this is **not** what Rust's `UdpSocket` does and you have to implement this **yourself**. Perhaps change this to 'a 4-byte header' instead of 'the leading 4 bytes' – doliphin Dec 21 '22 at 19:57