4

We decided to use UDP to send a lot of data like coordinates between:

  • client [C++] (using poll)
  • server [JAVA] [Apache MINA]

My datagrams are only 512 Bytes max to avoid as possible the fragmentation during the transfer.

Each datagram has a header I added (with an ID inside), so that I can monitor :

  • how many datagrams are received
  • which ones are received

The problem is that we are sending the datagrams too fast. We receive like the first ones and then have a big loss, and then get some, and big loss again. The sequence of ID datagram received is something like [1], [2], [250], [251].....

The problem is happening in local too (using localhost, 1 network card only) I do not care about losing datagrams, but here it is not about simple loss due to network (which I can deal with)

So my questions here are:

  • On client, how can I get the best :
    • settings, or socket settings?
    • way to send as much as I can without being to much?
  • On Server, Apache MINA seems to say that it manage itself the ~"size of the buffer socket"~ but is there still some settings to care about?
  • Is it possible to reach something like 1MB/s knowing that our connection already allow us to have at least this bandwidth when downloading regular files?

Nowadays, when we want to transfer a ~4KB coordinates info, we have to add sleep time so that we are waiting 5 minutes or more to get it to finish, it's a big issue for us knowing that we should send every minute at least 10MB coordinates informations.

Useless
  • 64,155
  • 6
  • 88
  • 132
itMaxence
  • 1,230
  • 16
  • 28
  • 1
    I'd start by installing wireshark to confirm that packets are hitting the box. Also would play with OS UDP buffer size. It sounds suspicious, even if mina is too slow processing them, OS should buffer. – MK. Oct 01 '15 at 13:00
  • 3
    _"to transfer a ~4KB coordinates info, we have to add sleep time so that we are waiting 5 minutes or more"_ That's 1 packet every 40s... It would be faster to switch to telegraph and morse code. – ElderBug Oct 01 '15 at 13:01
  • For example, "on Windows the network stack will drop UDP datagrams (as it's allowed to do) if its buffers get full" - try to increase the buffer size to check. – SChepurin Oct 01 '15 at 13:14
  • 2
    It looks like a congestion problem, so you would need some congestion control, or at least throttle the sending. But the 4KB in 5min make it looks like the problem is much bigger, probably in your code. – ElderBug Oct 01 '15 at 13:14
  • 1
    @SChepurin "My app is eating 1TB on my hard drive when it shouldn't", your answer : "Buy a 2TB hard drive". That's not a solution. – ElderBug Oct 01 '15 at 13:17
  • 1
    @ElderBug - saw "to check" at the end? This is a test, not a solution (what solution without code can be advised?) . – SChepurin Oct 01 '15 at 13:20
  • Just to verify, but this is an **upload** (client-to-server) scenario? The first step should be to get a network trace. Grab WireShark and find out whether the missing packets made it to the wire. After that, you can probably blame one side for >99% of the packet loss. – MSalters Oct 01 '15 at 14:37
  • The longer I think about it, the worse it gets. 4 kB in 5 minutes? That's about 1 packet per 30 seconds. I imagine the only way to be that bad is to (1) do a new name lookup for **every** send, (2) have name lookup configured as DNS-then-etc/hosts, and (3) have the DNS lookup time-out after 30 seconds, on every send. Either that, or something as epically stupid. – MSalters Oct 01 '15 at 14:51
  • Good idea I'll check with Wireshark in first instance. My says about the time to transfert is not precisely what I wrote but it explains at least how slow it can be. I'll keep ou informed. – itMaxence Oct 01 '15 at 16:05

2 Answers2

2

If you want reliable transport, you should use TCP. This will let you send almost as fast as the slower of the network and the client, with no losses.

If you want a highly optimized low-latency transport, which does not need to be reliable, you need UDP. This will let you send exactly as fast as the network can handle, but you can also send faster, or faster than the client can read, and then you'll lose packets.

If you want reliable highly optimized low-latency transport with fine-grained control, you're going to end up implementing a custom subset of TCP on top of UDP. It doesn't sound like you could or should do this.

... how can I get the best settings, or socket settings

Typically by experimentation.

If the reason you're losing packets is because the client is slow, you need to make the client faster. Larger receive buffers only buy a fixed amount of headroom (say to soak up bursts), but if you're systematically slower any sanely-sized buffer will fill up eventually.

Note however that this only cures excessive or avoidable drops. The various network stack layers (even without leaving a single box) are allowed to drop packets even if your client can keep up, so you still can't treat it as reliable without custom retransmit logic (and we're back to implementing TCP).

... way to send as much as I can without being to much?

You need some kind of ack/nack/back-pressure/throttling/congestion/whatever message from the receiver back to the source. This is exactly the kind of thing TCP gives you for free, and which is relatively tricky to implement well yourself.

Is it possible to reach something like 1MB/s ...

I just saw 8MB/s using scp over loopback, so I would say yes. That uses TCP and apparently chose AES128 to encrypt and decrypt the file on the fly - it should be trivial to get equivalent performance if you're just sending plaintext.

Useless
  • 64,155
  • 6
  • 88
  • 132
  • I'll make some experimentation. But my goal today is to transfer big quantity of data, as fast as I can, without taking care about if there is 1% or 30% of loss, but right now my problem is just to make UDP working fine. I'll investigate more and keep you all informed. Thanks – itMaxence Oct 01 '15 at 16:10
-1

UDP is only a viable choice when any number of datagrams can be lost without sacrificing QoS. I am not familiar with Apache MINA, but the scenario described resembles the server which handles every datagram sequentially. In this case all datagrams arrived while the one is serviced will be lost - there is no queuing of UDP datagrams. Like I said, I do not know if MINA can be tuned for parallel datagram processing, but if it can't, it is simply wrong choice of tools.

SergeyA
  • 61,605
  • 5
  • 78
  • 137
  • I am relatively sure that you are wrong about this. OS will queue received UDP packets. – MK. Oct 01 '15 at 13:08
  • Since an UDP datagram can be up to 64KB, the system buffers are almost guaranteed to hold 128 packets of 512 bytes. Even with a badly coded app, it would be hard to drop packets that way. – ElderBug Oct 01 '15 at 13:09
  • This is what I've seen with my own eyes, so I stand by it. – SergeyA Oct 01 '15 at 13:18
  • 1
    *NIX will typically queue packets into a fixed size buffer _and then silently drop packets that don't fit in the buffer_. There is no back-pressure or throttling on the sender. So @MK. is right that packets are queued, and SergeyA is right that packets are dropped. I have no idea what other OSs may do. – Useless Oct 01 '15 at 13:42
  • @Useless, i would amend that it is not the *NIX, but the stack which does this. However, I would also admit that my experience in the matter might be with some exotic stacks. – SergeyA Oct 01 '15 at 13:50
  • As written, it's pure nonsense. Processing of a **single** UDP packet causes the loss of all further packets?! Processing never causes that. A buffer overflow would cause that, but since Linux 2.6 (ages ago) these buffers are 4 MB = 8192 packets of 512 bytes. – MSalters Oct 01 '15 at 14:43
  • Linux. The only system we know of. – SergeyA Oct 01 '15 at 14:51