5

I am receiving ~3000 UDP packets per second, each of them having a size of ~200bytes. I wrote a java application which listens to those UDP packets and just writes the data to a file. Then the server sends 15000 messages with previously specified rate. After writing to the file it contains only ~3500 messages. Using wireshark I confirmed that all 15000 messages were received by my network interface. After that I tried changing the buffer size of the socket (which was initially 8496bytes):

(java.net.MulticastSocket)socket.setReceiveBufferSize(32*1024);

That change increased the number of messages saved to ~8000. I kept increasing the buffer size up to 1MB. After that, number of messages saved reached ~14400. Increasing buffer size to larger values wouldn't increase the number of messages saved. I think I have reached the maximum allowed buffer size. Still, I need to capture all 15000 messages which were received by my network interface.

Any help would be appreciated. Thanks in advance.

bezmax
  • 25,562
  • 10
  • 53
  • 84
nath
  • 2,848
  • 12
  • 45
  • 75

5 Answers5

5

Smells like a bug, most likely in your code. If the UDP packets are delivered over the network, they will be queued for delivery locally, as you've seen in Wireshark. Perhaps your program just isn't making timely progress on reading from its socket - is there a dedicated thread for this task?

You might be able to make some headway by detecting which packets are being lost by your program. If all the packets lost are early ones, perhaps the data is being sent before the program is waiting to receive them. If they're all later, perhaps it exits too soon. If they are at regular intervals there may be some trouble in your code which loops receiving packets. etc.

In any case you seem exceptionally anxious about lost packets. By design UDP is not a reliable transport. If the loss of these multicast packets is a problem for your system (rather than just a mystery that you'd like to solve for performance reasons) then the system design is wrong.

tialaramex
  • 3,761
  • 1
  • 21
  • 23
  • yes there is dedicated thread to for this task. And I have lose all latter messages. As your explanation I should able to capture all the messages which has received to my network interface? Am I correct?? – nath Nov 25 '11 at 10:30
  • 1
    Well, it all depends on your platform, I expect. Certainly on Windows the network stack will drop UDP datagrams (as it's allowed to do) if its buffers get full - so you'll see exactly what you're experiencing (wireshark shows them all getting to the NIC and your program doesn't get all of them because the stack finds that the recv buffer is full and drops some). You can tune how it drops them with the SIO_ENABLE_CIRCULAR_QUEUEING ioctl, see http://msdn.microsoft.com/en-us/library/windows/desktop/ms741621(v=vs.85).aspx - In summary, read the datagrams faster. – Len Holgate Nov 25 '11 at 14:10
1

The problem you appear to be having is that you get delay writing to a file. I would read all the data into memory before writing to the file (or writing to a file in another thread)

However, there is no way to ensure 100% of packet are received with UDP without the ability to ask for packets to be sent again (something TCP does for you)

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • 1
    This sounds like the right solution. Basically if you want a larger buffer than the socket library allows, you will have to create the buffer within your own application, between the reading-from-the-network thread and the writing-to-the-disk thread. – caf Nov 25 '11 at 11:03
1

I see that you are using UDP to send the file contents. In UDP the order of packets is not assured. If you not worried about the order, you put all the packets in a queue and have another thread process the queue and write the contents to file. By this the socket reader thread is not blocked because of file operations.

avis
  • 599
  • 1
  • 6
  • 18
0

The receive buffer size is configured at OS level.

For example on Linux system, sysctl -w net.core.rmem_max=26214400 as in this article https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html

woodings
  • 7,503
  • 5
  • 34
  • 52
0

This is a Windows only answer, but the following changes in the Network Controller Card properties made a DRAMATIC difference in packet loss for our use-case.

We are consuming around 200 Mbps of UDP data and were experiencing substantial packet loss under moderate server load.

The network card in use is an Asus ROG Aerion 10G card, but I would expect most high-end network controller cards to expose similar properties. You can access them via Device Manager->Network card->Right-Click->Properties->Advanced Options.

1. Increase number of Receive Buffers:

Default value was 512; we could increase it up to 1024. In our case, higher settings were accepted, but the network card becomes disabled once we exceed 1024. Having a larger number of available buffers at the network-card level gives the system more tolerance to latency in transferring data from the network card buffers to the socket buffers where our apps finally can read the data.

2. Set Interrupt Moderation Rate to 'Off':

If I understood correctly, interrupt moderation coalesces multiple "buffer fill" notifications (via interrupts) into a single notification. So, the CPU will be interrupted less-often and fetch multiple buffers during each interrupt. This reduces CPU usage, but increases the chance a ready buffer is overwritten before being fetched, in case the interrupt is serviced late.

Additionally, we increased the socket buffer size (as the OP already did) and also enabled Circular Buffering at the socket level, as suggested by Len Holgate in a comment, this should also increase tolerance to latency in processing the socket buffers.

BlueStrat
  • 2,202
  • 17
  • 27