Exception thrown: 'System.OutOfMemoryException' in ZeroMQ.dll

Question

I am trying to transfer a huge array of bytes (size: 12 MegaBytes) using ZeroMQ in Windows 10 64 Bit OS. The rate of transfer is 15 Frame Per Second approximately.

The code is throwing following error:

Exception thrown: 'System.OutOfMemoryException' in ZeroMQ.dll

During debugging using Visual Studio, it shows 4GB process memory as shown in the screenshot below:

However, when I checked the memory consumption in OS, it shows 7.1GB out of 8GB. But 801MB is still unused, which makes me wonder about System.OutOfMemoryException exception.

See below the screenshot:

The computer is equipped with Gigabit Ethernet card. See below the card usage:

Please see below the code:

ZContext zmqContext;
ZSocket publisher;
Stopwatch watch;
readonly byte[] longByteArray = new byte[12000000];

Program()
{
    zmqContext = new ZContext();
    publisher = new ZSocket(zmqContext, ZSocketType.PUB);
    publisher.Bind("tcp://*:11111");

    var timer = new Timer();
    timer.Elapsed += new ElapsedEventHandler(OnTimedEvent);
    timer.Interval = 1000.0 / 15;//15 FPS
    timer.Enabled = true;
}

private void OnTimedEvent(object source, ElapsedEventArgs e)
{
    var frame = new ZFrame(longByteArray);
    publisher.Send(frame);
    frame.Dispose();
}

Below is my observation:

1. Although, it is not possible to change the FPS in real code, however, the above code snippet is sample code and in this code, I changed FPS from 15 to 10. The error still pops up after some time. Initially, the memory consumption was less but it keeps on increasing as time passes.

My questions are following:

1. As per my observation above it seems that ZeroMQ publisher is keeping messages in a queue, which is eating a lot of memory. I am wondering if it is possible to configure ZeroMQ to keep only one message (last i.e. newest message) in the queue so that ZeroMQ can discard other messages resulting less memory consumption.
2. Why is the exception being thrown, even though 801MB memory is still available?
3. Is there any other workaround to make the code work?

Answer 1

Ad 2 ) Why is the exception ... though 801MB memory is still available?

Better do not rely on static figures. Each JUMBO-message ( as depicted above ) has to be stored in several places, some of which are outside of the ZeroMQ domain of control ( typically the O/S & driver buffers ). Sure, ZeroMQ has done a great job on zero-copy, where possible, but injecting data onto network is typically an O/S job, so any free-space is "evaporating" pretty fast on Mammoth-sized data-blocks ( did I say, that ZeroMQ delivers either the whole, whatever sized data en-block, or nothing at all? ).

Next, there are Context-class instance settings ( High Water Marks and other ) that explicitly warn about not relying on having XYZ [B] free and available for message-data if having set that very value of XYZ [B] -- due to whatever internal buffer-management tricks, the API documentation and comments promote to expect anything under 2/3, if not 1/2 of that setting to remain actually available for any message-related data ( not mentioning, what would happen with buffer-space "consumption" if there would happen to be more than a single .connect()-ed counterparties

( 1-sends : {2,3,4,...}-receive )

---

Ad 1 ) ... is it possible to ... keep only one message _{( last )}?

This one has a clean solution. It is typical not only for fast video-streaming, but in general, cool for any process-monitoring, non-rigid sensorics networks or even audio-streaming, where target processing does not need to "hear" each and every fragment of the data-flow, but will appreciate to keep surfing on the "front-wave" of the TimeDOMAIN evolution ( getting rather the most recent data, while not mourning on not receiving the previous "old" readings ):

Use the .setsockopt( ZMQ_CONFLATE, 1 ); method and you are done!

---

Ad 3 )

If indeed in a need to go for an utmost performance ( and if end-to-end latency permits ) try the effects of a data-compression step ( moving "less" bytes, at a tradeoff of an increased local/remote processing workloads ) and may experiment on moving a set of "independently" processed/transferred "split" datablocks, where multiple Context( <_nIOthreads_> ) instance IO-threads may provide for almost parallel delivery of smaller blocks over multiple Socket-instances ( IO-threads may get load-balanced on both local/remote sides,
again via using the .setsockopt( ZMQ_AFFINITY, <_IOthreadORD#_> )-method

Last, but not least, I would also do my best to control .setsockopt( ZMQ_TOS, <_TOS#_> ) for TypeOfService End-to-End control, if latency-jitter becomes critical for QoS.