2

When I send a data from a socket and then immediately close it (without changing the default settings), small data sizes are sent successfully but large ones (> 2MB) are not.

Here is how I set up the receiver (blocking on the recv() at the end):

In [1]: import zmq

In [2]: ctx = zmq.Context()

In [3]: socket = ctx.socket(zmq.PULL)

In [4]: socket.connect("ipc://@foo")

In [5]: msg = socket.recv()

And then the sender:

In [1]: import zmq

In [2]: ctx = zmq.Context()

In [3]: socket = ctx.socket(zmq.PUSH)

In [4]: socket.bind("ipc://@foo")

In [5]: arr = bytearray([1]*100)

In [6]: len(arr)
Out[6]: 100

In [7]: socket.send(arr); socket.close()

The receiver gets the data just fine:

In [5]: msg = socket.recv()

In [6]: len(msg)
Out[6]: 100

If however I use a larger message, say arr = bytearray([1]*int(2e6)), then the receiver keeps blocking, waiting for data.

Changing the LINGER settings don't seem to make any difference (and I believe default to infinite wait anyway).

Adding a sleep(1) between sending and closing on the sender like socket.send(arr); time.sleep(1); socket.close() solves the problem: the receiver gets the data correctly.

Why is this not working without the sleep if LINGER defaults to -1? What is the correct way to send and then immediately close a socket when dealing with larger amounts of data?

capitalistcuttle
  • 1,709
  • 2
  • 20
  • 28
  • 3
    See example `flushSocketBeforeClose()` in the answer http://stackoverflow.com/a/12730776/318716 – Joseph Quinsey Jan 24 '14 at 17:46
  • @JosephQuinsey Very interesting, thank you. It seems I can use zmq.Poller (analagous to your select) to loop with a small timeout till the result is an empty list. Going to try a few combinations right now. – capitalistcuttle Jan 24 '14 at 18:48
  • The main problem (as you've likely gathered) is that Zeromq does all network calls asynchronously. Your likely closing the socket before the whole message is sent. – JSON Feb 17 '14 at 10:13

1 Answers1

0

It hangs because it didn't receive any message (you closed the pusher too soon). If you open a new PUSH socket and send a short message, you'll see the PULL receiver is working well and still listening for new messages.

In the ZeroMQ FAQs, you can read:

How can I flush all messages that are in the ZeroMQ socket queue?

There is no explicit command for flushing a specific message or all messages from the message queue. You may set ZMQ_LINGER to 0 and close the socket to discard any unsent messages.

So, basically, if you want to make sure the message was sent, then you'll need to implement a synchronization method between the nodes (i.e.: using REQ-REP).

Peque
  • 13,638
  • 11
  • 69
  • 105