How to run a zmq and other while True: at the same time independently?

Question

I have this code for my server:

import time
import zmq

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5000")

while True:
    message = socket.recv()
    socket.send(b"World")
    print "sent"

while True:
    print "done."

I have a separate client script that sends a message through zmq to this one whenever i send a message. On the server (this code), if i only have the first while True:, it prints "sent" every time i send a message, and if i only have the second while True:, it prints "done." continuously. But if i put both, it never prints done (or if i switch their order and put both it never prints "sent" when i send a message").

As an output i want it to continuously print "done.", and also print "sent" when I get a message. So something like this:

done.
done.
done.
done.
done.
sent
done.
lots more done....

Basically i want both loops to run continuously and completely independently of each other.

N.B. I have tried using multiprocessing (such as in the 3rd answer here How do I run two python loops concurrently?), but couldn't get that to work either. I tried it as below:

import time
import zmq
from multiprocessing import Process

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5000")
i = time.time()

def zmq_loop():
    while True:
        message = socket.recv()
        socket.send(b"World")
        print "sent"

def done_loop():
    global i
    while True:
        i2 = time.time()-i
        if i2%2 == 0:
            print "done."

if __name__ == "__main__":
    Process(target=zmq_loop).start()
    Process(target=done_loop).start()

Answer 1

As was explained yesterday in this, the [CONCURRENT]-processing is technically achievable in several different ways in python, each with a way different cost.

Today, let's have a look onto another approach - using a framework, that was developed with the very same motivation - having the natural [CONCURRENT]-scheduling-already-in-DNA - originally intended for easy composing and smooth operating complex GUI Man-Machine-Interactions ( MMI ).

This framework may and will help you achieve a lot, right due to the fact, it has evolved with a lot of care for exactly the same scenarios, where more than one thing has to be monitored at once:

Welcome to Tkinter GUI framework, which we will use just for its smart concurrently operated event handlers.

I was many times positively surprised, how easy it comes to build a quite complex composition of Finite-State-Automata ( FSA ), that smoothly cooperate together ( a coalition of FSA-s ), using tools for both independent, isolated operations ( the inner-logic of each FSA ), yet being easily able to propagate signals / messages from one FSA towards another(s). Yes, they can actually operate in 1-event-source-FSA : N-consumer(s)-FSA(s)

There you can create ( with ZeroMQ always in a non-blocking manner ) handlers -- one "sniffer" for a regular checking ( best by a timeout-controlled .poll() method to { NACK | POSACK } anything to read ) -- another "reader" for actual reading from the ZeroMQ Socket() instance ( triggered by the POSACK-signal from the "sniffer", as was mentioned previously -- another "do-a-work-er" for any other task one may wish to operate

Tkinter .mainloop() method is the global controller, which orchestrates the dirty job for you.

The hich level concept of Tkinter-mediated agent's co-processing starts in main() as simple as:

def main():
    root = Tk()                      # INIT a Tk() instance
    root.lift()                      #      + make it visible
    app = myApplication( root )      # SET ( [HERE] are all your app gems )
    root.mainloop()                  # START the core event-handling orchestrator

Tkinter might look as a garrage full of various GUI-gadgets, that have nothing to do with your problem, but don't panic.

Tkinter has incredibly well created tools right for your needs.

• using control variables that will be used as a means for storing, signalling and propagating changes of values among otherwise independent and explicitly un-coordinated actors ( ref. "sniffer", "reader", "worker" and any others ... )

• tools for handling events - real, abstract and even virtual

• tools for handling timed-operations - in a form of an almost a lightweight real-time system, using setups with preferred timing of what shall happen next, the .mainloop()-yet tasked to bear in mind
  an explicitly specified timing .after( thisAmountOfMILLISECONDS, callThisFUNCTION ) or a liberal .after_idle( callAlwaysThatFUNCTION ).

One does not need indeed anything more to go and solve your task, using these already perfect tools.

So all the rest is just in principle under your creativity how to re-use these smart tools.

---

A small demo,
how to make two ( 3 ! ) things happen "at the same time independently"

Let's setup the case, when one wants to process ( here demonstrated by a printing ) several independent processes, all at the same time.

    >>> #-----------------------------------------------FAST MOCK-UP EXAMPLE
    >>> import Tkinter as tk                          # python27
    >>> root = tk.Tk()
    >>> root.protocol( "WM_DELETE_WINDOW", root.quit() )
    '3071841620Ldestroy'
    >>> #------VAR-------------------------------------IMPORTANT TOOL:
    >>> aStringVAR = tk.StringVar()
    >>> aStringVAR.set( "_init_" )

    >>> def aKeyPressEventHANDLER( anEvent ): # SIMPLE EventHANDLER,
            #                                 #        also ignites remote responsive processes
    ...     aTemplate = "[KEY]::{3: >10s}\n<s/n>::{0: >10d}\n(=@=)::{1: > 10d}\n^from::({5:})"
    ...     sString   = aTemplate.format( anEvent.serial,
    ...                                   anEvent.time,
    ...                                   anEvent.char,
    ...                                   anEvent.keysym,
    ...                                   anEvent.keysym_num,
    ...                               str(anEvent.widget )
    ...                               )
    ...     aStringVAR.set( sString )
    ...     print sString
    ... 
    >>> #----VAR_TRACER----------------------------------------[#1]
    >>> def aVAR_TRACER_A( p1_quasiNAME, p2_indexOrEmptyString, p3_accessMODE ):
    ...     print "aVAR_TRACER_A()-called::(on){0:} traced_event({1:})".format( str( p1_quasiNAME ), str( p3_accessMODE ) )
    ...     # ###############=[A]#######
    ...     # < do some task =[A] here >
    ...     # ###############=[A]#######
    ...     print "aVAR_TRACER_A()         [{0:}]".format(   str( root.globalgetvar( p1_quasiNAME ) ).replace( " ", "" ) )
    ...

    >>> #----VAR_TRACER----------------------------------------[#2]
    >>> def aVAR_TRACER_B( p1_quasiNAME, p2_indexOrEmptyString, p3_accessMODE ):
    ...     print "aVAR_TRACER_B()-called::(on){0:} traced_event({1:})".format( str( p1_quasiNAME ), str( p3_accessMODE ) )
    ...     # ###############=[B]#######
    ...     # < do some task =[B] here >
    ...     # ###############=[B]######
    ...     print "aVAR_TRACER_B()         [{0:}]".format(   str( root.globalgetvar( p1_quasiNAME ) ).replace( " ", "" ) )
    ... 

    >>> #-----VAR_A_tracer_ID------------------------------"w" EVENT SNIFFER
    >>> aTraceVAR_A_tracer_ID = aStringVAR.trace_variable( "w", aVAR_TRACER_A )

    >>> #-----VAR_B_tracer_ID------------------------------"w" EVENT SNIFFER
    >>> aTraceVAR_B_tracer_ID = aStringVAR.trace_variable( "w", aVAR_TRACER_B )

    >>> #-----------tracer_ID values for ev. theirs resp. de-activation:
    >>> aTraceVAR_A_tracer_ID
    '3071960124LaVAR_TRACER_A'
    >>> aTraceVAR_B_tracer_ID
    '3071961284LaVAR_TRACER_B'

    >>> #---.bind()-----------------------EventHANDLER with a system event <KeyPress>
    >>> root.bind( "<KeyPress>", aKeyPressEventHANDLER ) # <-since here LIVE (!)
    '3071841740LaKeyPressEventHANDLER'
    >>> #------------------------------------------------^^^ since here, it went live
    >>> #                                                 1: having put a mouse on tk-window,
    >>> #                                                 2: set-focus by click
    >>> #                                                 3: started keys:
    >>> #                                                    ( "a",
    >>> #                                                       6-on-<NumKeyPad>,
    >>> #                                                       *-on-<NumKeyPad>

    >>> # this happened "independently, at the same time" ( see time (=@=):: values )
    >>> 

    aVAR_TRACER_B()-called::(on)PY_VAR0 traced_event(w)
    aVAR_TRACER_B()         [[KEY]::a<s/n>::832(=@=)::88486992^from::(.)]
    aVAR_TRACER_A()-called::(on)PY_VAR0 traced_event(w)
    aVAR_TRACER_A()         [[KEY]::a<s/n>::832(=@=)::88486992^from::(.)]
    [KEY]::         a
    <s/n>::       832
    (=@=)::  88486992
    ^from::(.)
    aVAR_TRACER_B()-called::(on)PY_VAR0 traced_event(w)
    aVAR_TRACER_B()         [[KEY]::KP_6<s/n>::832(=@=)::88509107^from::(.)]
    aVAR_TRACER_A()-called::(on)PY_VAR0 traced_event(w)
    aVAR_TRACER_A()         [[KEY]::KP_6<s/n>::832(=@=)::88509107^from::(.)]
    [KEY]::      KP_6
    <s/n>::       832
    (=@=)::  88509107
    ^from::(.)
    aVAR_TRACER_B()-called::(on)PY_VAR0 traced_event(w)
    aVAR_TRACER_B()         [[KEY]::KP_Multiply<s/n>::832(=@=)::88541180^from::(.)]
    aVAR_TRACER_A()-called::(on)PY_VAR0 traced_event(w)
    aVAR_TRACER_A()         [[KEY]::KP_Multiply<s/n>::832(=@=)::88541180^from::(.)]
    [KEY]::KP_Multiply
    <s/n>::       832
    (=@=)::  88541180
    ^from::(.)