0

UPDATED POST So, I found a solution to what I was doing it is to use Is there any way to kill a Thread in Python?

I would like to get some validation on design. We have a fairly large system, and one components of the system is the communication component. It is in charge of two things, one sending messages, and two queuing any messages that get received (to file,db, queue, etc.) I am receiving the message via a provided library that basically can just wait forever for a message to arrive.

Questions:

1 ) Is it best to have One main thread, and two sub threads?

2) Is it better to allow the Receive thread to just block forever until a message arrives? Or should I allow for it to timeout(which is an exception), and just continue to loop?

Let me know whatever more information you may need.

Right now I basically have a receive thread, and a main thread (which also handles the sending), which you can see below. The only thing not included below is the fect I can now call "terminate" on MessageReceiver and it will exit.

ORIGINAL POST I am trying to wrap a thread around some receiving logic in python. Basically we have an app, that will have a thread in the background polling for messages, the problem I ran into is that piece that actually pulls the messages waits forever for a message. Making it impossible to terminate... I ended up wrapping the pull in another thread, but I wanted to make sure there wasn't a better way to do it.

Original code: 

class Manager:
   def __init__(self):
        receiver = MessageReceiver()
        receiver.start()
        #do other stuff sending/etc...

class MessageReceiver(Thread):
   receiver = Receiver()

   def __init__(self):
        Thread.__init__(self)


   def run(self):           
       #stop is a flag that i use to stop the thread...
       while(not stopped ):
          #can never stop because pull below blocks
          message  = receiver.pull()
          print "Message" + message

I know the obvious locking issues exist, but is this the appropriate way to control a receive thread that waits forever for a message?

One thing I did notice was this thing eats 100% cpu while waiting for a message... **If you need to see the stopping logic please let me know and I will post.

Community
  • 1
  • 1
Nix
  • 57,072
  • 29
  • 149
  • 198
  • Could you please post the code for `Receiver`? – Travis Mehlinger Feb 14 '11 at 22:20
  • to us the receiver is a black box, and basically takes 1 argument "timeout". As I said below, we want this thing to wait forever, unless we decide to close the program then we need a way to be able to stop the pull. – Nix Feb 15 '11 at 02:14
  • Can you tell us anything about `Receiver`? Do you know if `pull()` is CPU-bound or IO-bound? It's difficult to guess what the problem is without knowing anything about the apparent cause. – Travis Mehlinger Feb 15 '11 at 03:34
  • The problem is if i have a while(true) receiver.pull loop, I have no way to stop it... I have no doubt that the reason I am using 100% cpu is bc of my threading above, not the receiver. – Nix Feb 15 '11 at 15:09
  • I didn't see your mention of 100% CPU usage before. That would make most sense if, called without arguments, receiver.pull() is not actually blocking. You could easily test this with a print statement on the line above. – jd. Feb 15 '11 at 20:48
  • I fixed the CPU usage issue, it was a dumb thing on my side where i was looping versus waiting for a signal... – Nix Feb 15 '11 at 21:27

3 Answers3

0

What function ultimately waits for a message to come? Most probably there's a way to make it time out and thus check a termination flag periodically.

9000
  • 39,899
  • 9
  • 66
  • 104
  • The receiver .pull method has an optional timeout, but basically the logic will wait forever, unless someone decides to stop it. – Nix Feb 15 '11 at 02:11
0

1) If your third thread is just going to be waiting on the receiver and sender, there is no need.

Working with multiple processes is less convenient than multiple threads. The big advantage of processes is that they avoid CPython's limitation with regard to threading, that is, two threads cannot do processing at the time (but one thread can run while the others are blocking for IO). So unless both thread may be doing a lot of processing, you are better off keeping them in the same process.

2) You should allow your receiver to timeout and check for a termination flag in the loop.

jd.
  • 10,678
  • 3
  • 46
  • 55
  • Is that better than using an exception to interrupt the receive thread? – Nix Feb 15 '11 at 17:25
  • If you mean raising an Exception like in the accepted answer to http://stackoverflow.com/questions/323972/is-there-any-way-to-kill-a-thread-in-python, then yes, the timeout + flag solution is better. Raising the exception requires using the Python C API from within Python code, which is not portable, feels hackish, requires a lot more code and could perhaps be broken in future versions. – jd. Feb 15 '11 at 17:46
  • It would be different, the exception would be a timeout exception from the receiver, which I would just catch, check the loop flag, and continue if terminate has not been called. – Nix Feb 15 '11 at 18:10
  • I don't see a problem with that. That's how you would have to do it with Queue.Queue() in the standard library I believe. – jd. Feb 15 '11 at 18:13
0

Since there was never a real answer to this question I will summarize:

  1. One Main thread which handles the sending, and spawns a receiving thread
  2. Block forever, exceptions are costly, so there is no need to let it time out.

In order to stop the thread I basically set the stop flag then terminate the underlying connection in a "terminate" message of the parent. In return which causes an exception in the child.

In the exception handling block I then check the ConnectionException, if the"stop" flag has been set, I gracefully stop the receiving thread, if it was an ungraceful exit, I will then notify the parent thread.

Hope this helps, this was a super annoying problem, but I am glad it is over.

class Manager:
   def __init__(self):
      self.receiver = MessageReceiver(shutdown_hook = self.shutdown_hook)
      self.receiver.start()
      #do other stuff sending/etc...
   def terminate(self):
      self.receiver.stop()
      self.receiver.close()

   def shutdown_hook(self, t_id, child):
       print '%s - Unexpected thread shutdown, handle this.. restart thread?' % str(t_id))

class MessageReceiver(Thread):


   def __init__(self,shutdown_hook = None):
     Thread.__init__(self)
     self.receiver = Receiver()
     self.shutdown_hook = shutdown_hook 


   def run(self):           
     #stop is a flag that i use to stop the thread...
     while(not stopped ):
      try:
        message  = self.receiver.pull()
        print "Message" + message   
      except ConnectionException as e:
         if stopped:
            #we are in the process of stopping
            pass
         else:
            self.shutdown_hook(self.iden, e)
            break
      finally:
            receiver.close()
Nix
  • 57,072
  • 29
  • 149
  • 198