0

I'm trying to put a timeout on a function that runs in a thread in python. My first attempt was using a signal, as suggested in any number of answers here, but since the function I want a timeout on will sometimes be run in a thread (as part of a larger function that is allowed to take some time), this turned out to not be an option.

So I then tried this answer, which involves using a Thread from the multithreading class. That worked fine in initial testing, but the comments on that answer indicate that if an exception is raised, that answer will hide the exception, and also that it can be unsafe due to GIL interactions. Additionally, I note that there is no way to kill a thread, which would seem to indicate that there is the potential of ending up with numerous hung threads, should the process never complete. Or even if it does eventually complete, it's still using resources that are now useless, since the result will never be seen.

Digging around, it would appear a multiprocessing library based solution would be preferable, using the same basic structure as in the above referenced answer and a Queue for communication back to the originating thread. No GIL issues to worry about (since it is a separate process), and I can kill off the process should the timeout expire. So I tried that, only to run into another problem: to put something on a Queue, it has to be pickle-able - and the item being returned from the function is not.

So, given the situation:

  • running in a thread
  • return value is not pickle-able
  • want to avoid the issues presented with the threading answer

is there a solution that solves all the above issues? Or do I just need to go with the above referenced threading answer, and live with the limitations?

Community
  • 1
  • 1
ibrewster
  • 3,482
  • 5
  • 42
  • 54
  • What makes the return non-picklable? Can you return a picklable view of the data? You can't pickle things like locks but they don't make sense to return anyway. If its your own class, you can add `__getstate__` and `__setstate__` magic methods for pickling. – tdelaney Feb 27 '17 at 22:32
  • It's actually a python wrapped C object, I think. Database cursor. I'd have to look into the code more to remind myself exactly, but for sure it references C objects at the least. – ibrewster Feb 27 '17 at 23:54
  • Sounds like something that's not useful to pass back. Why not pass back just the data? – tdelaney Feb 28 '17 at 00:42
  • Because there *is* no data yet. You use the cursor to *obtain* data from a SQL database. Pretty standard DB-API stuff - make a connection, get a cursor, use the cursor. The problem, in this case, is that with the specific DB I am connecting to, connecting and getting a cursor can hang - so I need a timeout. – ibrewster Feb 28 '17 at 00:44
  • If you are just getting a cursor then you really need to do it in a thread in the current process. Which API do you use? Can you post an example? You can't cancel threads but sometimes you can close whatever resource the thread is waiting on. If we know the API we may find a way to do that. – tdelaney Feb 28 '17 at 00:50

0 Answers0