0

Regarding the code below of the process class MyProcessClass, sometimes I want to rerun all of the self.run tasks.

self.run(retry=True) is what I use to rerun the run(self) tasks within the class. It allows me rerun the tasks of the process class run(self) whenever I want to from wherever I want to from any class function.

MyProcessClass(Process):
     def __init__(self):
          Process.__init__(self)

     #gets called automatically on class initialization 
     #process.start().
     #it also gets called when a class function calls 
     #self.run(retry=True)
     def run(self,end=False,retry=False):
          if end==True:
             sys.exit()
          elif retry==True:
              redo_prep()
          do_stuff()

     #represents class functions doing stuff
     def do_stuff():
         #stuff happens well
         return
         #stuff happens and need to redo everything
         self.run(retry=True)

I don't want the thread/process to end, but I want everything to rerun. Could this cause problems because the run function is being called recursively-ish and I am running hundreds of these process class objects at one time. The box hits about 32GB of memory when all are running. Only objects that need to will be rerun.

My goal is to rerun the self.run tasks if needed or end the thread if needed from anywhere in the class, be it 16 functions deep or 2. In a sense, I am resetting the thread's tasks, since I know resetting the thread from within doesn't work. I have seen other ideas regarding "resetting" threads from How to close a thread from within?. I am looking for the most pythonic way of dealing with rerunning class self.run tasks.

I usually use try-catch throughout the class:

  def function():
     while True:
       try:
         #something bad
       except Exception as e:
         #if throttle just wait
         #otherwise, raise
       else:
         return

Additional Question: If I were to raise a custom exception to trigger a @retry for the retries module, would I have to re-raise? Is that more or less pythonic than the example above?

My script had crapped out in a way I hadn't seen before and I worried that calling the self.run(retry=True) had caused it to do this. I am trying to see if there is anything crazy about the way I am calling the self.run() within the process class.

Cloudish123
  • 847
  • 1
  • 9
  • 14
  • 1
    Consider using the `retrying` module. – sytech Mar 21 '18 at 21:02
  • This definitely has sparked my interest and is helping me see more about what I might need to do - so I appreciate the help, but I do not see how this can be called from within a try-catch without it handling it instead of the @retry. This is why I moved away from wrapping the whole thing within a try catch and defining special error exception classes like RetryThread. It ended up not being caught by the most outer try-catch. – Cloudish123 Mar 21 '18 at 21:22
  • 1
    The `retrying` module should cover your use case. What obstacle do you think you have with using `retrying`? Consider that you can create your own custom exceptions and handle them in a clever way (IE raise errors in a specific way to know what error-handling is required) -- I would advise against trying to invent complex retrying logic, since it should be easy to delegate to `retrying`. I think the problems you ran into can be overcome. Think about your problem carefully. Can you explain *why* you think this approach is necessary? If so, consider updating your question with those specifics. – sytech Mar 21 '18 at 21:32
  • ok. I am beginning to see the light. Do I just re-raise the custom exception within a try-catch? I am worried about this being even more complicated because I will have try-catch throughout my class and the @retry(specialexception) won't be hit unless I re-raise the custom exception. Do you suggest this is how I should deal with this? Is the way I mentioned above easier than this? I guess I really like the retry, but worry that having to re-raise this special exception in my try-catches isn't ok to do. I want to make sure it is pythonic. I edited the question to state this. – Cloudish123 Mar 21 '18 at 22:19
  • 1
    I would think about your design and where your try/excepts are happening and what specific errors you are catching. Calling a function that is decorated with `@retry` will not be caught by a try/except that is outside that decorated function and the retrying logic will work as expected. I'm not sure what you think your problem is, my guess is you're misunderstanding how the try/except affects retrying -- try putting together some simple examples and test how it works. If there's any reorganization needed, it should be trivial to do. Remember, simple is better than complex. Don't overthink it. – sytech Mar 21 '18 at 22:27
  • Thanks! I will be implementing the retrying module. After trying around a little bit I realized that the retrying modue does the retry logic I need perfectly. For ending threads, I am going to maybe have to use the self.run(endthread=True) to end my threads. I would rather use the retrying module for that too but I do not see a way to. I will play around more to see. – Cloudish123 Mar 22 '18 at 13:31

1 Answers1

1

It looks like you're implementing a rudimentary retrying scenario. You should consider delegating this to a library for this purpose, like retrying. This will probably be a better approach compared to the logic you're trying to implement within the thread to 'reset' it.

By raising/retrying on specific exceptions, you should be able to implement the proper error-handling logic cleanly with retrying. As a best-practice, you should avoid broad excepts and catch specific exceptions whenever possible.

Consider a pattern whereby the thread itself does not need to know if it will need to be 'reset' or restarted. Instead, if possible, try to have your thread return some value or exception info so the main thread can decide whether to re-queue a task.

sytech
  • 29,298
  • 3
  • 45
  • 86