0

I want to

  1. load data from files,
  2. work on that data,
  3. and eventually save that data back to files.

However, since step 2 may take several hours I want to make sure that progress is saved in case of an unexpected exception. The data is loaded into an object to make it easy to work with it. First thing that came to my mind was to turn that objects class into a context manager and use the with-statement. However I'd have to write practically my entire program within that with-statement and that doesn't feel right. So I had a look around and found this question which essentially asks for the same thing. Among the answers this one suggesting weakref.finalize seemed the most promising to me. However, there is a note at the bottom of the documentation that says:

Note: It is important to ensure that func, args and kwargs do not own any references to obj, either directly or indirectly, since otherwise obj will never be garbage collected. In particular, func should not be a bound method of obj.

Since I'd want to save fields of that object, I'd reference them, running right into this problem.

Does this mean that the objects __exit__-function will never be called or that it will not be called until the program crashes/exits?

What is the pythonic way of doing this?

Klaus
  • 40
  • 7

1 Answers1

1

It's a little hacky, and you still wrap your code, but I normally just wrap main() in a try except block.

Then you can handle except with a pdb.set_trace(), which means whatever exception you get, your program will drop into an interactive terminal instead of breaking.

After that, you can manually inspect the error and dump any processed data into a pickle or whatever you want to do. Once you fix the bug, setup your code to read the pickle and pick up from where it left off.

CasualScience
  • 661
  • 1
  • 8
  • 19
  • This is a very helpful tip for debugging, learned something new today! But I'm looking for a more long-term solution. When I'm talking about exceptions I don't necessarily mean just those introduced by bugs, but also keyboard exceptions or computer reboots triggered by a cat or something similarly out of control :D – Klaus May 05 '22 at 23:22
  • `try-except` will catch keyboard interrupts as they are handled as exceptions in python. For the power turning off, I don't think there's anything you can do outside of regularly dumping a cache of the processed data... callbacks and stuff aren't going to run if the computer loses power... – CasualScience May 06 '22 at 02:40