Pickling Self and Return to Run-state?

Question

I found the following post extremely helpful: How to pickle yourself?

however the limitation with this solution is that when the class is reloaded, it is not returned in its "runtime" state. i.e. it will reload all the variables etc and the general state of the class at the moment it was dumped.. but it won't continue running from that point.

Consider:

class someClass(object):
   def doSomething(self):
       i = 0         
       while i <= 20:
          execute
          i += 1
          if i == 10:
             self.dumpState()

   def dumpState(self):
      with open('somePickleFile','wb') as handle:
         pickle.dump(self, handle)

   @classmethod
   def loadState(cls, file_name):
      with open(file_name, 'rb') as handle:
          return pickle.load(handle)

If the above is run, by creating an instance of someClass:

 sC = someClass()
 sC.doSomething()
 sC.loadState('somePickleFile')

This does not return the class to its runtime state, it does not continue through the while loop until i == 20..

This may not be the correct approach, but I am trying to find a way to capture the runtime state of my program i.e. freeze/hibernate it, and then relaunch it after possibly moving it to another machine.. this is due to issues I have with time restrictions enforced by a queuing system on a cluster which does not support checkpointing.

Answer 1

That approach won't be possible with Pickle and Unpickle alone without your code being aware of it.

Pickle can save fundamental Python objects, and ordinary user classes that reference those fundamental types. But it can't freeze information of a running context as you want.

Python does allow limited (yet powerfull) ways of acessing a running code context trough its frame objects - you can get a frame object with a call to "inspect.currentframe" in the inspect module. This will allow you to see the current running line of code, local variables, content of local variables, and so on -- but there is no way inside pure-python, without resorting to raw memory manipulation of the Python interpreter's data structures to rebuild a mid-execution frame object and jump execution to there.

So - for that approach it would be better to "freeze" the entire process and it's memory data structures using an O.S. way to do that (probably there is a way to that in Linux and it should work with no file/file like resources in use by the process).

Or, from within Python, like you want, you have to keep "book check" of all your state data in a manner that Pickle would be able to "see it". In your basic example, you should refactor your code to something like:

class someClass(object):
   def setup(self):
       self.i = 0
   def doSomething(self):       
       while self.i <= 20:
          execute
          i += 1
          if i == 10:
             self.dumpState()
   ...
  @classmethod
   def loadState(cls, file_name):
      with open(file_name, 'rb') as handle:
          self = pickle.load(handle)
      if self.i <= 20: # or other check for "running context"
          return self.doSomething()

The fundamental difference here is the book-keeping of the otherwise local "i" varianble as an object variable, and separate the initization code. In this way, all the state needed to continue the execution - for this small example - is recorded on the object attributes - which can be properly pickled.

Answer 2

loadState is a classmethod returning a new instance of someClass (or something else pickled into the file). So you should write instead:

sC = someClass()
sC.doSomething()
sC = someClass.loadState('somePickleFile')

Answer 3

I believe pickle only keeps the attribute values of the instance, not the internal state of any methods executing. It will not save the fact that a method was executing, and it won't save the values of the local variables, like i in your example.