Save workspace in IPython

Question

Is it possible to save an IPython workspace (defined functions, different kinds of variables, etc) so that it can be loaded later?

This would be a similar function to save.image() in MATLAB or R. Similar questions has been asked before, such as:

Save session in IPython like in MATLAB?

However, since a few years passed, I am wondering if there is a good solution now.

Answer 1

You can use the dill python package:

import dill                            
filepath = 'session.pkl'
dill.dump_session(filepath) # Save the session
dill.load_session(filepath) # Load the session

To install it:

pip install dill

Answer 2

EDIT: this answer (and gist) has been modified to work for IPython 6

I added a somewhat ad-hoc solution that automates the process of storing/restoring user space variables using the underlying code from IPython's %store magic which is from what I understand what you wanted. See the gist here. Note that this only works for objects that can be pickled.

I can't guarantee its robustness, especially if any of the autorestore mechanisms in IPython change in the future, but it has been working for me with IPython 2.1.0. Hopefully this will at least point you in the right direction.

To reiterate the solution here:

1. Add the save_user_variables.py script below to your ipython folder (by default $HOME/.ipython). This script takes care of saving user variables on exit.

2. Add this line to your profile's ipython startup script (e.g., $HOME/.ipython/profile_default/startup/startup.py):

   get_ipython().ex("import save_user_variables;del save_user_variables")

3. In your ipython profile config file (by default $HOME/.ipython/profile_default/ipython_config.py) find the following line:

   # c.StoreMagics.autorestore = False

   Uncomment it and set it to true. This automatically reloads stored variables on startup. Alternatively you can reload the last session manually using %store -r.

save_user_variables.py

def get_response(quest,default=None,opts=('y','n'),please=None,fmt=None):
    try:
        raw_input = input
    except NameError:
        pass
    quest += " ("
    quest += "/".join(['['+o+']' if o==default else o for o in opts])
    quest += "): "

    if default is not None: opts = list(opts)+['']
    if please is None: please = quest
    if fmt is None: fmt = lambda x: x

    rin = input(quest)
    while fmt(rin) not in opts: rin = input(please)

    return default if default is not None and rin == '' else fmt(rin)

def get_user_vars():
    """
    Get variables in user namespace (ripped directly from ipython namespace
    magic code)
    """
    import IPython
    ip = IPython.get_ipython()    
    user_ns = ip.user_ns
    user_ns_hidden = ip.user_ns_hidden
    nonmatching = object()
    var_hist = [ i for i in user_ns
                 if not i.startswith('_') \
                 and (user_ns[i] is not user_ns_hidden.get(i, nonmatching)) ]
    return var_hist

def shutdown_logger():
    """
    Prompts for saving the current session during shutdown
    """
    import IPython, pickle
    var_hist = get_user_vars()
    ip = IPython.get_ipython()
    db = ip.db

    # collect any variables that need to be deleted from db
    keys = map(lambda x: x.split('/')[1], db.keys('autorestore/*'))
    todel = set(keys).difference(ip.user_ns)
    changed = [db[k] != ip.user_ns[k.split('/')[1]]
               for k in db.keys('autorestore/*') if k.split('/')[1] in ip.user_ns]

    try:
        if len(var_hist) == 0 and len(todel) == 0 and not any(changed): return
        if get_response("Save session?", 'n', fmt=str.lower) == 'n': return
    except KeyboardInterrupt:
        return

    # Save interactive variables (ignore unsaveable ones)
    for name in var_hist:
        obj = ip.user_ns[name]
        try:
            db[ 'autorestore/' + name ] = obj
        except pickle.PicklingError:
            print("Could not store variable '%s'. Skipping..." % name)
            del db[ 'autorestore/' + name ]

    # Remove any previously stored variables that were deleted in this session
    for k in todel:
        del db['autorestore/'+k]

import atexit
atexit.register(shutdown_logger)
del atexit

Answer 3

Although not so convenient as save.image(), you can use one of the checkpoint/restore applications. If you're using Linux, you might try http://criu.org. I'm using it from time to time to dump my ipython state and restore it later.

In order to dump a shell app with CRIU, you need to find its PID (e.g. pstree -p) and then use something like this (you'll need a second terminal for this; CRIU can't dump stopped jobs):

sudo criu dump -t PID --images-dir ~/tmp/imgs --log-file dump.log -v4 --shell-job

this will write all necessary images to ~/tmp/imgs (remember the --shell-job option). In order to restore the state later to the current terminal (don't forget to hit enter to get the next ipython prompt):

sudo criu restore --images-dir ~/tmp/imgs/ --log-file restore.log -v4 --shell-job

Check out the logs in case of any problems.

Obviously CRIU will work with any app (with some limits, of course). It's just an idea so you can use it for ipython.

Answer 4

You can try

%save name lines

Like if you have input 67 commands and you want to save all of them:

%save myhistory 1-67

Answer 5

you can certainly do this in the ipython notebook.

when the notebook is saved--either manually or by default config--the notebook is persisted as an .ipynb file, which is just a json file (an example in a github gist).

Next time you start the ipython server in the directory where that file resides, the server will detect it.

when you open that notebook in the browser, all of the code & config is there but unexecuted; you can execute the code in every cell by selecting execute all cells from the cells menu.

in addition, you can manually persist snapshots of your notebook, as ipynb_checkpoints, which are stored in a directory of that name preceded by a dot.

and finally, from the file menu option, you can persist your notebook as a pure python source file (.py)