how to use dir() as default argument within a function

Question

I can't figure out what I'm doing wrong. It seems I'm not able to use dir() as the default argument to a function's input

def print_all_vars(arg=dir()):
    for i in arg:
        print(i)

foo=1

If I use

print_all_vars()

gives the output

__builtins__
__cached__
__doc__
__file__
__loader__
__name__
__package__
__spec__

but

print_all_vars(dir())

outputs

__builtins__
foo
print_all_vars

Expressions in default arguments are executed just *once* and stored with the function object. They are not for each call. You are seeing the module namespace (the globals in the module the function is defined in). — Martijn Pieters, Mar 03 '17 at 18:36
@Lee88: the `dir()` of *what* exactly? What namespare are you looking to introspect? — Martijn Pieters, Mar 03 '17 at 18:41
Maybe it would be better if you explain what you are trying to accomplish. This sounds like the XY problem to me. — juanpa.arrivillaga, Mar 03 '17 at 18:44
@MartijnPieters I thought of dir() as listing all variables in the current scope (frame? not sure of the correct terminology). So in this minimal working example, I was hoping to print all variables. — Lee88, Mar 03 '17 at 18:46
@juanpa.arrivillaga Outside of this minimal working example, I am actually trying to create an analog of MATLAB's save function that saves all existing variables to an external file that can be recalled later — Lee88, Mar 03 '17 at 18:47
@Lee88 that seems fraught with pitfalls. Are you going to `pickle` every object? — juanpa.arrivillaga, Mar 03 '17 at 18:48
@juanpa.arrivillaga I am working with the shelve package, because I thought pickle would require creation of a separate file for each variable — Lee88, Mar 03 '17 at 18:49
@Lee88: see [How to access ALL global variables in a python function?](//stackoverflow.com/q/15748303) then — Martijn Pieters, Mar 03 '17 at 18:51
@Lee88 you might be interested in [dill](https://pypi.python.org/pypi/dill) which provides the capability to "save" interpreter sessions, which is what I presume you want to achieve. — juanpa.arrivillaga, Mar 03 '17 at 18:54
@MartijnPieters: I'd propose editing the question to read "how can I retrieve variable names from the caller context" and reopening - what do you say? — jsbueno, Mar 03 '17 at 20:10

score 1 · Accepted Answer · answered Mar 03 '17 at 20:08

So - first, what is going on -

The line containing the def statement is run only once, when the module containing the function is first imported (or on the interactive prompt) - anyway, any expressions used as default parameter values are evaluated a single time, and their resulting object is kept as the default value for that parameter.

So, your dir() call is executed, and the resulting list - a dir on the namespace of the module containing your example function is saved. (The most common related error for Python beginners is to try to put an empty list or dictionary ( [] or {}) as a paramter default: the same list or dictionary will be reused for the lifetime of the function).

That explains why you see in the output some values that won't show up if you call it with a dir from the interactive prompt, like __package__ or __spec__.

Now, your real question is:

... I am actually trying to create an analog of MATLAB's save function that saves all existing variables to an external file that can be recalled later –

So, even if your dir() would be lazily executed, when the function is called only, that would still fail: it would still retrieve the variable names from the module where the function is defined, not on the callers.

The coding pattern for doing that would be:

def print_all_vars(arg=None):
    if arg is None:
        arg = dir()
    ...

(And this is the most common way to enforce empty lists of dictionaries as default parameters on the function body)

But even this, or if you pass your own list of variable names explicitly to the function, you will still have just a list of strings - not Python objects that need you will want to save.

And - if you want to retrieve the variables from the caller by default, you have a problem in common with having this list of strings: you need to be able to access the namespace of the caller code, in order to either retrieve the variable names existing there, and later to retrieve their contents in order to be saved.

The way to know "who've called your function" in Python is an advanced thing, and I am not even sure it is guarantee to work across all Python implementations - but it will certainly work on the most common (cPython, Pypy, Jython) and even some considered smaller ones (such as Brython):

You call sys._getframe() to retrieve the frame object of the code currently being executed (that is the code in your function itself) - and this frame object will have a f_back attribute, which points to the frame object of the code which called your function. The frame object, on its side, has reference to the globals and locals variables of the code being run as plain dictionaries as f_globals and f_locals respectively.

You can them "save" and "restore" as you want the contents of the global variables on the caller code - but you can't update - using pure Python code - the local variables.

So, just to list by default the global variables on the caller code - you can do:

import sys


def print_all_vars(arg=None):
    if arg is None:
        arg = list(sys.get_frame.f_back.f_globals.keys())
    print (arg)

Being able to properly saving and restoring those variables is another question - as you can see in the comments, you can use pickle to serialize a lot of Python objects - but the caller namespace will have module names (such as mathnp`, etc...) which can't be ordinarily pickled - so you will have to come with an strategy to detect, annotate the used modules, and restoring them, by using further introspection on all objects retrieved.

Anyway, experimenting with the information here should put you on the track for what you want.

how to use dir() as default argument within a function

1 Answers1