2

The documentation suggests the following mechanism to dynamically create data containers in Python:

class Employee:
    pass

john = Employee() # Create an empty employee record

# Fill the fields of the record
john.name = 'John Doe'
john.dept = 'computer lab'
john.salary = 1000

The above allows one to easily group a diverse set of variables within one single identifier (john), without having to type quotes ('') as one would do with a dictionary.

I am looking for a solution that allows me to "dump" the pieces (the attributes) back into the current namespace. There are three ideas/problems that come to mind to address this:

1. Given the identifier above john, how can I programatically get a list of it's attributes?

2. How can I easily dump john's attributes in the current namespace? (i.e. create local variables called name, dept, salary either via shallow or deep copies)

3. The top answer in the following thread describes a way to dump variables from the namespace created by argparse: Importing variables from a namespace object in Python

Perhaps I could use a Namespace object as a data container, as in the above post, and then easily dump those variables with:

locals().update(vars(john))

?

For convenience, below I include a list of threads discussing other approaches for creating data containers in Python, some of which don't seem to be pickable:

Connection with MATLAB workflows:

For reference, MATLAB provides this exact functionality through save and load, and variables can be nested and unnested easily, eliminating the need for quotes/dictionaries for this purpose). The motivation behind this question is to identify mechanisms that support such "pickable workspaces" in Python.

Community
  • 1
  • 1
Amelio Vazquez-Reina
  • 91,494
  • 132
  • 359
  • 564

1 Answers1

2
  • Given the identifier above john, how can I programatically get a list of it's attributes?

    vars(john)

Technically, this will give you a dictionary mapping. If you only want the list of attributes, then you actually will need vars(john).keys()

  • How can I easily dump john's attributes in the current namespace? (i.e. create local variables called name, dept, salary either via shallow or deep copies)

I'm not sure what you mean here about the shallow or deep copies. If you're talking about simple references, there is no (good) way to do this. If you're in the global (module level) namespace, you can do:

globals().update(vars(john))

If you're using CPython, using locals().update(vars(john)) works (in some places), but the documentation explicitly warns against doing this. The best you can do is some sort of exec loop (yuck!):

d = vars(john)
for k in john:
    exec '{key} = d["{key}"]'.format(key=k)

beware that there is a very good reason why this code is ugly -- mainly -- YOU SHOULDN'T BE DOING SOMETHING LIKE THIS :-P

and when using exec, the usual warnings apply -- Make sure you trust the attributes on john. e.g. setattr(john,'__import__("os").remove("useful_file"); foo',"anything here") would make for a pretty bad day ...

mgilson
  • 300,191
  • 65
  • 633
  • 696
  • Yeah, the `locals()` trick won't even work from within a function, which dramatically limits its usefulness. The `exec` approach is -- Heaven help us -- the "right" one to achieve this end.. – DSM Feb 05 '13 at 21:17
  • @DSM -- Yeah, when I started writing the answer, I was planning on saying "There is no way to do this" ... but then I thought of `exec` and realized that statement was wrong. So I thought it my duty to at least give the correct answer, and then slap it with a bold warning (maybe I should capitalize that too??) about it not being the right thing to do ... – mgilson Feb 05 '13 at 21:19
  • An `exec` statement with just an expression *calls `locals()`* (or rather, the C function `PyEval_GetLocals`, which is the same thing) to update. If `locals()[key] = val` does not work, neither will `exec`. – Martijn Pieters Feb 05 '13 at 22:15
  • See [`ceval.c` (python 2.7)](http://hg.python.org/cpython/file/2.7/Python/ceval.c#l4675) and [`bltinmodule.c`](http://hg.python.org/cpython/file/2.7/Python/bltinmodule.c#l1328). – Martijn Pieters Feb 05 '13 at 22:19
  • 1
    @MartijnPieters: I'm pretty sure that's not true. `def f(): locals()['x'] = 2; print x` doesn't work. `def g(): exec("x = 2"); print x` does. – DSM Feb 06 '13 at 00:24
  • @DSM: I linked to the source code; it is certainly true judging by the code. Use dynamic variables; judging by the `dis.dis()` result the python compiler marked `x` as a local for `x` in `g()`. It'd be interesting to find out *why* the Python compiler determined `x` to be local in the `exec()` case though, I think that might actually be a bug. – Martijn Pieters Feb 06 '13 at 00:26
  • 1
    @MartijnPieters: I think I'm missing something. If your theory were right, then "If locals()[key] = val does not work, neither will exec". But the example I gave shows a case where the first one doesn't work, but the second one does. I seem to vaguely recall something about `exec` disabling certain interpreter-level optimizations, possibly including the ones which prevent `locals()['x'] = 2` from working, but I could be wrong about that. What I'm pretty sure of is that they're not equivalent in practice. – DSM Feb 06 '13 at 00:29
  • 1
    @DSM: I think your hunch is right; using `exec` in the function seems to lift the `locals()` restriction, full stop. `def g(): exec("x = 2"); locals()['y'] = 2; print y` works too! On closer look, all `LOAD_FAST` opcodes have been replaced by `LOAD_NAME` instead, which looks in both locals and globals. Very interesting indeed. – Martijn Pieters Feb 06 '13 at 00:34
  • Wow, a whole conversation that went on behind my back ;-) – mgilson Feb 06 '13 at 00:59
  • @DSM: It's the `CO_OPTIMIZED` flag; it's not set in `g.__code__.co_flags`. The flag is left unset when `exec` is present in the symbol table for the current scope. When the flag is set, `frameobject.c` prevents certain manipulations of `locals()`, and the scope is left undetermined in `compile.c`, leading to `LOAD_NAME` being used. This is cool! :-P It is an unintended side-effect; you cannot optimize a function that uses `exec` and thus you cannot lock down `locals()`. Something to remember! – Martijn Pieters Feb 06 '13 at 01:23
  • @MartijnPieters -- I wish I knew 25% as much about the cpython implementation that you know ... However, sometimes, you just need to [trust the documentation](http://docs.python.org/2/reference/simple_stmts.html#grammar-token-exec_stmt) when it says **"In all cases, if the optional parts are omitted, the code is executed in the current scope"** – mgilson Feb 06 '13 at 03:19
  • @mgilson: yet that doesn't state that the current scope will actually be affected. :-P As it wasn't explicit about it, I didn't trust it. Or at least, I wanted to *know* how that worked in that case. – Martijn Pieters Feb 06 '13 at 03:25
  • @MartijnPieters -- I thought that was taken care of by the statement : "This statement [exec] supports dynamic execution of python code" ... – mgilson Feb 06 '13 at 03:36