0

This question may sound similar to the following, but I'm not sure how to apply their solutions to my use-case:

How to import a module given the full path?
Can you use a string to instantiate a class?

I have a class Foo defined and imported in my project. However, at runtime I may have a string containing a different definition of Foo (along with lots of other classes and import statements). I'd like to be able to replace the already loaded Foo with the one in my string, in a way that after the operation, anybody who instantiates f = Foo() would instantiate the definition from my string. At the same time, I'd like to ignore any other definitions/imports in my string. How to do this?

Assume the following project structure and use-case:

project/
    __init__.py
    mytypes/
        __init__.py
        foo.py  # contains the definition 'class Foo'
    another_package/
        bar.py
    main.py

Inside main.py and bar.py I have from mytypes.foo import Foo. After the replace operation detailed above I want both to use the new definition of Foo from the replacement string, but no other definition from my string.

Maghoumi
  • 3,295
  • 3
  • 33
  • 49
  • Where did this string come from, and why is it a string? This sounds like a job better handled at a different level. – user2357112 May 10 '20 at 21:36
  • If `Foo` is imported using `from ... import Foo` then any changes you make to `Foo` in the module will not change the already imported class. What would you want to happen with any instances of `Foo` that have already been initialized? – Iain Shelvington May 10 '20 at 21:39
  • @user2357112supportsMonica This string came from an older definition of Foo. Somebody thought to back up the definition of `Foo` (along with some other classes) in the form of a string. – Maghoumi May 10 '20 at 21:45
  • @IainShelvington The use-case guarantees that this "patching" operation happens before any first instantiation of `Foo` – Maghoumi May 10 '20 at 21:46
  • This sounds like you should take the class definition from that string and put it back in `foo.py`, with an `if` that picks which one to use at module loading time. – user2357112 May 10 '20 at 21:47
  • (Be careful about dependencies, though - you say you want to ignore any other definitions and imports, but the old version of the class likely depends on old imports or old versions of other functions and classes, and it may not be drop-in compatible with the current code.) – user2357112 May 10 '20 at 21:48
  • @user2357112supportsMonica That's the first thing that I thought of. Problem is, I need to parse the string, extract `class Foo()` all the way to the end, then do the patch work. I thought Python might support this in an easier way – Maghoumi May 10 '20 at 21:49
  • Also, about the compatibility... The interface of Foo is guaranteed to be compatible with the new definition (constructor, functions defs, etc). – Maghoumi May 10 '20 at 21:49
  • No, I mean you, the human, should read the string and edit `foo.py`. I'm not suggesting that the program handle it at runtime. Also, even if the outsides of the old `Foo` are guaranteed to be compatible with the outsides of the new `Foo`, the insides may not be compatible with the new `foo.py`. – user2357112 May 10 '20 at 21:51
  • If this string is only available at runtime, why is that the case? It's supposed to be a backup of an old version. That sounds like the kind of thing that should be available in advance. – user2357112 May 10 '20 at 21:52
  • There are multiple so-called "backups" of `Foo`. So depending on the runtime environment, one needs to instantiate a different version of `Foo` (I know the design is awkward, but that's all I have to work with for now). In other words, the runtime is what determines what `Foo`'s definition should be – Maghoumi May 10 '20 at 21:53
  • So put them all in `foo.py` and pick which one to use at runtime. `if case1: class Foo: version1code... else if case2: class Foo: version2code... else if case3: class Foo: version3code...` etc. – user2357112 May 10 '20 at 21:58
  • @user2357112supportsMonica While this approach is valid, it's not feasible given how many different versions of `Foo` may exist. I'm looking for an automatic way to do this. – Maghoumi May 10 '20 at 22:04
  • If there are that many versions, that makes the compatibility problems even worse. I think you're being way too optimistic about how compatible all these versions of `Foo` will be. You'll have issues like, what if one version relied on `from os import path` and another had `import os.path`, and a later version switched to `pathlib`? Even if all 3 versions have the exact same interface, none of them are compatible enough to do what you're trying, because the imports they rely on aren't part of the class definition. – user2357112 May 10 '20 at 22:10
  • I realize this is a genuine concern, but the structure of `Foo` is relatively simple, hence my optimism. There are only two functions inside each version of `Foo` that could vary. These function do some mathematical operations that are import-independent. This whole use-case seems like a half-hearted attempt at serialization, but also making sure the changes between different versions of `Foo` is tracked, and one could still recover the serialized information after `Foo` is changed – Maghoumi May 10 '20 at 22:15

1 Answers1

2

The short answer is: don't do this. You will run into all kinds of strange errors that you will not expect

You can use exec to run some arbitrary code, if you pass a dictionary as the second argument the resulting "globals" from the executed string will be stored in the dictionary

namespace = {}
exec('class Foo:\n    x = 10', namespace)
namespace['Foo']  # This will be a class named Foo

You could then assign this to the module

import your_module
your_module.Foo = namespace['Foo']

Now anywhere that your_module.Foo is accessed you will get the class from your string.

However, your module may have been imported at some time before you have patched it, it's very difficult to be able to say for certain when your module will be imported. Someone may have bound Foo using from your_module import Foo, if this has run before your patch then you will not change this Foo class. Even if you only ever access your_module.Foo, if an instance has been initialised before your patch then any subsequent instances will not even have the same type!

f = your_module.Foo()
# run your patch
isinstance(f, your_module.Foo)  # False

In the case above f is an instance of a completely different type to the current your_module.Foo

Iain Shelvington
  • 31,030
  • 3
  • 31
  • 50
  • Thanks for your answer. One immediate problem with this is the bold part in my question: `along with lots of other classes and import statements`. Basically, if I do `exec` on the string, unrelated statements may get executed. Imagine the string has `import numpy`, but the target environment doesn't have `numpy` installed (and Foo does not depend on `numpy`). Unless I somehow parse the string manually and extract `class Foo` all the way to the end of the class definition, your solution (as is) could run into other issues. – Maghoumi May 10 '20 at 22:23
  • Is there no way in Python to selectively import definitions from a string? – Maghoumi May 10 '20 at 22:23
  • 1
    There is a section in this talk that goes over why this is not possible https://www.youtube.com/watch?v=DlgbPLvBs30 – Iain Shelvington May 10 '20 at 22:24
  • I wrote a regex to extract `class Foo:....`, and tried your solution. It works, but involved more work than I was hoping for. Will wait for a bit to see if anybody can suggest an alternative. Will mark your solution as the answer if nobody can come up with something simpler. – Maghoumi May 10 '20 at 23:17
  • Do your Foo classes use any imported modules, including built in modules? If you’re going to go completely crazy you could override import behaviour while executing your file. I seriously would reconsider this approach though – Iain Shelvington May 10 '20 at 23:27
  • It only imports two non built-in modules. Since my regex only extracts the class definition, and I'm adding those two imports manually (basically `class_def = "import X\nimport Y\n" + class_def`) -- the code is certainly ugly... What do you have in mind wrt import behavior? – Maghoumi May 10 '20 at 23:32
  • Marking this one as the answer. – Maghoumi May 19 '20 at 18:45