Using exec to load module variables, using a file path

Question

A textbook suggests I should be able to do this:

d = {}
exec("C://Users//Dave//Desktop//Bot//bot_config_data.py", globals(), d)
File "<string>", line 1   C://Users//Dave//Desktop//Bot//bot_config_data.py
                           ^
SyntaxError: invalid syntax

I can do this:

d = {}
exec('from bot_config_data import price_data', globals(), d)

But, I would like to do the former.

I'm trying to write a method which overrides config data from various files.

Am I completely off base here?

Update The book was quite misleading. It posted part of the code, the result of the complete code block, and then gave the remainder. As I was referencing, rather than working through it cover to cover, I tripped over myself.

This is the code I now have:

data = {}
file = 'C:\\Users\\Dave\\Desktop\\Bot\\bot_config_data.py'
with open(file) as f:
    code = compile(f.read(), file, "exec")
    exec(code, globals(), data)
price_data = data["price_data"]

Update2 Using Mad Physicist's answer below, my code would be:

from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader

filepath = 'C:\\Users\\Dave\\Desktop\\Bot\\bot_config_data.py'
module_data = os.path.basename(filepath)
spec = spec_from_loader(module_data, SourceFileLoader(module_data, filepath))
bot_config_data = module_from_spec(spec)
spec.loader.exec_module(bot_config_data)
price_data = bot_config_data.price_data

_"But, I would like to do the former."_ Why? What's wrong with the import statement? — Aran-Fey, Feb 27 '18 at 15:58
@Aran-Fey. It's not a good way to do it, although using `exec` is the real problem here. — Mad Physicist, Feb 27 '18 at 16:03
@Dave. I suspect that your textbook is not very good. Which one is it? — Mad Physicist, Feb 27 '18 at 16:03
'Mastering Object-orientated Programming' by Steven F.Lott. I suspect the mistake might be on my part. I'll double check. — Dave, Feb 27 '18 at 20:26
I think you need to strip off the extension and you probably don't need anything more complicated than load_module. — Mad Physicist, Feb 28 '18 at 15:31

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

You are completely off-base and the textbook is giving you some very bad advice.

exec runs Python code. As in Python statements, not the name of a file. From the docs:

exec(object[, globals[, locals]])

This function supports dynamic execution of Python code. object must be either a string or a code object. If it is a string, the string is parsed as a suite of Python statements which is then executed (unless a syntax error occurs)

That is why your second statement works fine. If you want to run a Python module, import it. If you need to import using a dynamic name instead of a hard-coded import statement, you can use importlib.import_module:

price_data = importlib.import_module('bot_config_data').price_data

This will run the whole import machinery for you, including ensuring that bot_config_data ends up in sys.modules.

If you really need something even fancier, you can use the __import__ machinery. __import__ is the under-the-hood implementation of the import statement:

d = {}
bot_config_data = __import__('bot_config_data', locals=d, from_list=['price_data'])
price_data = bot_config_data.price_data

If you want full control over the process, you can use the low-level machinery described here: How to import a module given the full path?. In particular see my answer to that question, since it describes how to load random text files as Python scripts: https://stackoverflow.com/a/43602557/2988730:

from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader 

spec = spec_from_loader("bot_config_data", SourceFileLoader("bot_config_data", "C:/Users/Dave/Desktop/Bot/bot_config_data.py"))
bot_config_data = module_from_spec(spec)
spec.loader.exec_module(bot_config_data)
price_data = bot_config_data.price_data

Final note: It looks like you blindly converted all the \\ to // in your path. Forward slashes do not need to be escaped, so you only need single forward slashes. If you want to avoid escaping your backslashes, put r in front of your string to make it a raw string.

The "bot_config_data" argument should be "bot_config_data.price_data". — Dave, Feb 28 '18 at 11:28
@Dave. Where? `bot_config_data` is a module, `price_data` is an attribute of that module. You'll end up having to load the whole module even if you want just the one attribute anyway. — Mad Physicist, Feb 28 '18 at 14:28
The arg to SourceFileLoader()? Otherwise, it throws an ImportError? — Dave, Feb 28 '18 at 14:37

Naysayer · Answer 2 · 2020-05-08T14:11:25.793

If you need to import using a dynamic name instead of a hard-coded import statement, you can use importlib.import_module:

price_data = importlib.import_module('bot_config_data').price_data

Huh? How is that "dynamic"? 'bot_config_data' is literally a string literal.

If you really want to import modules dynamically, contrary to what is widely believed, the only way to do this is with exec(). I challenge anyone to find or submit an example of importlib.import_module or __import__ being used to truly dynamically import modules.

To be clear: __import__ allows for the dynamic handling of imports, which is not the same as dynamically importing, as that is a chicken/egg type problem only exec() can solve.

The problem is that importlib.import_module() returns an object, which needs an assignment. (It doesn't just add to the global namespace as is often implied.) But you can't use the same argument on both sides of the assignment. So while it allows you to do handle imports in a more dynamic way, it provides no way of dynamically importing modules.

Using exec to load module variables, using a file path

2 Answers2

Linked