1

I have a fairly extensive coding background in multiple environments, but, am new to Python. I thought I had figured out a way to import functions dynamically from an external '.py' file, but am unsure if it is the best way. I found problems with using importlib.import_module() and importlib.__import__ based on my goal. Is there another way to accomplish what I am doing here? Effectively, I am wanting to have the same result as I would get when using from x import y where x and y are variables. I thought I would be able to use eval('from '+x+' import '+y) but this throws a syntax error.

I was hoping to accomplish this by contriving a dictionary with files as the keys (i.e., a file named 'file1.py' would create a key of 'file1') and a list of the desired functions as a list associated with its relative key. This can easily be built either literally, or by reading a path for file names and then using the dir() function to get a list of the functions in each individual file (among many other ways). Next, I hoped to simply use nested for loops to walk the dictionary keys and their associated key value lists and used eval('from '+key+' import '+currentListItem). Unfortunately, this throws a syntax error on the execution of the generated 'from...import...' statement. See below for example code. My problems with importlib (and getattr) is that I am unable to maintain the 'abstraction' provided by this method as I have to define a 'handle' in order to use importlib (i.e., handle = getattr(...) or handle = importlib.import_module(key) meaning that I basically have to hard code a 'handle name' for the given module being imported and thus may just as well hard code the 'from file_name import function' statements).

# simplistic example of what I was thinking....
# FILE file.py contains the code...
def asub(p1, p2 = None, p3 = None):
    print(p1, p2 if p2 else 'p2 defaulted', p3 if p3 else 'p3 defaulted')
    return

# FILE b.py contains the code...

#!/usr/local/bin/python3
subs = {'file':['asub']}
for key in subs:
    for subrt in subs[key]:
        print("==>", key, subrt)
        eval('from '+key+' import '+subrt)

# on execution of b.py (i.e., ```python3 b.py``` I get the following...

Traceback (most recent call last):
  File "b.py", line 9, in <module>
    eval('from '+key+' import '+subrt)
  File "<string>", line 1
    from file import asub
       ^
SyntaxError: invalid syntax

NOTE: I understand that this may not be the greatest thing for importing modules/functions as it tends to 'hide' the information that you would need for documentation and usage of the imported functions.

However, I am sure that there will be instances where this may be useful or where similar methodology may be useful in a case other than importing.

As I said, I am new to Python, so, am looking for feedback/guidance on this in a (cringe) 'Pythonic' sense.

One other thing: I get that eval() can open the door for code insertion but for the above specific use, and given that the files containing the functions are well locked down, I think it should be plenty safe (?)...

Thanks in advance for any feedback

DOQ
  • 13
  • 3
  • What sort of problems did you run with `importlib`? `importlib` is the way to go. You're calling eval, which is always slightly wonky, but since you do only import it should be fine. See this question https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path , which probably does what you want. – Radosław Cybulski Jun 19 '19 at 15:57
  • problem with importlib was that I could not keep the different functions 'abstracted' and then simply use them by name. i.e., I was not able to get it to work will keeping the abstraction/dynamics offered by the variable method I am using as I had to do something like ```handle = importlib.import_module(key)``` (only one for loop needed here) and then I had to use ```handle.f1()``` to use the desired function. Which wouldn't work for many different files/functions as a group. – DOQ Jun 19 '19 at 16:36
  • You can also use `f1 = getattr(handle, 'f1')` or `f1 = handle.f1` – Radosław Cybulski Jun 19 '19 at 16:38
  • Thanks Radoslaw, I looked at that also, but, if I do that I still have to hard code 'f1' which, again, removes the abstraction/dynamics provided by the discovery and variablization of the references. Is there some way to get the same effect as ```from file import function``` where 'file' and 'function' are variables using importlib? Since the importing will be completely variable based on the 'list' defining what to import, then you won't be able to define a useful 'handle' in an abstract way. Which means you might as well just hard code the imports. – DOQ Jun 19 '19 at 16:53
  • Apologies Radoslaw, the eval(from...import...) is not working after all.... ''' File "b.py", line 9, in eval('from '+key+' import '+subrt) File "", line 1 from a import asub''' so, back to square 1 – DOQ Jun 19 '19 at 18:08
  • Trivially, `eval` is for **expressions** and `exec` is for **statements** including all forms of `import`. That’s not to say you should be using either here! – Davis Herring Jun 19 '19 at 18:31
  • OP here, and trivially, 'exec' worked. Thank you very much Davis. Can you offer a suggestion on how I may approach this using importlib or another method and yet still maintain the ability to keep it 'dynamic'? As I stated in the post, if I have to define a correlated handle for each import, then that removes the dynamic aspect of this method. Thanks, again... – DOQ Jun 19 '19 at 18:37
  • Why do you care about having a named “handle” for the module? That’s no different from having a variable for a list being searched or a file being read. Also: do you plan to use the imported things by explicit name (implying that the selection is a set fixed when you write the code) or will you select among them with strings (for example, looking up items based on user input)? – Davis Herring Jun 19 '19 at 18:38
  • Hi Davis, this started as an educational exercise and the reason for not wanting to 'name' a handle is that I can't do that dynamically in correlation with dynamic imports. The idea here is that I would like to simply 'construct' a list of things to import (perhaps a common set of functions for scripts) and be able to refer to the imported functions easily without hard coding for each one. How I 'derive' the list of things to be imported may vary. For instance, I may read a directory for the .py files that contain the functions to be imported and iteratively do ```from filevar import *```. – DOQ Jun 19 '19 at 19:04

2 Answers2

0

Try this:

from importlib.machinery import SourceFileLoader

subs = {'file':['asub']}

for key in subs:
    # import module by file path
    # since path is relative, it will import against working directory
    mod = SourceFileLoader("", "%s.py" % key).load_module()

    for subrt in subs[key]:
        print("==>", key, subrt)

        # retrieve object from module by name subrt
        obj = getattr(mod, subrt)

        # this should call file.asub and it does
        obj('1', '2')

        # this add local variable with the same name
        locals()[subrt] = obj

# so now we can call asub as if it were done
# from file import asub
asub('3', '4')

This code prints:

==> file asub
1 2 p3 defaulted
3 4 p3 defaulted

This works on my python3 and requires python 3.3 at least. This is built on top of question How to import a module given the full path?. locals()[subrt] = obj puts imported object into local variables directory, which allows calling to it as if it were done from file import asub. If i understood correctly this is what you wanted to achieve?

Radosław Cybulski
  • 2,952
  • 10
  • 21
  • 1
    Thank you sir, that definitely helps. It was the 'locals()' variable setting that I think I was missing. Was unaware of that. And the deeper view into Python was just what I was hoping for with this. Your time and effort is much appreciated. I up voted this answer, but, I apparently do not have enough reputation points for it to be published, but, I have accepted it as 'the answer'. – DOQ Jun 20 '19 at 12:24
0

FYI, Final form based on accepted answer provided by Radoslaw. NOTE: I also tested with importlib.import_module(key) as well and that worked the same way except that you may have to use sys.path.insert(...) to make sure your module can be located.

import os, sys, re
from importlib.machinery import SourceFileLoader

# Build list of 'things' to be imported
#   (presumably via a lookup of some sort)
# I hard coded it here as a simple dictionary element for testing/dev.
# Notice that a fully qualified file name is used so that the target
#   subroutine/function can come from any specified/discovered file
# Another option may be to use only a list of file names here and then # use 'dir()' to get the functions from the file and then choose from
# the the list returned.  This is illustrated below.
subs = {'/some/fully/qualified/path/file.txt':['asub']}

# import the files
for key in subs:
    mod = SourceFileLoader("", key).load_module()

# Example of how to 'extract' the function names from the module for
# filtering and loading of explicit modules when using 'subs' as a
# list vs. a dictionary.  Here I simply excluded anything beginning
# and ending with double '_' and then processed those 'functions'.
#    for d in dir(mod):
#        if not re.match(r'^__.+?__$', d):
#            locals()[subrt] = getattr(mod, subrt)

# otherwise, if 'subs' is used as a dictionary as
# described above, just walk the list of functions
# built as part of the 'subs' dictionary
    for subrt in subs[key]:
        locals()[subrt] = getattr(mod, subrt)

# done
asub('parm1', 'p2_parm')

quit()

An interesting exercise. Thanks much for the help Radoslaw.

DOQ
  • 13
  • 3