Import Python module based on argument

Question

I'm trying to dynamically import a python-based SQL query module from a sub-folder, and that folder is obtained by using the argparse module.

My project structure :

main_file.py
Data_Projects/
   ITA_data/
      __init__.py
      sqlfile.py
   UK_data/
      __init__.py
      sqlfile.py

Within main_file.py is the argparse module with the argument 'dir' containing the location of the specified directory i.e.

parser.add_argument('--dir', default='Data_Projects/ITA_data/', type=str,
                        help="set the data directory")

My understanding thus far is that modules should be imported at the top and to import just one sql query I would use: from Data_Project.ITA_data import sqlfile

I know I can't set the import statement after the args have been defined, so how can I keep the format correct with the imports at the top, and yet retrospectively update this with the arguments that get defined afterwards?

Many thanks.

UPDATE

Thanks to the below answer. I've now tried to assign :

sqlfile = __import__(in_arg.dir + 'sqlfile.py')

However I'm getting the following error:

*** ModuleNotFoundError: No module named 'Data_Projects/ITA_data/sqlfile'

I've tried using things like

os.path.join(Path(__file__).resolve().parents[0], in_arg.dir + 'sqlfile')

If it helps, when I try just :

__import__('Data_Projects') - works fine
__import__('Data_Projects/ITA_data') - doesn't work - ModuleNotFound

And as a check to verify I'm not crazy:

os.path.exists('Data_Projects/ITA_Data/sqlfile.py') >>> True
os.path.exists(in_arg.dir + 'sqlfile.py') >>> True

There is no limitation that the import must be done at the top, you can import a module at the scope (before) it is used. — Johan van Breda, Dec 04 '18 at 07:45

score 3 · Accepted Answer · answered Dec 04 '18 at 07:47

I don't see anything wrong with

import argparse
parser = ...
parser.add_argument('data', choices=['UK', 'ITA'])
args = parser.parse_args()
if args.dir == 'UK':
    import UK_data as data
elif args.dir == 'ITA':
    import ITA_data as data
else ...

You could refine this with functions and __name__ etc. But a conditional import is ok, just so long as it occurs before the data module is used.

grapes · Answer 2 · 2018-12-04T10:02:32.353

2

You can use __import__(filename: str) function instead of import statement. It does the same:

# option 1
import foo as bar
# option 2
bar = __import__('foo')

If you need to import from aside, you need to add your directory to module search paths. There are several ways to achieve that, depending on your version of Python. You can find them all in great post:

How to import a module given the full path?

edited Dec 04 '18 at 10:02

answered Dec 04 '18 at 07:39

grapes

8,185
1
19
31

Thanks - can I add bar after the args have been defined then, effectively having import statements in the middle of my module? – ML_Engine Dec 04 '18 at 07:41
1

You can put this statement wherever you want – grapes Dec 04 '18 at 07:44
I've updated my answer, would you be able to help again? Thanks :) – ML_Engine Dec 04 '18 at 09:58

score 0 · Answer 3 · answered Dec 04 '18 at 10:15

The issue was resolved by using :

import sys
sys.path.insert(0, os.getcwd() + "/" + in_arg.dir)

This sets the PYTHONPATH variable to include the directory I want (which changes depending on the argument) to use to search for the file.

From there using grapes help it was a case of doing:

sqlfile = __import__('sqlfile')

And from there I could use the variable to perform the relevant sql query.

Import Python module based on argument

3 Answers3