Structuring python module so it can be run from a command line with and without -m switch

Question

Background

I'm working on Python module that consists of a number of of scripts. The end goal is to make functions in this module importable via import statement when working interactively and to make certain parts of module executable via command line arguments.

Desired outcome

Module can be run using:

python -m ./my_module --help
# No -m switch
python ./my_module --help

Structure

Following on this answer I would like to understand relationship that -m switch has to __main__.py and __init__.py files. The current structure is a follows

__main__.py

# Shebang line


###########
# Modules #
###########
import argparse
import logging
import tempfile


###################
# Package modules #
###################
from utilities import check_directory_access
# ...


#################
# Run functions #
#################
def run(args):
    """Run all functions with provided arguments"""
    # Start logging configuration
    # If file is not provided use temporary file
    if args.log_file is None:
        args.log_file = tempfile.NamedTemporaryFile(delete=False,
                                                    prefix='my_module',
                                                    suffix='.log').name
    # Create handlers: console
    # logging configuration
    logging.shutdown()


def main():
    """Process arguments and run main function"""
    parser = argparse.ArgumentParser(description='Dop stuff module',
                                     epilog='Epilog')
    parser.add_argument("-t", "--tables", nargs='+', dest='tables',
                        help="List tables to refresh", type=str)
    parser.add_argument("-l", "--log-file", dest='log_file',
                        type=str, help="Log file")
    parser.set_defaults(func=run)
    args = parser.parse_args()
    args.func(args)

if __name__ == "__main__":
    main()

__init__.py

###################
# Package modules #
###################
from .utilities import check_directory_access
# Other components

Problem

Running:

python -m my_module --help

Returns

ImportError: No module named 'utilities'

whereas

python my_module --help

works with no problem

Desired outcome

• Structring import in a manner that both statements python my_module and python -m my_module work.
• Not breaking import my_module when working interactively

• (bonus) Running without calling python interpreter first by ./my_module --help. I'm not sure how to do it with tree:

  |-- my_module
  |   |-- my_module.py
  |   |-- __init__.py
  |   |-- __main__.py
  |   |-- module_component_A.py
  |   |-- utilities.py
  

  Is there specific content that should go to my_module.py?

Answer 1

Python 3 does not have implied relative imports. Use absolute or explicit relative imports:

from .utilities import check_directory_access
from my_module.utilities import check_directory_access

This makes your package work with the -m switch. It also allows for import my_module in an interactive session.

---

A bare package stored as a folder cannot be executed directly. This is due to the operating system itself. You have to create an executable file that runs your package, if you wish to avoid calling python explicitly.

Either store the package as an executable zip file, or create a script that runs your package.

#!/usr/bin/env python3
import my_module.__main__

Note that the later requires your module to be installed or directly adjacent to your script. If your module is viable for installation, a console_scripts entry_point allows to have such a script created automatically.

You should not have the script reside inside your package - this requires you to change sys.path to a parent directory, which can lead to duplicate modules. For example, utilities.py would be available as the separate modules my_module.utilities and utilities.