How to load a dylib-file as CPython-extension?

Question

This has been asked before (e.g. here), but the given solution (i.e. renaming file to *.so) is not acceptable. I have a CPython extension called name.dylib, which cannot be imported. If the filename is changed to use name.so it is imported correctly. Changing the filename is not an option**, and should not be necessary.

Python has a lot of hooks for searching for modules, so there must be a way to make it recognise a dylib-file. Can someone show how to do this? Using a low level import which spells out the whole filename is not nice but is an acceptable solution.

** because the build code forces dylib, and, other contexts I have assume it. The extension module is dual purpose, it can by used both as an ordinary shared library and a Python extension. Using a symlink does work, but is a last resort because it requires manual intervention in an automated process.

ead · Answer 1 · 2020-01-30T22:16:24.770

You could manipulate sys.path_hooks and replace FileFinder-hook with one which would accept .dylib-extensions. But see also the simpler but less convinient alternative which would import given the full file-name of the extension.

More information how .so, .py and .pyc files are imported can be found for example in this answer of mine.

This manipulation could look like following:

import sys
import importlib
from importlib.machinery import FileFinder, ExtensionFileLoader

# pick right loader for .dylib-files:
dylib_extension = ExtensionFileLoader, ['.dylib']

# add dylib-support to file-extension supported per default
all_supported_loaders = [dylib_extension]+ importlib._bootstrap_external._get_supported_file_loaders()


# replace the last hook (i.e. FileFinder) with one recognizing `.dylib` as well:
sys.path_hooks.pop()
sys.path_hooks.append(FileFinder.path_hook(*all_supported_loaders))

#and now import name.dylib via
import name

This must be the first code executed, when python-script starts to run. Other modules might not expect sys.path_hooks being manipulated somewhere during the run of the program, so there might be some problems with other modules (like pdb, traceback and so). For example:

import pdb

#above code

import name

will fail, while

#above code

import pdb

import name

will work, as pdb seems to manipulate the import-machinery.

Normally, FileFinder-hook is the last in the sys.path_hooks, because it is the last resort, once path_hook_for_FileFinder is called for a path, the finder is returned (ImportError should be raised if PathFinder should look at further hooks):

def path_hook_for_FileFinder(path):
        """Path hook for importlib.machinery.FileFinder."""
        if not _path_isdir(path):
            raise ImportError('only directories are supported', path=path)
        return cls(path, *loader_details) # HERE Finder is returned!

However, one might want to be sure and check, that really the right hook is replaced.

A simpler alternative would be to use imp.load_dynamic (neglecting for the moment that imp is depricated):

import imp
imp.load_dynamic('name', 'name.dylib') # or what ever path is used

That might be more robust than the first solution (no problems with pdb for example) but less convinient for bigger projects.

How to load a dylib-file as CPython-extension?

1 Answers1