How to dynamically add and load entry points?

Question

I am developing a slack bot with plugins using entry points. I want to dynamically add a plugin during runtime.

I have a project with this structure:

+ ~/my_project_dir/
    + my_projects_python_code/
    + plugins/
        - plugin1.py
        - plugin2.py
        - ...
        - pluginN.py
    - setup.py
    - venv/
    - install.sh

My setup.py file looks like this:

from setuptools import setup, find_packages

setup(
    name="My_Project_plugins",
    version="1.0",
    packages=['plugins'],
    entry_points="""
        [my_project.plugins]
        plugin1 = plugins.plugin1:plugin1_class
        plugin2 = plugins.plugin2:plugin2_class
        ...
        pluginN = plugins.pluginN:pluginN_class
    """
        )

Running sudo install.sh does the following:

1. Copies the needed files to /usr/share/my_project_dir/

2. Activate virtualenv at /usr/share/my_project_dir/venv/bin/activate

3. Run: python setup.py develop

This works as expected and sets up my entry points correctly so that I can use them through the bot.

But I want to be able to add a plugin to setup.py and be able to use it while the bot is running. So I want to add a line: pluginN+1 = plugins.pluginN+1:pluginN+1_class and have pluginN+1 available to use.

What I've tried/learned:

• After /usr/share/my_project_dir/venv/bin/activate I open a Python interactive shell and iterate through pkg_resources.iter_entry_points(), which lists everything that was loaded from the initial state of setup.py (i.e. plugin1 through pluginN)

• If I add a line to setup.py and run sudo python setup.py develop and iterate again with the same Python shell, it doesn't pick up the new plugin but if I exit the shell and reopen it, the new plugin gets picked up.

• I noticed that when I install the bot, part of the output says:

  • Copying My_Project_plugins-1.0-py2.7.egg to /usr/share/my_project-dir/venv/lib/python2.7/site-packages

• When I cd /usr/share/my_project_dir/, activate my virtualenv, and run setup.py from the shell it says:

  • Creating /usr/local/lib/python2.7/dist-packages/My_Project-plugins.egg-link (link to .) My_Project-plugins 1.0 is already the active version in easy-install.pth

Answer 1

I needed to do something similar to load a dummy plugin for test purposes. This differs slightly from your use-case in that I was specifically trying to avoid needing to define the entry points in the package (as it is just test code).

I found I could dynamically insert entries into the pkg_resources data structures as follows:

import pkg_resources
# Create the fake entry point definition
ep = pkg_resources.EntryPoint.parse('dummy = dummy_module:DummyPlugin')

# Create a fake distribution to insert into the global working_set
d = pkg_resources.Distribution()

# Add the mapping to the fake EntryPoint
d._ep_map = {'namespace': {'dummy': ep}}

# Add the fake distribution to the global working_set
pkg_resources.working_set.add(d, 'dummy')

This, at run time, added an entry point called 'dummy' to 'namespace', which would be the class 'DummyPlugin' in 'dummy_module.py'.

This was determined through use of the setuptools docs, and dir() on the objects to get more info as needed.

Docs are here: http://setuptools.readthedocs.io/en/latest/pkg_resources.html

You might especially look at http://setuptools.readthedocs.io/en/latest/pkg_resources.html#locating-plugins if all you need to do is load a plugin that you have just stored to your local filesystem.

Answer 2

It's more than at least 5 years, since the time when I first asked myself almost the same question, and your question now is an impulse to finally find it out.

For me it was as well interesting, if one can add entry points from the same directory as the script without installation of a package. Though I always knew that the only contents of the package might be some meta with entry points looking at some other packages.

Anyway, here is some setup of my directory:

ep_test newtover$ tree
.
├── foo-0.1.0.dist-info
│   ├── METADATA
│   └── entry_points.txt
└── foo.py

1 directory, 3 files

Here is the contents of foo.py:

ep_test newtover$ cat foo.py
def foo1():
    print 'foo1'

def foo2():
    print 'foo2'

Now let's open ipython:

In [1]: def write_ep(lines):  # a helper to update entry points file
   ...:     with open('foo-0.1.0.dist-info/entry_points.txt', 'w') as f1:
   ...:         print >> f1, '\n'.join(lines)
   ...:        

In [2]: write_ep([  # only one entry point under foo.test
   ...: "[foo.test]",
   ...: "foo_1 = foo:foo1",
   ...: ])

In [3]: !cat foo-0.1.0.dist-info/entry_points.txt
[foo.test]
foo1 = foo:foo1

In [4]: import pkg_resources

In [5]: ws = pkg_resources.WorkingSet()  # here is the answer on the question

In [6]: list(ws.iter_entry_points('foo.test'))
Out[6]: [EntryPoint.parse('foo_1 = foo:foo1')]

In [7]: write_ep([  # two entry points
   ...: "[foo.test]",
   ...: "foo_1 = foo:foo1",
   ...: "foo_2 = foo:foo2"
   ...: ])

In [8]: ws = pkg_resources.WorkingSet()  # a new instance of WorkingSet

With default parameters WorkingSet just revisits each entry in sys.path, but you can narrow the list. pkg_resources.iter_entry_points is bound to a global instance of WorkingSet.

In [9]: list(ws.iter_entry_points('foo.test'))  # both are visible
Out[9]: [EntryPoint.parse('foo_1 = foo:foo1'), EntryPoint.parse('foo_2 = foo:foo2')]

In [10]: foos = [ep.load() for ep in ws.iter_entry_points('foo.test')]

In [11]: for func in foos: print 'name is {}'.format(func.__name__); func()
name is foo1
foo1
name is foo2
foo2

And the contents of METADATA as well:

ep_test newtover$ cat foo-0.1.0.dist-info/METADATA
Metadata-Version: 1.2
Name: foo
Version: 0.1.0
Summary: entry point test

UPD1: I thought it over once again and now understand that you need an additional step before using the new plugins: you need to reload the modules.

This might be as easy as:

In [33]: modules_to_reload = {ep1.module_name for ep1 in ws.iter_entry_points('foo.test')}

In [34]: for module_name in modules_to_reload:
   ....:     reload(__import__(module_name))
   ....:

But if a new version of your plugins package is based on significant changes in other used modules, you might need a particular order of reloading and reloading of those changed modules. This might become a cumbersome task, so that restarting the bot would be the only way to go.

Answer 3

I had to change a bit solution by @j3p0uk to work for me. I wanted to use this inside an unit test (unittest framework). What I did is:

def test_entry_point(self):
    distribution = pkg_resources.Distribution(__file__)
    entry_point = pkg_resources.EntryPoint.parse('plugin1 = plugins.plugin1:plugin1_class', dist=distribution)
    distribution._ep_map = {'my_project.plugins': {'plugin1': entry_point}}
    pkg_resources.working_set.add(distribution)

This made also entry_point.load() work for me the in the code which is being tested, to really load the symbol referenced by the entry point. In my test I also have my_project.plugins having the name of the test file, and then the symbol to load is at the global scope of that file.