5

I created a nice python Twisted app with a plugin for the twistd runner, as specified in the Twisted Documentation: http://twistedmatrix.com/documents/current/core/howto/tap.html. I am having problems packaging this with PyInstaller: my twistd plugin is not found during execution of the frozen application.

To ship my project, I created my own top-level startup script using the twistd runner modules, e.g.

#!/usr/bin/env python
from twisted.scripts.twistd import run
from sys import argv
argv[1:] = [
  '--pidfile', '/var/run/myapp.pid',
  '--logfile', '/var/run/myapp.log',
  'myapp_plugin'
]
run()

Next, I use PyInstaller to freeze this as a single directory deployment. Executing the frozen script above fails as it cannot find my twistd plugin (edited for brevity):

~/pyinstall/dist/bin/mystartup?16632/twisted/python/modules.py:758:
UserWarning: ~/pyinstall/dist/mystartup?16632 (for module twisted.plugins)
not in path importer cache (PEP 302 violation - check your local configuration).

~/pyinstall/dist/bin/mystartup: Unknown command: myapp_plugin

Normally, Twistd inspects the Python system path to discover my plugin in twisted/plugins/myapp_plugin.py. If I print the list of twistd plugins in my startup script, the list is empty in the executable resulting from PyInstaller, e.g.

from twisted.plugin import IPlugin, getPlugins
plugins = list(getPlugins(IPlugin))
print "Twistd plugins=%s" % plugins

I use a somewhat default PyInstaller spec file, no hidden imports or import hooks specified.

I like the functionality of twistd with logging, pid files, etc, so I would like to avoid having to abandon the twistd runner altogether to circumvent the plugin issue. Is there a way to ensure my twistd plugin is found in the frozen executable?

Martijn Rutten
  • 763
  • 6
  • 14
  • Great question. Thanks for asking it. Packaging Twisted plugins is a sore spot - packaging tools often don't take it into account - and I am anxiously awaiting an answer from someone who knows more about pyinstaller :). – Glyph Apr 16 '12 at 22:14
  • Thanks. In the mean time I managed to run PyInstaller successfully on a twisted server python script that does not use twistd at all but simply starts the twisted reactor, e.g. `reactor.listenTCP(9999, site); reactor.run()`. So no packaging problems with twisted in general, only with the twistd plugin system. – Martijn Rutten Apr 17 '12 at 13:02
  • The path in the warning, `~/pyinstall/dist/mystartup?16632`, looks suspicious. I assume it is something pyinstaller creates - so pyinstaller is probably responsible for adding it to the right parts of the Python import config (eg, but not limited to, `sys.path`). What I really wonder is why the path has `~` in it. As you may know, that character has no special significance unless explicitly interpreted using `os.path.expanduser`. The path operations performed by `twisted.plugins` are all going to be confused by it. Does that path, with the literal `~`, appear in the import config somewhere? – Jean-Paul Calderone Apr 17 '12 at 13:26
  • ah, sorry for confusing you, I manually edited the error and replaced /home/martijn with '~', my bad – Martijn Rutten Apr 17 '12 at 15:52

1 Answers1

2

I found a workaround by reverse engineering some of the twisted code. Here I hardcode the plugin import. This works fine with PyInstaller for me.

#!/usr/bin/env python
import sys

from twisted.application import app
from twisted.scripts.twistd import runApp, ServerOptions

import myapp_plugin as myplugin


plug = myplugin.serviceMaker


class MyServerOptions(ServerOptions):
    """
    See twisted.application.app.ServerOptions.subCommands().
    Override to specify a single plugin subcommand and load the plugin
    explictly.
    """
    def subCommands(self):
        self.loadedPlugins = {plug.tapname:plug}
        yield (plug.tapname,
               None,
               # Avoid resolving the options attribute right away, in case
               # it's a property with a non-trivial getter (eg, one which
               # imports modules).
               lambda plug=plug: plug.options(),
               plug.description)

    subCommands = property(subCommands)


def run():
    """
    Replace twisted.application.app.run()
    To use our ServerOptions.
    """
    app.run(runApp, MyServerOptions)


sys.argv[1:] = [
    '--pidfile', '/var/run/myapp.pid',
    '--logfile', '/var/run/myapp.log',
    plug.tapname] + sys.argv[1:]

run()
Martijn Rutten
  • 763
  • 6
  • 14