17

I have a pure C module for Python and I'd like to be able to invoke it using the python -m modulename approach. This works fine with modules implemented in Python and one obvious workaround is to add an extra file for that purpose. However I really want to keep things to my one single distributed binary and not add a second file just for this workaround.

I don't care how hacky the solution is.

If you do try to use a C module with -m then you get an error message No code object available for <modulename>.

Roger Binns
  • 3,203
  • 1
  • 24
  • 33
  • What tools do you use, what are you trying to achieve, what is the platform? How much of a standard library do you use? Is the module yours and closed-source, or something we can look at? I cannot help without some information. – janislaw May 31 '11 at 14:18
  • Thanks for the enthusiasm but the issue is far deeper than your questions indicate. The module is written in C using the standard Python approaches for doing that. Start at this page to get an idea: http://docs.python.org/extending/extending.html The "tools" are standard Python mechanism for C extensions, what I am trying to achieve is in the description, all platforms, standard library is not relevant (code is in C not Python), module is mine and open source and sufficiently complex that I recommend using the example module from the python doc page. – Roger Binns May 31 '11 at 17:56
  • Have you ever produced a frozen executable in Python? Depending on the platform, it bundles with all the executable code it needs to run. I once ported a python program to linux, and had to ship libglib and libz along the frozen executable. On Windows, OTOH there is msvcrtxx.dll that you may need to ship as well. Even if you write C program with Python.lib linked statically, you'd need to attach dynamic libraries, which defeats single-file purpose. – janislaw Jun 03 '11 at 20:31
  • 2
    Yes, for Windows, Linux and Mac. This is not remotely the same thing. My module is an ordinary module and can be used as such. It also includes a shell. Currently to get the shell you have to do "python -c 'import module;module.main()' " whereas I want it do be "python -m module". – Roger Binns Jun 11 '11 at 03:15
  • 1
    Is there some obvious reason you prefer to distribute a single (presumably) precompiled shared library for your users to install on their own rather than using [distutils](http://docs.python.org/distutils/) or similar? I just about won't ever install any python module, no matter how useful, unless it can be installed with pip or easy_install. – SingleNegationElimination Oct 30 '11 at 21:12
  • It can be compiled by regular distutils from C source and there are no ulterior motives, other than it is so much cleaner as a single file. – Roger Binns Nov 04 '11 at 04:08

4 Answers4

7

-m implementation is in runpy._run_module_as_main . Its essence is:

mod_name, loader, code, fname = _get_module_details(mod_name)
<...>
exec code in run_globals

A compiled module has no "code object" accociated with it so the 1st statement fails with ImportError("No code object available for <module>"). You need to extend runpy - specifically, _get_module_details - to make it work for a compiled module. I suggest returning a code object constructed from the aforementioned "import mod; mod.main()": (python 2.6.1)

    code = loader.get_code(mod_name)
    if code is None:
+       if loader.etc[2]==imp.C_EXTENSION:
+           code=compile("import %(mod)s; %(mod)s.main()"%{'mod':mod_name},"<extension loader wrapper>","exec")
+       else:
+           raise ImportError("No code object available for %s" % mod_name)
-       raise ImportError("No code object available for %s" % mod_name)
    filename = _get_filename(loader, mod_name)

(Update: fixed an error in format string)

Now...

C:\Documents and Settings\Пользователь>python -m pythoncom

C:\Documents and Settings\Пользователь>

This still won't work for builtin modules. Again, you'll need to invent some notion of "main code unit" for them.

Update:

I've looked through the internals called from _get_module_details and can say with confidence that they don't even attempt to retrieve a code object from a module of type other than imp.PY_SOURCE, imp.PY_COMPILED or imp.PKG_DIRECTORY . So you have to patch this machinery this way or another for -m to work. Python fails before retrieving anything from your module (it doesn't even check if the dll is a valid module) so you can't do anything by building it in a special way.

ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
  • Thanks for the detailed answer. I was hoping I could pull some stunt in my C code that returns the module attaching a code object there. – Roger Binns Dec 06 '11 at 19:50
  • From management POV, a wrapper script looks the best solution to me. It's easy to name and place it so that it is called instead of python.exe whenever a user invokes "python". – ivan_pozdeev Dec 07 '11 at 13:58
  • The problem with anything other than `python -m module' is that I have to distribute more than one file, and then the user has to figure out where to put that second file and be able to execute it. Consider different permissions (user versus admin), different platforms,a distinction between developers and users for my module etc. It is just a pity that Python doesn't let -m work with C extensions and there isn't even a disgusting hack that will do the trick. – Roger Binns Dec 09 '11 at 04:18
  • 2
    It's almost 8 years since this answer was created. Has anyone know if I still need to make a workaround to execute cythonized scripts with the "-m" flag? ie. `python -m app.module`? – HereHere Oct 15 '19 at 16:46
0

Does your requirement of single distributed binary allow for the use of an egg? If so, you could package your module with a __main__.py with your calling code and the usual __init__.py...

If you're really adamant, maybe you could extend pkgutil.ImpLoader.get_code to return something for C modules (e.g., maybe a special __code__ function). To do that, I think you're going to have to actually change it in the Python source. Even then, pkgutil uses exec to execute the code block, so it would have to be Python code anyway.

TL;DR: I think you're euchred. While Python modules have code at the global level that runs at import time, C modules don't; they're mostly just a dict namespace. Thus, running a C module doesn't really make sense from a conceptual standpoint. You need some real Python code to direct the action.

nfirvine
  • 1,479
  • 12
  • 23
  • I don't believe eggs support compiled code. I could only extend pkgutil while my module is loading since that is the earliest the code runs, but it would be too late. It is really irritating that currently the solution is `python -c "import module;module.main()"' or having a .py file with just those 4 words. – Roger Binns Nov 04 '11 at 04:03
-2

I think that you need to start by making a separate file in Python and getting the -m option to work. Then, turn that Python file into a code object and incorporate it into your binary in such a way that it continues to work.

Look up setuptools in PyPi, download the .egg and take a look at the file. You will see that the first few bytes contain a Python script and these are followed by a .ZIP file bytestream. Something similar may work for you.

Michael Dillon
  • 31,973
  • 6
  • 70
  • 106
  • 1
    Getting Python bytecode is trivial - Py_CompileString will get that. However I see no way to attach the bytecode to the PyObject returned from Py_InitModule3/PyModule_Create. – Roger Binns Jun 01 '11 at 06:47
-2

There's a brand new thing that may solve your problems easily. I've just learnt about it and it looks preety decent to me: http://code.google.com/p/pts-mini-gpl/wiki/StaticPython

janislaw
  • 300
  • 1
  • 7
  • 1
    Not remotely applicable. I want my module to work with the existing installed Python not have a separately built one. – Roger Binns Aug 08 '11 at 12:49