5

I am managing a quite large python code base (>2000 lines) that I want anyway to be available as a single runnable python script. So I am searching for a method or a tool to merge a development folder, made of different python files into a single running script.

The thing/method I am searching for should take code split into different files, maybe with a starting __init___.py file that contains the imports and merge it into a single, big script.

Much like a preprocessor. Best if a near-native way, better if I can anyway run from the dev folder.

I have already checked out pypp and pypreprocessor but they don't seem to take the point.

Something like a strange use of __import__() or maybe a bunch of from foo import * replaced by the preprocessor with the code? Obviously I only want to merge my directory and not common libraries.

Update What I want is exactly mantaining the code as a package, and then being able to "compile" it into a single script, easy to copy-paste, distribute and reuse.

Filippo Valsorda
  • 938
  • 1
  • 10
  • 16
  • 1
    The "right" way to do this is to put all the separate bits into a [package](http://docs.python.org/tutorial/modules.html#packages) which you can then import. If you just want a quick, hacky way then you can write a script that does `from foo import *`. Why do they all have to be in one file? – Katriel Mar 24 '12 at 11:03
  • http://docs.python.org/tutorial/modules.html#packages – Aprillion Mar 24 '12 at 11:05

2 Answers2

5

It sounds like you're asking how to merge your codebase into a single 2000-plus source file-- are you really, really sure you want to do this? It will make your code harder to maintain. Python files correspond to modules, so unless your main script does from modname import * for all its parts, you'll lose the module structure by converting it into one file.

What I would recommend is leaving the source structured as they are, and solving the problem of how to distribute the program:

  1. You could use PyInstaller, py2exe or something similar to generate a single executable that doesn't even need a python installation. (If you can count on python being present, see @Sebastian's comment below.)

  2. If you want to distribute your code base for use by other python programs, you should definitely start by structuring it as a package, so it can be loaded with a single import.

  3. To distribute a lot of python source files easily, you can package everything into a zip archive or an "egg" (which is actually a zip archive with special housekeeping info). Python can import modules directly from a zip or egg archive.

Community
  • 1
  • 1
alexis
  • 48,685
  • 16
  • 101
  • 161
  • 2
    +1 for *distribute* part. btw, python can also execute ordinary (not eggs) zip files e.g., if there is `__main__.py` file. – jfs Mar 24 '12 at 11:32
  • Cool, I'd forgotten all about `__main__.py`. Yet another way to distribute it as a stand-alone script, then (but requiring python). – alexis Mar 24 '12 at 11:39
  • Yeah, I want to keep it as a package for maintainance but I wanted the "single compiled" file for distribution, as it is the simplest way to distribute a python script, as it is immediately recognized by all platforms, both for running and for importing. I like @J.F.Sebastian idea, but try executing a zip on the command line, I need to look into it. – Filippo Valsorda Mar 24 '12 at 17:44
  • A zip file doesn't begin with `#!...python`' and must end with .zip to be recognized by python. So it doesn't make a very good command name. But you can distribute it with a bash alias or one-line shell script that just runs `python -m mypackage.zip "$@"` – alexis Mar 24 '12 at 22:38
  • Actually, a zip file CAN start with a shebang and python DOES recognize it without extension, so it is a perfect replacement for a script!! – Filippo Valsorda Mar 25 '12 at 10:09
  • I obviously have a lot to learn about zipped archive support in python. @Filo, can you elaborate or provide a reference? Nothing of the sort is implied by the documentation. [http://docs.python.org/using/cmdline.html] – alexis Mar 25 '12 at 10:54
  • I am surprised (and amazed), too. I actually found the shebang suggestion looking around here on stackoverflow.com and discovered the auto-recognition simply trying it. That's an awesome feature; I will search more reference or look at the source and write about it later today. – Filippo Valsorda Mar 25 '12 at 13:18
  • Here is some incomplete reference http://www.voidspace.org.uk/python/weblog/arch_d7_2008_12_06.shtml#e1038 – Filippo Valsorda Mar 26 '12 at 06:15
  • 1
    Update: I looked into it and the zip format does indeed allow other stuff to be mixed with zipped content in the same file, so the hybrid is a valid zip archive (though linux unzip gives a warning). But python 2.5 or older does not support reading from zipped files, even pure ones that end in .zip. Also, I _still_ can't find any mention of detecting zip format in files with names that don't end in .zip, even in python 3 documentation, so I wonder just how much this feature can be relied on. In short, I wouldn't adopt this as a distribution format just yet. – alexis Mar 26 '12 at 19:04
  • I read from msg55338 "support automatically importing \_\_main\_\_ when sys.argv[0] is an importable path. This allows zip files, directories, and any future importable locations (e.g. URLs) to be used on the command line." So the interpreter doesn't care about the filename, if it is importable, the import it and run \_\_main\_\_. I think that makes it quite stable! – Filippo Valsorda May 01 '12 at 14:11
  • Cool, so that's where execution functionality for zip files came from. But (repeating my last comment): I _still_ can't find any mention of detecting zip format in files with names that do _not_ end in .zip. That's the really surprising feature. – alexis May 01 '12 at 15:35
2

waffles seems to do exactly what you're after, although I've not tried it

You could probably do this manually, something like:

# file1.py
from .file2 import func1, func2
def something():
    func1() + func2()

# file2.py
def func1(): pass
def func2(): pass

# __init__.py
from .file1 import something
if __name__ == "__main__":
    something()

Then you can concatenate all the files together, removing any line starting with from ., and.. it might work.

That said, an executable egg or regular PyPI distribution would be much simpler and more reliable!

dbr
  • 165,801
  • 69
  • 278
  • 343