7

Does anybody know if there is any tool for packing a Python project that uses several files and modules into a single script?

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
Dmitry Trofimov
  • 7,371
  • 1
  • 30
  • 34
  • 2
    What's the purpose of packing into a single .py script as stated in the title? If you need to combine (python itself + third party modules + code) for installation purposes or other: look into `pyinstaller`... – ChristopheD Dec 06 '10 at 15:41
  • I need to be able to copy only one single file and then import it or execute. Sometimes it is much more convenient to copy a single file instead of copying whole project tree. – Dmitry Trofimov Dec 06 '10 at 15:43
  • Perhaps possible (if you e.g. abused classes as namespaces and if your code isn't too clever/hacky with regards to scope/namespaces/import internals), but certainly harmful for development and not needed for deployment. So I expect that nobody bothered making such a tool. –  Dec 06 '10 at 15:51
  • 2
    If you need to have only one file, you can have a look at the zipimport (http://docs.python.org/library/zipimport.html) module. Using that you can essentially do `export PYTHONPATH=mymodules.zip; python -m startmodule` That is probably a prettier solution than lumping everything together in one big file. – Mattias Nilsson Dec 06 '10 at 15:57
  • What's wrong with copying or syncing a whole directory tree? If you `rsync` it you get the added benefit of only needing to copy new(er) files. This can surely be automated quite easily... – ChristopheD Dec 06 '10 at 16:00

3 Answers3

5

make the .egg file and install it or put it on pythonpath may solve your problem.
similar fellow

Community
  • 1
  • 1
shahjapan
  • 13,637
  • 22
  • 74
  • 104
5

Save this as python_header.py:

#!/bin/env/python
# -*- coding: ascii -*-
import os
import sys
import imp
import tarfile
import tempfile


RUN_MODULE = "__run__"
SENTINEL = 'RzlBTXhya3ljIzl6PFFkQiRKLntEdHF+c2hvWid0IX5NVlxWd' \
           'FxcJ0NWQ2xKVUI0TVEuNl0rWUtnKiRr'.decode('base64')


class FileOffset(object):
    def __init__(self, fileobj, offset=0):
        self._fileobj = fileobj
        self._offset = offset
        self._fileobj.seek(offset)

    def tell(self):
        return self._fileobj.tell() - self._offset

    def seek(self, position, whence=os.SEEK_SET):
        if whence == os.SEEK_SET:
            if position < 0: raise IOErrror("Negative seek")
            self._fileobj.seek(position + self._offset)
        else:
            oldposition = self._fileobj.tell()
            self._fileobj.seek(position, whence)
            if self._fileobj.tell() < self._offset:
                self._fileobj.seek(oldposition, os.SEEK_SET)
                raise IOError("Negative seek")

    def __getattr__(self, attrname):
        return getattr(self._fileobj, attrname)

    def __enter__(self, *args):
        return self._fileobj.__enter__(*args)

    def __exit__(self, *args):
        return self._fileobj.__exit__(*args)


class TarImport(object):
    def __init__(self, tarobj, tarname=None):
        if tarname is None:
            tarname = '<tarfile>'
        self._tarname = tarname
        self._tarobj = tarobj

    def find_module(self, name, path=None):
        module_path = os.path.join(*name.split('.'))
        package_path = os.path.join(module_path, '__init__')

        for path in [module_path, package_path]:
            for suffix, mode, module_type in imp.get_suffixes():
                if module_type != imp.PY_SOURCE:
                    continue
                member = os.path.join(path) + suffix
                try:
                    modulefileobj = self._tarobj.extractfile(member)
                except KeyError:
                    pass
                else:
                    return Loader(name, modulefileobj,
                                  "%s/%s" % (self._tarname, member),
                                  (suffix, mode, module_type))


class Loader(object):
    def __init__(self, name, fileobj, filename, description):
        self._name = name
        self._fileobj = fileobj
        self._filename = filename
        self._description = description

    def load_module(self, name):
        imp.acquire_lock()
        try:
            module = sys.modules.get(name)
            if module is None:
                module = imp.new_module(name)

            module_script = self._fileobj.read()
            module.__file__ = self._filename
            module.__path__ = []
            sys.modules[name] = module
            exec(module_script, module.__dict__, module.__dict__)
        finally:
            imp.release_lock()

        return module


def find_offset(fileobj, sentinel):
    read_bytes = 0
    for line in fileobj:
        try:
            offset = line.index(sentinel)
        except ValueError:
            read_bytes += len(line)
        else:
            return read_bytes + offset + len(sentinel)
    raise ValueError("sentinel not found in %r" % (fileobj, ))


if __name__ == "__main__":
    sys.argv[:] = sys.argv[1:]
    archive_path = os.path.abspath(sys.argv[0])
    archive_offset = find_offset(open(archive_path), SENTINEL)

    archive = FileOffset(open(archive_path), archive_offset)

    tarobj = tarfile.TarFile(fileobj=archive)
    importer = TarImport(tarobj, archive_path)

    sys.meta_path.insert(0, importer)

    importer.find_module(RUN_MODULE).load_module(RUN_MODULE)

Save this as sh_header.sh:

#!/bin/sh

head -n @@TO@@ "$0" | tail -n +@@FROM@@ | python - "$0"

exit $?

Save this as create_tarred_program.py:

#!/usr/bin/env python
# -*- coding: latin-1 -*-

import sys
import imp
import shutil

sh_filename, runner_filename, tar_archive, dst_filename = sys.argv[1:]

runner = imp.load_module("tarfile_runner",
                        open(runner_filename, 'U'),
                        runner_filename,
                        ('.py', 'U', imp.PY_SOURCE))



sh_lines = open(sh_filename, 'r').readlines()
runner_lines = open(runner_filename, 'r').readlines()

sh_block = ''.join(sh_lines)
runner_block = ''.join(runner_lines)

if runner.SENTINEL in runner_block or runner.SENTINEL in sh_block:
    raise ValueError("Can't have the sentinel inside the runner module")
if not runner_block.endswith('\n') or not sh_block.endswith('\n'):
    raise ValueError("Trailing newline required in both headers")

to_pos = len(sh_lines) + len(runner_lines)
from_pos = len(sh_lines) + 1

sh_block = sh_block.replace("@@TO@@", str(to_pos))
sh_block = sh_block.replace("@@FROM@@", str(from_pos))


dst = open(dst_filename, 'wb')

dst.write(sh_block)
dst.write(runner_block)
dst.write(runner.SENTINEL)

shutil.copyfileobj(open(tar_archive, 'rb'), dst)

dst.flush()
dst.close()

Create a tar archive with your packages named packages.tar. The main module should be called __run__.py, you should never import __main__. Run:

create_tarred_program.py sh_header.sh python_header.py packages.tar program.sh

Distrubute program.sh.

It's possible to avoid dependency on /bin/sh by an extended first line, but it still won't work on anything but *nix, so there's no point in it.

Rosh Oxymoron
  • 20,355
  • 6
  • 41
  • 43
-2

I would just do this: https://github.com/fgarcia/snips/tree/master/pydeploy

SystematicFrank
  • 16,555
  • 7
  • 56
  • 102