Python 3: Large Application Structure

Question

EDITED to provide better clarity of question

Is there a recommended or standard way to organize a large python application -- as in, requires multiple files in different sub directories for a clearly organization project.

All the documentation I have been able to find talk about packages, which to my eye are what in any other language would be called a library. i.e. code that is included / required by the main program. The setup.py in a package adds to this confusion because it looks like it should be the main program/script for the entire application, but is used to "Install" the modules as a library instead.

In short, standards/recommendations for organizing code to be a application(a program that you actually run). Not a library/package(something which is included/used by an application)

possible duplicate of [How should I organize Python source code?](http://stackoverflow.com/questions/1849311/how-should-i-organize-python-source-code) — thegrinner, Jul 23 '13 at 13:59
nope, that's the kind of example that is exactly what I stated I don't want. Its modules and packages not applications. — Ryhnn, Jul 23 '13 at 14:01
A Python file *is* a module. Asking how to organize multiple Python files without using modules doesn't make sense. — interjay, Jul 23 '13 at 14:07
my understanding is a python file *can* be a module, as a module does not have to be directly runable. An application would require modules. All of the examples of python applications I look at seem to be packages, with setup.py being the main application script. This is what I'm trying to make sense of. — Ryhnn, Jul 23 '13 at 14:13
@Ryhnn Any app with setup.py will be a package. It doesn't mean it's not a standalone application at the same time. Package is just a form of getting a number of modules together, similar to a namespace. It doesn't mean the same thing as "library". Most likely you want to distribute your application as a package. — viraptor, Jul 23 '13 at 15:07
@viraptor I think you hit it, I'm thinking of packages in terms of libraries. Which is what they "feel" like to me. They just don't look like an executable application. The entry point for an application should be obvious and not burried in a sub directory. — Ryhnn, Jul 23 '13 at 15:18
@Ryhnn I mentioned it in the comment below, but this explains more about entry point definitions in setup.py http://pythonhosted.org/distribute/setuptools.html#automatic-script-creation - there is no "main" entry point in packages. There are just functions exposed through scripts. — viraptor, Jul 23 '13 at 15:24
thanks @viraptor that makes so much more sense now. All the documentation out there seems to skip right over this. It's been so frustrating, every page has been a big "yup here's another how to create a library" There has got to be a good book for people coming to python from other languages out there somewhere... — Ryhnn, Jul 23 '13 at 15:38
@Ryhnn: I went throught the same thing. All I found were Python eggs, and I didn't even know what the heck they were. Remember, just because the program you downloaded has a `setup.py` file doesn't mean the file is part of the actual program. Think Makefiles: all they do is build. — kirbyfan64sos, Jul 23 '13 at 21:58

Duncan · Answer 1 · 2013-07-23T15:11:51.987

One way (not the only one) to organize a large application is to use a combination of virtualenv and buildout.

virtualenv lets you create an isolated Python environment. This is important because any large Python application is going to depend on a bunch of packages (some you write yourself and others you just install from elsewhere). You don't want your application to require version 1.2.3 of a package when your system python already has version 3.2.1 installed, so you need to isolate your application from any other applications that are running.

buildout is useful because it lets you define all of the application's dependencies and build it from scratch repeatably.

The directory structure of a project using buildout looks like this:

project/
   bootstrap.py
   buildout.cfg
   .installed.cfg
   parts/
   develop-eggs/
   bin/
       buildout
       mypython
   eggs/
   downloads/

The bootstrap.py file is standard to a buildout and creates the bin folder containing a local python and the buildout command. bin\buildout then uses buildout.cfg (which is the only file you actually have to create) to download and install everything else your application requires. .installed.cfg holds a record of the commands that were run so if the buildout.cfg is updated and you re-run bin/buildout it can uninstall anything no longer required and install anything new.

Large Python projects such as Plone and Django use buildout, see http://jacobian.org/writing/django-apps-with-buildout/ for a good blog post describing it.

score 2 · Answer 2 · answered Jul 23 '13 at 14:58

Ok, so that setup.py file is an installer script, NOT the main program. Think of the setup.py file like a Window Installer.

Python files are called modules. Now, say you have an application that's split into several modules. You can do two things:

1.Place the additional modules in the same directory as the main script so they can be imported.

2.Create a seperate folder in the directory of the main script and put the additional modules there, along with an __init__.py script. This is a package.

Now, you're probably wondering what the heck a __init__.py file does. What it does is it tells Python how to import the modules in the folder. Here's a sample one:

import module1, module2'

__all__ = ['module1', 'module2']

The __all__ variable tells Python which modules can be imported from the outside. The imports actually tell Python which modules are imported when you import the package. The package name is the folder name. Say the folder is named hello. You'd import the package like this:

import hello

#Do stuff with hello

hello.module1.somefunction()

So, the imports in the init file tell Python which modules are imported when you import the package.

But, say you don't want to import all of hello. Say you only want module1. You'd do this:

from hello import module1

module1.somefunction()

That's what the __all__ variable is for. It tells Python what can be imported individually.

If you need further clarification, post a comment.

This was very well explained. Thank you! – WillJones Dec 09 '17 at 16:55 — WillJones, Dec 09 '17 at 16:55

score 1 · Answer 3 · answered Jul 23 '13 at 14:31

1

To clear up some confusion, setup.py is the installation script, not the main entry point. A Python application might include several ways of interacting with the "main entry points", including a couple of command-line scripts, a "main" entry point which can be executed with python -m name.of.package and a GUI.

If you need to include extra data that is generally included at the top level of the application:

your_project_root/
    data/
        additional/
        directories/
        and.files
    project_name/
        __init__.py
        package.py
        and.py
        module.py
        files.py
        go/
            __init__.py
            here.py
    setup.py
    README.rst
    # Other top-level stuff as needed

answered Jul 23 '13 at 14:31

Sean Vieira

155,703
32
311
293

Thanks, that clears up a lot. But, how does python know where the "main entry point" is? – Ryhnn Jul 23 '13 at 14:41
2

@Ryhnn python itself doesn't know anything about that. You just execute a file or module and that's it. But normally the setup script handles these kinds of things, e.g. by adding an appropriately-named bash script to your path which calls the python interpreter with the right module/file. – l4mpi Jul 23 '13 at 14:48
1

There's no "main" entry point. You can have as many auto-generated scripts which call your library as you want. Have a look at http://pythonhosted.org/distribute/setuptools.html#automatic-script-creation to learn how to define them. – viraptor Jul 23 '13 at 15:19

Python 3: Large Application Structure

3 Answers3