Python - Packaging Alembic Migrations with Setuptools

Question

What is the right way to package Alembic migration files in a Setuptools setup.py file? Everything is in my repo root as alembic/.

This is a Python application, not a library.

My desired installation flow is that someone can pip install the wheel that is my application. They would then be able to initialize the application database by running something like <app> alembic upgrade --sqlalchemy.url=<db_url>. Upgrades would then require a pip install -U, after which they can run the Alembic command again.

Is this unorthodox?

If not, how would I accomplish this? Certainly a console_scripts entry_points. But beyond that?

Answer 1

I am not sure this is the right way but I did it this way:

First, you can add sort of custom options to alembic using the -x option and you can find details explained in this great answer. This allows you to specify the db_url at runtime and make it override the value in the config.ini.

Then I managed to package alembic and my migrations by moving the alembic.ini file and the alembic directory from my project root to my top-level python package:

<project root>
├── src
│   └── <top-level package dir>
│       ├── alembic
│       │   ├── env.py
│       │   ├── README
│       │   ├── script.py.mako
│       │   └── versions
│       │       ├── 58c8dcd5fbdc_revision_1.py
│       │       └── ec385b47da23_revision_2.py
│       ├── alembic.ini
│       ├── __init__.py
│       └── <other files and dirs>
└── <other files and dirs>

This allows to use the setuptools package_data directive inside my setup.py:

setup(
    name=<package_name>,
    package_dir={'': 'src'},
    packages=find_packages(where='src'),
    package_data={
        '<top-level package dir>': ['alembic.ini', 'alembic/*', 'alembic/**/*'],
    },
    [...]
)  

A this point, the alembic config and revisions are correctly packaged but the alembic.ini settings have to be tweaked to reflect the new directory tree. It can be done using the %(here)s param which contains the absolute path of the directory containing the alembic.ini file:

# A generic, single database configuration.

[alembic]
# path to migration scripts
script_location = %(here)s/alembic

[...]

# version location specification; this defaults
# to alembic/versions.  When using multiple version
# directories, initial revisions must be specified with --version-path
# version_locations = %(here)s/bar %(here)s/bat alembic/versions
version_locations = %(here)s/alembic/versions

[...]

Finally, you have to call alembic with the -c option which allows to provide the path of the config file:

alembic -c <path to alembic.ini> ...

Answer 2

One way to do this which keeps the main alembic folder along the main package folder is to treat the alembic folder as it's own package to be installed along side your main package.

To do this you must rename it (it can't be called alembic, as it will be a top level package, so needs a unique name - I've used migrations), and add a __init__.py file in the alembic folder and the versions folder.

Running the migrations on deployment requires knowing the path to the installed package - a simple way to do this is to provide a console scripts that applies the migrations.

So the project structure looks like this:

<project root>
├── setup.py
├── mypackage
│   └── <project source files...>
│
├── migrations
│   ├── __init__.py
│   ├── alembic.ini
│   ├── apply.py
│   ├── env.py
│   ├── README
│   ├── script.py.mako
│   └── versions
│       ├── __init__.py
│       ├── 58c8dcd5fbdc_revision_1.py
│       └── ec385b47da23_revision_2.py
│
└── <other files and dirs>

And setup.py:

from setuptools import find_packages
from setuptools import setup


setup(
    name='mypackage',
    packages=find_packages(exclude=('tests',)),
    package_data={'migrations': ['alembic.ini']},
    entry_points={
        'console_scripts': ['apply-migrations=migrations.apply:main'],
    },
    install_requires=[
        "SQLAlchemy==1.3.0",
        "alembic==1.0.10",
        # ...
    ]
)

And finally migrations/apply.py:

# Python script that will apply the migrations up to head
import alembic.config
import os

here = os.path.dirname(os.path.abspath(__file__))

alembic_args = [
    '-c', os.path.join(here, 'alembic.ini'),
    'upgrade', 'head'
]


def main():
    alembic.config.main(argv=alembic_args)

Now after installing your wheel, you will have a command apply-migrations which you can invoke directly. Note the version I've implemented here doesn't have any arguments - though if you wanted to pass eg. --sqlalchemy.url you could add it in alembic_args.

Personally I prefer to set the url in migrations/env.py. For example if you had an environment variable called SQLACLHEMYURL you could add this in migrations/env.py:

import os
config.set_main_options(os.getenv('SQLALCHEMYURL'))

Then you can invoke:

SQLALCHEMYURL=... apply-migrations

On deploment.

Answer 3

To expose all of alembic's arguments, pass sys.argv to alembic. This extends of @Alice Heaton's answer.

Project layout:

my-app
├── my_app
│   └── migrations
│       ├── alembic.ini
│       ├── apply.py
│       ├── env.py
│       ├── script.py.mako
│       └── versions
└── setup.py

setup.py:

from setuptools import setup

setup(
    name="my-app",
    packages=["my_app"],
    entry_points={
        "console_scripts": [
            # This overwrites the existing alembic command
            "alembic = my_app.migrations.apply:main",
        ],
    },
)

migrations/apply.py:

import sys
from pathlib import Path

import alembic.config

here = Path(__file__).parent


def main():
    argv = [
        # Use our custom config path
        "--config", str(here / "alembic.ini"),
        # Forward all other arguments to alembic as is
        *sys.argv[1:]
    ]
    alembic.config.main(argv=argv)

alembic.ini:

[alembic]
script_location = %(here)s