95

What fork, or combination of packages should one to use to make PyPy, Django and PostgreSQL play nice together?

I know that PyPy and Django play nice together, but I am less certain about PyPy and PostgreSQL. I do see that Alex Gaynor has made a fork of PyPy called pypy-postgresql. I also know that some people are using psycopg2-ctypes.

Is there a difference between these forks? Or should we use the stable 1.9 PyPy and use psycopg2-ctypes? Using the ctypes options could hurt performance, see the comment below.

Also, has anyone experienced any pitfalls with using PyPy with pyscopg2? It seems easy enough to fall back on CPython if something isn't working right, but mostly I'm looking for things a programmer can do ahead of time to prepare.

I looked around, it doesn't seem that psycopg2 works natively with PyPy. Although, psycopg2-ctypes does seem to be working for some people, there was a discussion on pypy-dev. I work on Windows, and I don't think psycopg2-ctypes is ready for Windows yet, sadly.

Ed.
  • 1,404
  • 4
  • 17
  • 26
James R
  • 4,571
  • 3
  • 30
  • 45
  • 8
    There's a nice talk by Alex from the Djangocon Europe 2011, where he also talks about the postgres issues: http://blip.tv/djangocon-europe-2011/monday-1400-alex-gaynor-good-5343953 – Bernhard Vallant Feb 19 '12 at 22:25
  • 3
    BTW, calling c code from PyPy is still much slower than using pure python modules. So you probably won't get any benefits from running Django + psycopg2 via PyPy. You can rewrite psycopg2 in [RPython](http://doc.pypy.org/en/latest/coding-guide.html#id1) or use CPython and optimize critical parts of your application with [Cython](http://cython.org/). – Vladimir Protasov May 19 '12 at 07:03
  • Also: https://bitbucket.org/alex_gaynor/pypy-postgresql (kind of old...) – jperelli Jun 07 '12 at 20:28
  • 2
    Some people around me recommending [psycopg2ct](http://pypi.python.org/pypi/psycopg2ct), but without django... May can help... – Dingo Jun 12 '12 at 20:45
  • what is your os, distribution, version ? – pylover Jun 17 '12 at 14:23
  • The DjangoconEU video was removed Nov7, 2013 by blip.tv. The talk can now be found a pyvideo.org http://pyvideo.org/video/2486/0_django-and-pypy-performant-is-a-wordm4v – umeboshi Dec 01 '13 at 21:46

2 Answers2

38

psycopg2cffi (Updated 2015)

psycopg2cffi is yet another psycopg2-compatible replacement and should provide the best PostgreSQL performance with PyPy. Add this to your settings.py to remain compatible with both:

try:
    import psycopg2
except ImportError:
    # Fall back to psycopg2cffi
    from psycopg2cffi import compat
    compat.register()

psycopg2-ctypes (2012)

I also know that some people are using psycopg2-ctypes.

This is the easiest way; to stay compatible with both, just add this code in your Django settings.py:

try:
    import psycopg2
except ImportError:
    # Fall back to psycopg2-ctypes
    from psycopg2ct import compat
    compat.register()

I tested this a few releases ago; sadly in my experience, psycopg2-ctypes negates the small performance gains afforded by PyPy. But YMMV, it depends on how JIT-friendly your code is in general and what fraction of time you actually spend running Python code. And maybe PyPy has just improved since then.

and I don't think psycopg2-ctypes is ready for Windows yet

I haven't tried this, but ctypes is platform-independent. AFAICT you just have to make sure that the libpq.dll library is loadable (located in a directory in your PATH environment variable or local directory) and it should work on Windows just like in Linux.

pypy-postgresql

I do see that Alex Gaynor has made a fork of PyPy called pypy-postgresql.

I don't think this is a good choice in the long term. The branch hasn't been updated for more than a year and my attempts to build it have failed. And it seems wrong to hard-code a PostgreSQL driver in the interpreter anyway.

I believe there are no binaries out there of pypy-postgresql either, so if you want to use it, you'd need to build the whole PyPy branch yourself. Not for the faint of heart: it takes tens of minutes and a machine with at least 4 GB of memory. (Official instructions: http://pypy.org/download.html#building-from-source)

To build, you first need the source. If you have Mercurial installed, you can simply hg clone https://bitbucket.org/alex_gaynor/pypy-postgresql. If not, you can download the automagic "tip" zip file: https://bitbucket.org/alex_gaynor/pypy-postgresql/get/tip.zip

Open a command line, go into the decompressed directory, and then inside pypy/translator/goal

If you have PyPy installed, it's recommended to use that for building:

pypy translate.py -Ojit

Otherwise:

python translate.py -Ojit

Sadly this is where my knowledge ends. I get the error "BytecodeCorruption: unimplemented opcode, ofs=234, code=203, name=BUILD_LIST_FROM_ARG"

intgr
  • 19,834
  • 5
  • 59
  • 69
  • Could you add a little, provided, you know, about what it might take to get pypy-pyscopg2 working with pypy without needing to compile pypy from source? – James R Jul 01 '12 at 12:33
  • 1
    I updated the answer with details on how far I got. Unfortunately the build aborted with an error. I remember spending a whole afternoon trying to get it to build the last time, and failed, too. – intgr Jul 02 '12 at 10:40
  • psycopg2cffi won't pip install on Windows! ~cry~ – user3761555 May 29 '22 at 03:40
  • But this seems work: https://pypi.org/project/py-postgresql/ – user3761555 May 29 '22 at 03:49
16

Some additional resources:

  • PyPy compatibility information: DB adaptors
  • PostgreSQL page on the Python wiki
  • psycopg2cffi by Konstantin Lopuhin:
    cffi based implementation of psycopg2 for PyPy 2.0 and newer
    (blog post, GitHub repo, PyPI page, pypy-dev thread)
    – this looks like the strongest candidate currently, but I haven't tested it yet
  • psycopg2ct by Michael van Tellingen:
    ctypes based implementation of psycopg2 for PyPy 1.6 and newer
    (GitHub repo, PyPI page)
  • pypy-postgresql by Alex Gaynor:
    abandoned RPython port of psycopg2 implemented as a fork of PyPy (Bitbucket repo)
  • pypq:
    "Python PostgreSQL DBAPI 2.0 compliant driver using ctypes and libpq.so, works with PyPy"
    (discussion, PyPI page)
  • bpgsql:
    "Barebones pure-python PostGreSQL client. Mostly DB-API 2.0 (PEP 249) compliant. Includes an experimental Django 1.0 backend"
    (discussion, web page, Google Code page)
  • pg8000:
    "a DB-API 2.0 compatible Pure-Python interface to the PostgreSQL database engine [...] does not rely on any external libraries (such as a compiled python module, or PostgreSQL’s libpq library)"
    (web page, GitHub repo, PyPI page)
akaihola
  • 26,309
  • 7
  • 59
  • 69