9

Some background: we have a codebase written in Python 3 that uses Pyramid and the SqlAlchemy ORM to persist to a mysql database. To write tests for our classes using the ORM we are using Sqlite. All of this works fine together... locally.

Setting up our Jenkins (Ubuntu) server to run the test suite, inside a virtualenv, we run into a problem. The tests are executed like so:

coverage run --source src/ --omit=src/tests/ -m py.test

Tests not involving the ORM are fine. Those with the ORM throw this error:

____________________________________________________________ TestSGenre.test_get_all_success _____________________________________________________________

self = <tests.common.orm.models.test_s_genre.TestSGenre testMethod=test_get_all_by_discipline_success>

    def setUp(self):
        DBSession.remove()
>       self.engine = setup()

source/src/tests/common/orm/models/test_s_genre.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
source/src/tests/common/orm/orm_setup.py:7: in setup
    engine = create_engine('sqlite://', connect_args={'check_same_thread': False}, poolclass=StaticPool)
/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/engine/__init__.py:386: in create_engine
    return strategy.create(*args, **kwargs)
/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py:74: in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py:339: in dbapi
    raise e
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'sqlalchemy.dialects.sqlite.pysqlite.SQLiteDialect_pysqlite'>

    @classmethod
    def dbapi(cls):
        try:
>           from pysqlite2 import dbapi2 as sqlite
E           ImportError: No module named 'pysqlite2'

/var/www/hosts/company/virtualenv/swapenv/lib/python3.4/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py:334: ImportError

Checking for sqlite3 (from inside the virtualenv) is successful:

(swapenv)user@jenkins:/var/lib/jenkins/workspace/SWAP_Unit_Test$ which sqlite3
/usr/bin/sqlite3

It is also successful outside the virtualenv. We've tried installing and reinstalling all number of sqlite packages, sqlite-dev, etc. Supposedly the Sqlite library is part of Python 3, but then why can't it be found when the tests are being run?

Nathaniel Ford
  • 20,545
  • 20
  • 91
  • 102
  • 2
    The `which sqlite3` command only shows you have the `sqlite3` command line tool installed; this is not what Python uses. It uses the `libsqlite3` shared library (which the command line tool also uses). How was your Python binary compiled? You are missing the `sqlite3` Python module (verify with `bin/python -c 'import sqlite3'`. – Martijn Pieters Apr 19 '15 at 01:30
  • Note that the `py.test` traceback is misleading; it is line 339 that raises the final error, not the displayed line 334 (which is caught); see the [driver source code](https://bitbucket.org/zzzeek/sqlalchemy/src/bd61e7a3287079cf742f4df698bfe3628c090522/lib/sqlalchemy/dialects/sqlite/pysqlite.py?at=master#cl-331); it is the import for `sqlite3` that fails here. – Martijn Pieters Apr 19 '15 at 01:36
  • Paste the code of jenkins script? Including how you setup virtualenv? 1. Check whether all dependencies installed in virtualenv. 2. Make sure your coverage runs inside the correct virtualenv. – Shuo Apr 24 '15 at 16:44
  • `bin/python -c 'import sqlite3'` definitely fails to load the import. I'm not sure how python was compiled, unfortunately, but that does seem the root of the problem... – Nathaniel Ford Apr 24 '15 at 21:35
  • This question: http://stackoverflow.com/questions/8628774/python-3-2-cant-import-sqlite3-module had enough information to figure out how to recompile python with the correct libraries. Thanks @MartijnPieters for the pointer! If you want to put an answer in I'm happy to award the bounty. – Nathaniel Ford Apr 24 '15 at 23:27
  • @NathanielFord: ah, if that question contains what you needed then this is really a *duplicate question*. :-) – Martijn Pieters Apr 25 '15 at 08:20
  • @NathanielFord: but I guess the SQLAlchemy angle may be the confusing party here, plus your confusion about the command line tool. Posted an answer for you with more header packages for you to install. – Martijn Pieters Apr 25 '15 at 08:29
  • Yeah, it's a thin line between being a duplicate and being found in a different manner. I'm torn as to whether to leave it up, because duplication is bad but sometimes having a different approach vector to the same question is good. – Nathaniel Ford Apr 25 '15 at 16:56

2 Answers2

15

You are missing the sqlite3 Python module, which you can verify with:

bin/python -c 'import sqlite3'

The which sqlite3 command only shows you have the sqlite3 command line tool installed; this is not what Python uses. It uses the libsqlite3 shared library (which the command line tool also uses). If missing, it means Python was not able to find the SQLite development headers when you built Python.

On Ubuntu, you need to install libsqlite3-dev to get those headers.

You may be missing other dependencies; on Ubuntu I'd install:

libreadline6-dev
libbz2-dev
libssl-dev
libsqlite3-dev
libncursesw5-dev
libffi-dev
libdb-dev
libexpat1-dev
zlib1g-dev
liblzma-dev
libgdbm-dev
libmpdec-dev

Some of these are accelerator packages; Python will work without them but some modules will be slower (e.g. decimal without the mpdecimal library).

You may want to verify the Ubuntu Python 3.4 source package dependencies for your Ubuntu version.

Ye Lin Aung
  • 11,234
  • 8
  • 45
  • 51
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
2

Running Debian Buster, I discovered the same problem with python3.6, even though, python3.5 successfully imported sqlite3. And even though the sqlite3 module was installed and should have been available to python3.6. My solution was to run

export PYTHONPATH=$PYTHONPATH:/usr/lib/python3.6/lib-dynload

For some reason I have not yet determined, the module directory for python3.6 does not get loaded correctly for sqlite3. This solution worked both inside and outside of virtual environments.

Deleuzer
  • 21
  • 2