Multiple versions of Sqlite3 for Python on the same server

Question

On a Linux server, I have some Python scripts using the built-in sqlite3 module (+ some Sqlite extensions built from source, as detailed in Upgrade Python's sqlite3 on Debian).

For another Python script, I need a newer version of the Sqlite shared library than the one I already have on the system. Reason: I need Sqlite higher than 3.25.0 for Window Functions.

If I install it from source here and do make and make install, it will probably overwrite previous versions of this library on the server, and could potentially break other OS tools using it.

How do you handle the general problem of having multiple versions of the Sqlite shared library?

I don't think Python virtual environments can be used for this context, or would it be possible?

Note: pip3 install --upgrade sqlite3 does not exist: we cannot upgrade Python's built-in sqlite3 package like this. And by the way we probably should not, since it could break some OS tools using Python + sqlite3.

*No way* you are got an `failure` on `system_modules` and `customized_modules` , cause if related libraies used same `name_space` ! — dsgdfg, Jul 25 '21 at 14:38
Just to be sure, take a look at the sqlite3 library version that comes with your Python. Just execute: `python -c "import sqlite3; print(sqlite3.connect(':memory:').execute('SELECT sqlite_version();').fetchall())"` — Iñigo González, Jul 29 '21 at 12:01
A shorter version is to use: `python -c "import sqlite3; print(sqlite3.sqlite_version)"` — smartexpert, Apr 01 '22 at 13:34
@smartexpert Are you 100% sure? There is the `sqlite3` "Python module version", and also the underlying "sqlite library version". These are different numbers. — Basj, Apr 02 '22 at 09:57
@Basj Yes. The python module version can be obtained via `sqlite3.version` and the underlying sqlite library version via `sqlite3.sqlite_version` — smartexpert, Apr 06 '22 at 22:55

score 1 · Accepted Answer · answered Jul 30 '21 at 11:04

This is very tricky and will need a little code change in your scripts.

What to do:

First, check the sqlite3 library version included with python just in case:
```
python -c "import sqlite3; print(sqlite3.connect(':memory:').execute('SELECT sqlite_version();').fetchall())
```
In my computer (python 3.8, windows) the output is [('3.35.5',)] which means python has the sqlite 3.35.5 library. I have no sqlite installed in my system: this is the library that comes with python3.8.
IF your python sqlite3 library is not the one you need :-( you have an alternative: you can use the pysqlite3 instead of the sqlite3 standard library. In this case:
1. You'll need to build the pysqlite3 library by yourself using the Sqlite3 'amalgamation' that matches the version you want to use (more on later).
2. You'll need to install the library, and...
3. You will need to change your python script imports import pysqlite3 as sqlite3 # instead of sqlite3

Ok, what is the 'amalgamation` and how to build pysqlite3?

The amalgamation is the whole sqlite3 library in just one .c file (with the sqlite3.h file). You can get it from the sqlite3 download page: sqlite3.36 amalgamation.

Once you have the amalgamation, follow the instructions to build statically pysqlite3, and install the package.

Now you can use pysqlite3 in your code.

anarchy · Answer 2 · 2021-07-25T17:57:21.117

If you want 2 different version of sqlite3 (python3) on 2 different environments, you can do that.

Since you mentioned that sqlite3 is part of the std library, it seems like you can try the pysqlite3 package instead.

If you can't run pip, run the following command first.

sudo apt install python3-pip

Then,

pip install virtualenv

python3 -m venv sqlitev1 #(whatever name you want)
source sqlitev1/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate

python3 -m venv sqlitev2 #(whatever name you want)
source sqlitev2/bin/activate
pip install pysqlite3==0.4.4 #(this can be whatever version u want)
source deactivate

Now you have 2 python environments, sqlitev1 and sqlitev2, with 2 different version of sqlite3.

This doesn't work in the case of `sqlite` (or maybe you're using a specific technique?). As mentioned in my question, `sqlite` is not a package you can update with `pip`. It's a built-in in the Python install, part of the std lib. — Basj, Jul 25 '21 at 17:54

score 0 · Answer 3 · answered Jul 30 '21 at 08:11

It might be super hacky but you can make the new version of sqlite and then make sure that the path pointing to the new version is on the pythonpath environment before the built in one. Python will scan the python path from first to last to find an import, so the new version first in the python path for the processes that want the new version and then exclude that path with the old processes that need the built in one. You can accomplish this with a bash script that loads the env and then runs the python process for the new services.

Again this is super hacky so last resort.

snakecharmerb · Answer 4 · 2021-07-30T11:54:27.987

If you want a different version of Sqlite than that installed with your distro, and a Python that uses that version, then you could

Compile sqlite to an alternative location
Compile Python to a different location, and point it to the custom Sqlite installation.

The "pointing" is covered in the accepted answer to this question. The question body itself shows how you might compile sqlite to a custom location.

The other answer to that question proposes setting the LD_LIBRARY_PATH environment variable to the directory containing the custom sqlite build to avoid having to compile Python. This might work with a virtualenv (it could be set in the preactive hook, for example).

Multiple versions of Sqlite3 for Python on the same server

4 Answers4

Linked