How do I get libpq to be found by ctypes find_library?

Question

I am building a simple DB interface in Python (3.9.9) and I am using psycopg (3.0.7) to connect to my Postgres (14.1) database. Until recently, the development of this app took place on Linux, but now I am using macOS Monterey on an M1 Mac mini. This seems to be causing some troubles with ctypes, which psycopg uses extensively. The error I am getting is the following:

ImportError: no pq wrapper available.
Attempts made:
- couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
- couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary'
- couldn't import psycopg 'python' implementation: libpq library not found

Based on the source code of psycopg, this is an error of ctypes not being able to util.find_library libpq.dylib. Postgres is installed as Postgres.app, meaning that libpq.dylib's path is

/Applications/Postgres.app/Contents/Versions/14/bin/lib

I have tried adding this to PATH, but it did not work. I then created a symlink to the path in /usr/local/lib, but (unsurprisingly) it also did not work. I then did some digging and found this issue describing the same problem. I am not a big macOS expert, so I am unsure on how to interpret some of the points raised. Do I need to add the path to the shared cache? Also, I do not want to fork the Python repo and implement the dlopen() method as suggested, as it seems to lead to other problems.

Anyhow, is there a solution to quickly bypass this problem? As an additional reference, the code producing the above error is just:

import psycopg

print(psycopg.__version__)

You can try installing `pip install "psycopg[binary]"` for bin packages. Otherwise potentially this lib just doesn't support m1. Possibly related [issue](https://github.com/psycopg/psycopg/issues/162). — Hitobat, Jan 04 '22 at 21:16
Thank you for your answer! I just checked and according to psycopg3's documentation, ```pip install "psycopg[binary]"``` is not supported by M1 Macs (https://www.psycopg.org/psycopg3/docs/basic/install.html). — eslukas, Jan 04 '22 at 21:20

score 2 · Accepted Answer · answered Jan 17 '22 at 11:23

I had this problem but the solution was suggested to me by this answer to a related question: try setting envar DYLD_LIBRARY_PATH to the path you identified.

NB, to get it working myself, I:

used the path /Applications/Postgres.app/Contents/Versions/latest/lib and
had to install Python 3.9

score 1 · Answer 2 · answered Jan 05 '22 at 17:39

1

The problem has solved itself by using psycopg2 instead of psycopg3, so it seems to be a problem of the latter. It would be still good to know why so as to solve it.

answered Jan 05 '22 at 17:39

eslukas

245
2
8

2

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jan 05 '22 at 23:22

score 1 · Answer 3 · answered Jul 16 '22 at 16:20

A new way

This issue came up on my Windows laptop using Jupyter, although with WSL and Ubuntu it was fine...it was even fine using IntelliJ that was using python from Windows. I have a Mac Mini so I thought I'd test what I used instead pg8000

Now, pg8000 is a pure python implementation to connect to Postgresql. After a brew install of Postgres (and a few shenanigans with PSQL to get it working), this worked fine pip3 install pg8000

Basic Recipie

#!/usr/bin/env python3
import pg8000 as pg

def db_conn():
    conn = pg.Connection(host='localhost', 
                         database='dev_database', 
                         user='postgres', 
                         password='yourPassword!')
    return conn

# Now you have a connection, query your data with a cursor

conn = db_conn()
cur = conn.cursor()
cur.execute('SELECT * FROM tab1')
data = cur.fetchall()

# Now 'data' contains your data, or use the new way with conn.run

if __name__ == '__main__':
    print('Grabbing data from tab1...')

    for row in conn.run('SELECT * FROM tab1'):
        print(row)

    conn.close()

It all seems pretty much the same as using psycopg, but the new method connection.run is pretty good too!

score 1 · Answer 4 · answered Aug 09 '22 at 14:02

I got it to work by running:

pip uninstall psycopg
pip install "psycopg[c]"

This provides a local installation of psycopg3 which is needed since there are no binaries available for M1 macs. I installed postgresql (and libpg) via homebrew, running pg_config gives me a proper output.

Using the pure python package and setting the DYLD_LIBRARY_PATH didn't work for me.

How do I get libpq to be found by ctypes find_library?

4 Answers4