0

Environment:

python==3.7 
pandas==1.0.5 
sqlalchemy==1.4.46 
pg8000==1.29.4

My project is using pyspark 3.3 which is dependent upon pandas 1.0.5, hence the pandas requirement. I have already been able to connect with sqlalchemy==2.0.10 so I know my credentials are good and that is not the problem. I had to roll back to sqlalchemy==1.4.46 because sqlalchemy 2.0 does not like older versions of pandas.

However when I try run with this setup, I get this error:

from sqlalchemy import create_engine

engine = create_engine(f"postgresql+pg8000://{cfg['USERNAME']}:{cfg['PASSWORD']}@{cfg['HOST']}:{cfg['PORT']}/{cfg['DATABASE']}")


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-85-8d830c258e01> in <module>
      2 
      3 #create_engine("postgresql+psycopg2://scott:tiger@localhost:5432/mydatabase")
----> 4 engine = create_engine(f"postgresql+pg8000://{cfg['USERNAME']}:{cfg['PASSWORD']}@{cfg['HOST']}:{cfg['PORT']}/{cfg['DATABASE']}")
      5 
      6 with engine.connect() as conn:

~/miniconda3/envs/py37/lib/python3.7/site-packages/sqlalchemy/engine/create.py in create_engine(url, **kwargs)

~/miniconda3/envs/py37/lib/python3.7/site-packages/sqlalchemy/util/deprecations.py in warned(fn, *args, **kwargs)
    281                 "add warn_on_attribute_access=True.  Otherwise please add "
    282                 "enable_warnings=False." % api_name
--> 283             )
    284 
    285     if alternative:

~/miniconda3/envs/py37/lib/python3.7/site-packages/sqlalchemy/engine/create.py in create_engine(url, **kwargs)
    647         )
    648 
--> 649     engine = engineclass(pool, dialect, u, **engine_args)
    650 
    651     if _initialize:

NameError: name '_pool_translate_kwargs' is not defined

And I can't find much on this error.

I want to add that I'm trying different DB APIs. I started with pg8000 because psycopg2 gave me this error on pip install but pg8000 did not. After getting pyscopg2 installed, I get the same error as above so it seems independent of DB api but I'm not exactly sure where to go from here. It seems like the only option would be to upgrade pandas until sqlalchemy 2.0 works but I'd rather not go that route since it could end up breaking pyspark.

So I'm wondering if anyone has any thoughts on this.

Ken Myers
  • 596
  • 4
  • 21
  • I think I've figured it out. I was testing this in a notebook and I was using autoreload (which usually works) but I think with all the package reinstallations I was doing things got screwed up leading to the unusual error above. After installing the packages how I wanted and restarting the kernel I was able to get things working with pandas==1.0.5 and sqlalchemy==1.4.46. I am able to see the data in my table and things seem to be working – Ken Myers Apr 24 '23 at 05:13

0 Answers0