snowflake Python Connection KeyError: 'snowflake-connector-python

Question

I have installed snowflake sqlachemy package. and successfully developed the script to upload a csv file to snowflake.

But when I upload the script to aws glue and supply the wheel file, I encountered KeyError: 'snowflake-connector-python.

Any idea?

here is the full error message:

Traceback (most recent call last):
  File "/tmp/runscript.py", line 123, in <module>
    runpy.run_path(temp_file_path, run_name='__main__')
  File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 250, in <module>
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 39, in main
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 233, in load_data
  File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 479, in create_engine
    return strategy.create(*args, **kwargs)
  File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 222, in load
    return self.impls[name]()
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 245, in load
    mod = compat.import_(modulepath)
  File "/glue/lib/installation/snowflake/sqlalchemy/__init__.py", line 8, in <module>
    from . import snowdialect
  File "/glue/lib/installation/snowflake/sqlalchemy/snowdialect.py", line 27, in <module>
    from snowflake.connector.constants import UTF8
  File "/glue/lib/installation/snowflake/connector/__init__.py", line 17, in <module>
    from .connection import SnowflakeConnection
  File "/glue/lib/installation/snowflake/connector/connection.py", line 43, in <module>
    from .cursor import LOG_MAX_QUERY_LENGTH, SnowflakeCursor
  File "/glue/lib/installation/snowflake/connector/cursor.py", line 47, in <module>
    from .arrow_result import ArrowResult
  File "src/snowflake/connector/arrow_result.pyx", line 16, in init snowflake.connector.arrow_result
  File "/glue/lib/installation/snowflake/connector/options.py", line 36, in <module>
    _pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']
KeyError: 'snowflake-connector-python'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/runscript.py", line 142, in <module>
    raise e_type(e_value).with_traceback(new_stack)
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 250, in <module>
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 39, in main
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 233, in load_data
  File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 479, in create_engine
    return strategy.create(*args, **kwargs)
  File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 222, in load
    return self.impls[name]()
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 245, in load
    mod = compat.import_(modulepath)
  File "/glue/lib/installation/snowflake/sqlalchemy/__init__.py", line 8, in <module>
    from . import snowdialect
  File "/glue/lib/installation/snowflake/sqlalchemy/snowdialect.py", line 27, in <module>
    from snowflake.connector.constants import UTF8
  File "/glue/lib/installation/snowflake/connector/__init__.py", line 17, in <module>
    from .connection import SnowflakeConnection
  File "/glue/lib/installation/snowflake/connector/connection.py", line 43, in <module>
    from .cursor import LOG_MAX_QUERY_LENGTH, SnowflakeCursor
  File "/glue/lib/installation/snowflake/connector/cursor.py", line 47, in <module>
    from .arrow_result import ArrowResult
  File "src/snowflake/connector/arrow_result.pyx", line 16, in init snowflake.connector.arrow_result
  File "/glue/lib/installation/snowflake/connector/options.py", line 36, in <module>
    _pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']
KeyError: KeyError('snowflake-connector-python',)

And this is my requirement.txt

snowflake-connector-python
snowflake-sqlalchemy
sqlalchemy
pyarrow
pandas
asn1crypto==1.4.0
awscli==1.18.140
awswrangler==1.9.3
azure-common==1.1.25
azure-core==1.8.1
azure-storage-blob==12.5.0
boto3==1.14.63
botocore==1.17.63
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
colorama==0.4.3 ; python_version != '3.4'
cryptography==2.9.2
docutils==0.15.2
fsspec==0.8.2
idna==2.9
isodate==0.6.0
jmespath==0.10.0
msrest==0.6.19
numpy==1.19.2
oauthlib==3.1.0
oscrypto==1.2.1
packaging==20.4 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
psycopg2-binary==2.8.6
pyasn1==0.4.8
pycparser==2.20
pycryptodomex==3.9.8
pyjwt==1.7.1
pymysql==0.9.0
pyopenssl==19.1.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
pyyaml==5.3.1 ; python_version != '3.4'
requests-oauthlib==1.3.0
requests==2.23.0
rsa==4.5 ; python_version != '3.4'
s3fs==0.4.2
s3transfer==0.3.3
six==1.15.0
sqlalchemy-redshift==0.8.1
urllib3==1.25.10
xlrd==1.2.0

from the log file, here are the packages installed

Successfully installed asn1crypto-1.4.0 awscli-1.18.140 awswrangler-1.9.3 azure-common-1.1.25 azure-core-1.8.1 azure-storage-blob-12.5.0 boto3-1.14.63 botocore-1.17.63 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 cryptography-2.9.2 dependencies-0.1.0 docutils-0.15.2 fsspec-0.8.2 idna-2.9 isodate-0.6.0 jmespath-0.10.0 msrest-0.6.19 numpy-1.19.2 oauthlib-3.1.0 oscrypto-1.2.1 packaging-20.4 pandas-1.1.2 psycopg2-binary-2.8.6 pyarrow-1.0.1 pyasn1-0.4.8 pycparser-2.20 pycryptodomex-3.9.8 pyjwt-1.7.1 pymysql-0.9.0 pyopenssl-19.1.0 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.23.0 requests-oauthlib-1.3.0 rsa-4.5 s3fs-0.4.2 s3transfer-0.3.3 six-1.15.0 snowflake-connector-python-2.3.2 snowflake-sqlalchemy-1.2.3 sqlalchemy-1.3.19 sqlalchemy-redshift-0.8.1 urllib3-1.25.10 xlrd-1.2.0

score -1 · Answer 1 · edited Jun 24 '22 at 01:34

-1

I recommend you do a "clean" build of all the main libraries, and in this include the pyarrow library.

e.g.

pip install --upgrade --force-reinstall pandas
pip install --upgrade --force-reinstall pyarrow
pip install --upgrade --force-reinstall snowflake-connector-python
pip install --upgrade --force-reinstall sqlalchemy
pip install --upgrade --force-reinstall snowflake-sqlalchemy

There have been some issues here and there with old versions or missing versions of pyarrow, even the order of install could be a problem I believe. A fresh pip install or ensuring you have the latest/greatest of these libraries is a good way to go.

edited Jun 24 '22 at 01:34

Nimantha

6,405
6
28
69

answered Sep 18 '20 at 11:29

Rich Murnane

2,697
1
11
23

I have included the requirements.txt in the post. I used a wheel file to include the packages, then upload to s3 and use it in the glue job. But I still get the same error. – JOHN Sep 18 '20 at 15:05
Similar error and solution: https://stackoverflow.com/q/76259451/132438 – Felipe Hoffa May 16 '23 at 04:40

snowflake Python Connection KeyError: 'snowflake-connector-python

1 Answers1

Linked