1

I have AWS-glue job that runs a Python shell script. The script uses python package mysql_connector_python-8.0.19-cp37-cp37m-manylinux1_x86_64.whl that is located in a S3 bucket.

The job throws an error saying "ERROR: No matching distribution found for protobuf==3.6.1 (from mysql-connector-python==8.0.19)". It looks like the server that runs the script is unable to talk to pypi.org. More details of the error can be found below. What is wrong?

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/protobuf/

ERROR: Could not find a version that satisfies the requirement protobuf==3.6.1 (from mysql-connector-python==8.0.19) (from versions: none)

Traceback (most recent call last):
  File "/tmp/runscript.py", line 112, in <module>
    download_and_install(args.extra_py_files)
  File "/tmp/runscript.py", line 62, in download_and_install
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--target=
{}
".format(install_path), local_file_path])
  File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-lui8e6qb/mysql_connector_python-8.0.19-cp37-cp37m-manylinux1_x86_64.whl']' returned non-zero exit status 1.
Dustin Sun
  • 5,292
  • 9
  • 49
  • 87
  • Is it mandatory to use the mysql connector as a whl file? I would suggest download pymysql to your local environment, zip the pymysql alongwith your glue script. You can upload this to the S3, and use in your AWS Glue job. This would work. – Yuva Feb 06 '20 at 16:29
  • It is not required to use mysql connector. But it is more like a VPC/subnet settings issue to me. And I am not sure if mysql-connector matters or not. – Dustin Sun Feb 06 '20 at 17:04
  • Have you tried installing this inside your python shell job as mentioned in https://stackoverflow.com/a/54852126/4326922 ? – Prabhakar Reddy Feb 07 '20 at 04:42

0 Answers0