0
SystemExit: error: google-auth 2.0.0.dev0 is installed but google-auth<2.0dev,>=1.24.0 is required by {'google-cloud-core'}

This error appears now on AWS Glue Jobs, in my case, jobs using smart_open. Did not happen until now and I was already using .eggs on AWS Glue to be able to run smart_open.

There is a new version of google-auth 1.33.0 since 20th of July 2021, those anyone else encountered this error? AWS has to update this error?

Maybe using a .egg file of google-auth 1.33.0 will solve that issue. Refer to these post on how to do an .egg file How to create Python egg file

Aleix Molla
  • 75
  • 6

1 Answers1

1

I was already using an egg for my pythonshell jobs. However I was not freezing the version numbers.

setup.py

from setuptools import setup

setup(
  name="my_glue_dependencies",
  install_requires=[
    'google-auth==1.33.1',
    'google-cloud-storage==1.41.1',
    'google-cloud-bigquery==2.22.0'
  ]
)

The key to freezing the versions used is to use == instead of >= for the install_requires options.

The docker image from this article can be used to build the egg.

Build the egg $ python setup.py bdist_egg -d ./dist

The full article on python glue jobs, and how to Provide Your Own Python Library.

ObeseCoder
  • 391
  • 2
  • 6