1

I was using AWS glue python shell. The program uses multiple python libraries which not natively available for AWS. Glue can take .egg or .whl files for external library reference. All we need to do is put these .egg or .whl file in some S3 location and point to it using it's full path. I tried with one external library [for instance openpyxl] and it worked. Now the problem is since I have multiple external libraries like pandas, numpy, openpyxl and pytz to be referred, I cant give full path of all these packages as only path can be specified as external python library reference. I tried giving the s3 folder name where I placed all these packages, it does not work.

How can I specify these multile .egg or .whl files so that my glue job can use them.

Ludwig
  • 782
  • 1
  • 8
  • 24

2 Answers2

2
  • Save your library in (.whl) format in s3 location.
  • You can add libraries in "Python Library Path" in comma separated format. example- s3://libs/xyz.whl, s3://libs/abc.whl

Reference image

betelgeuse
  • 1,136
  • 3
  • 13
  • 25
GaneshMuni
  • 41
  • 6
  • This didn't work for me, I also uploaded my library.whl to S3 and then I added libraries in "Python Library Path". But it doesn't work with current version. Do you have any new config? – Tien Vu Dec 12 '22 at 16:46
  • Try Python 3.7 or below compatible libraries. Try to get the .whl file from pypi.org -> Release history. – GaneshMuni Dec 27 '22 at 17:23
0

This question is already answered by gbeaven, but for some reasons I am unable mark it as answer. This was fixed by comma separating the file paths in the additional python modules.

Ludwig
  • 782
  • 1
  • 8
  • 24