1

Trying to debug AWS Glue scripts locally using Glue ETL library. I have installed aws-glue-libs and spark-3.1.1-amzn-0-bin-3.2.1-amzn-3.tgz.

When I run job.init(), I get the following error trace:

py4j.protocol.Py4JJavaError: An error occurred while calling z:com.amazonaws.services.glue.util.Job.init. : java.lang.NoClassDefFoundError: com/typesafe/config/ConfigMergeable at com.amazonaws.services.glue.util.Job$.init(Job.scala:93) at com.amazonaws.services.glue.util.Job.init(Job.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.typesafe.config.ConfigMergeable at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 13 more

sheetal_158
  • 7,391
  • 6
  • 27
  • 44
  • Please edit the question to include the error stacktrace as text and not just as an image, see [Why should I not upload images of code/data/errors when asking a question?](https://meta.stackoverflow.com/a/285557). – bad_coder Apr 19 '22 at 06:41
  • There's only 1 more post on Stack Overflow having the [NoClassDefFoundError](https://stackoverflow.com/a/67506885) error on the aws-glue tag. Did you see the answer? – bad_coder Apr 19 '22 at 06:46
  • @bad_coder yes i tried that answer. The problem persists. – sheetal_158 Apr 19 '22 at 17:57
  • Not my downvote!! I don't now how to solve this problem (I don't work with AWS Glue) but check the link in the first comment, questions that don't have the full stacktrace included as text are closed per site-policy. Even if someone else knows the answer they're unlikely to attempt answering a question that doesn't include the stacktrace as text. – bad_coder Apr 20 '22 at 00:06

2 Answers2

1

This error wasn't resolved per say. But i found a workaround. Instead of running my scripts from pycharm, i run them using gluesparksubmit bash command. Now it doesn't throw error at job.init(). Still trying to figure out how to get access to data catalog when running glue scripts from local machine.

sheetal_158
  • 7,391
  • 6
  • 27
  • 44
0

If it was for ConfigMergeable, check if proper jar file - config-1.3.3.jar - exists in your /opt/spark/jars dir.

Whole idea is that jars in /opt/spark/jars and ./aws-glue-libs/jarsv1 should match.

marcin2x4
  • 1,321
  • 2
  • 18
  • 44