Problem installing hadoop-gremlin with janusgraph

Question

We are running into some issues while launching gremlin.sh with hadoop (ref: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin) using janusgraph.

Any tips/help appreciated! Thanks!

Error with Janus 0.6.3 (https://github.com/JanusGraph/janusgraph/releases/download/v0.6.3/janusgraph-0.6.3.zip): 20:00:36 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs

sshuser@hn0-spark:~/janusgraph/janusgraph-0.6.3$ echo $HADOOP_CONF_DIR
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-0.6.3$ echo $CLASSPATH
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-0.6.3$ bin/gremlin.sh

         \,,,/
         (o o)
-----oOOo-(3)-oOOo-----
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-0.6.3/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-0.6.3/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
22:04:12 WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)
        at org.apache.tinkerpop.gremlin.jsr223.LazyBindingsCustomizer.getBindings(LazyBindingsCustomizer.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
        at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
        at org.apache.tinkerpop.gremlin.console.PluggedIn$_activate_closure1.doCall(PluggedIn.groovy:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovy.lang.Closure.call(Closure.java:421)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2137)
        at org.codehaus.groovy.runtime.dgm$181.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.tinkerpop.gremlin.console.PluggedIn.activate(PluggedIn.groovy:59)
        at org.apache.tinkerpop.gremlin.console.PluggedIn$activate.call(Unknown Source)
        at org.apache.tinkerpop.gremlin.console.Console$_closure18.doCall(Console.groovy:149)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovy.lang.Closure.call(Closure.java:421)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2331)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2316)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2357)
        at org.codehaus.groovy.runtime.dgm$186.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:147)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
        at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:524)
Caused by: java.io.IOException: No FileSystem for scheme: wasbs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:121)
        ... 59 more

Error with Janus 1.0.0-rc2 (https://github.com/JanusGraph/janusgraph/releases/download/v1.0.0-rc2/janusgraph-1.0.0-rc2.zip): 21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader. - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found

sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ echo $HADOOP_CONF_DIR
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ echo $CLASSPATH
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ bin/gremlin.sh

         \,,,/
         (o o)
-----oOOo-(3)-oOOo-----
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib/log4j-slf4j-impl-2.20.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
22:05:05 INFO  org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib
22:05:06 WARN  org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)
        at org.apache.tinkerpop.gremlin.jsr223.LazyBindingsCustomizer.getBindings(LazyBindingsCustomizer.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
        at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
        at org.apache.tinkerpop.gremlin.console.PluggedIn$_activate_closure1.doCall(PluggedIn.groovy:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovy.lang.Closure.call(Closure.java:421)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2137)
        at org.codehaus.groovy.runtime.dgm$181.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.tinkerpop.gremlin.console.PluggedIn.activate(PluggedIn.groovy:59)
        at org.apache.tinkerpop.gremlin.console.PluggedIn$activate.call(Unknown Source)
        at org.apache.tinkerpop.gremlin.console.Console$_closure18.doCall(Console.groovy:156)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovy.lang.Closure.call(Closure.java:421)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2331)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2316)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2357)
        at org.codehaus.groovy.runtime.dgm$186.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:154)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
        at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:576)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2667)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:121)
        ... 59 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2571)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
        ... 67 more

The hadoop version we are using is 3.1.1

sshuser@hn0-spark:~/janusgraph$ hadoop version
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.14.2/hadoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.14.2/hadoop-hdfs/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Hadoop 3.1.1.4.1.14.2
Source code repository https://msdata.visualstudio.com/HDInsight/_git/bigtop -r 9482450b631bf06716ebd304e90e1ea0c806dd4d
Compiled by hdiuser on 2023-04-20T12:54Z
Compiled with protoc 2.5.0
From source with checksum 4224efc0ed85daa1281ef3d22ebb0c
This command was run using /usr/hdp/4.1.14.2/hadoop/hadoop-common-3.1.1.4.1.14.2.jar

The spark cluster is setup using Azure HDInsights.

https://stackoverflow.com/questions/38254771/spark-shell-error-no-filesystem-for-scheme-wasb might be related. — Boxuan Li, May 31 '23 at 23:32

score 3 · Answer 1 · answered Jun 01 '23 at 05:44

3

JanusGraph-0.6.3 ships with the hadoop-2.8 client, so you cannot expect it to work with a hadoop-3 cluster. So, indeed, you should depend on JanusGraph-1.0.0.rc2 which uses the hadoop-3.3 client. Regarding the java.lang.ClassNotFoundException exception: the JanusGraph distribution does not include the hadoop-aws and aws-java-sdk-s3 jars, so advice you to add the hadoop-aws-3.3.4.jar and the aws-java-s3-x.y.jar to the libs folder of the JanusGraph distribution.

To get things working you can ignore the warning "org.apache.hadoop.util.NativeCodeLoader".

answered Jun 01 '23 at 05:44

HadoopMarc

1,356
3
11

Thanks a lot @HadoopMarc! We went ahead to using Janus 1.0.0-rc2 on your recommendation. After copying these two binaries (/usr/hdp/4.1.14.2/hadoop/hadoop-azure-3.1.1.4.1.14.2.jar:/usr/hdp/4.1.14.2/hadoop/azure-storage-7.0.1.jar) in the Janus lib location, we were able to go past the original error. But now are stuck on a new error. `org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink ` – Paritosh Gote Jun 01 '23 at 21:55
Actually, after adding all the hadoop binaries (/usr/hdp/4.1.14.2/hadoop/) to the Janus lib location we are now left with this error `Exception in thread "main" java.lang.AbstractMethodError: org.apache.hadoop.metrics2.sink.WasbAzureIaasSink.init(Lorg/apache/hadoop/shaded/org/apache/commons/configuration2/SubsetConfiguration;)V at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:209) ` – Paritosh Gote Jun 01 '23 at 22:17
Apparently, the cloudera jars miss a plugin that is configured somewhere. I guess you put the cluster conf on the JanusGraph CLASSPATH (good idea!). You might want and try to use customized hadoop conf/ folder in which all metrics configs are removed/disabled. JanusGraph/TinkerPop only call their local hadoop-client. BTW, copying all the jars is not a good idea because some jars now have identical classes, some of which JanusGraph may not like. It is better to copy the jars one by one navigating on ClassNotFoundExceptions. – HadoopMarc Jun 02 '23 at 06:14
Using https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-azure/3.3.4 instead of the cloudera one might also prevent problems regarding incompatible transitive dependencies. I see that hadoop-azure has far more dependencies than hadoop-aws, unfortunately. – HadoopMarc Jun 02 '23 at 07:30

Problem installing hadoop-gremlin with janusgraph

1 Answers1