Spark job server installation on hdisght

Question

I have created a azure hdinsight spark cluster and I am trying to install Spark Job Server on the headnode. I have ssh'ed into the headnode and here are the steps I am following

Its HDI 3.5 with Spark 1.6.3

1. Install SBT:

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823

sudo apt-get update

sudo apt-get install sbt

2. git clone https://github.com/spark-jobserver/spark-jobserver.git

The master is for spark 1.6.3, so I am not changing branches.

3. Moving into the spark-observer directory

4. sbt assembly At this point I am getting errors like this

*

[warn] Merging 'reference.conf' with strategy 'concat'
[error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/kmk/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.37.Final.jar:META-INF/io.netty.versions.properties
    at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140)
    at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
    at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
    at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
    at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
    at sbtassembly.Assembly$.apply(Assembly.scala:83)
    at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:240)
    at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:237)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: Building python API received non-zero exit code 1
    at scala.sys.package$.error(package.scala:27)
    at PythonTasks$.buildPythonTask(PythonTasks.scala:25)
    at $b9e935155022d705b7b0$$anonfun$jobServerPythonSettings$5.apply(build.sbt:110)
    at $b9e935155022d705b7b0$$anonfun$jobServerPythonSettings$5.apply(build.sbt:110)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] (root/*:assembly) deduplicate: different file contents found in the following:
[error] /home/kmk/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] (job-server-python/*:buildPython) Building python API received non-zero exit code 1
[error] Total time: 151 s, completed Apr 1, 2017 3:41:54 PM

*

What I am doing wrong here ?

Answer 1

According to your error information, per my experience, I think the issue was caused by the conflict of different version for these jars which be named by same package name. So a normal way is to try to delete these duplicate jars. Or you can refer to the offical document Excluding JARs and files for sbt to resolve dependencies via merge strategy, there is a SO thread Resolving Dependencies in creating JAR through SBT assembly which may help for you as reference.

Meanwhile, according to manual steps for spark-job-server deployment, the sbt job-server/assembly operation seems not to be necessary, which depend on your current envionment.

Hope it helps.