1

I am trying to execute following query on Spark in yarn-client mode.

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi     --master yarn     --deploy-mode client         $SPARK_HOME/examples/target/scala-2.10/spark-examples*.jar     10

When I execute above query my application gets stuck on following

16/07/13 17:14:28 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:28 INFO yarn.Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1468430067384 final status: UNDEFINED tracking URL: http://hadoop-master:8088/proxy/application_1468428769910_0002/ user: nachiket

16/07/13 17:14:29 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:30 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:31 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

16/07/13 17:14:32 INFO yarn.Client: Application report for application_1468428769910_0002 (state: ACCEPTED)

I have already implemented most of suggestions mentioned on following link :

Application report for application_ (state: ACCEPTED) never ends for Spark Submit (with Spark 1.2.0 on YARN)

I am still facing the same issues . Are there any other solutions than above mentioned link ?

Finally Job fails with following clause 

    client token: N/A
         diagnostics: Application application_1468455134412_0001 failed 2 times due to Error launching appattempt_1468455134412_0001_000002. Got exception: org.apache.hadoop.net.ConnectTimeoutException: Call From sclab103/104.239.213.7 to 104.239.213.7:60640 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=104.239.213.7/104.239.213.7:60640]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:751)
        at org.apache.hadoop.ipc.Client.call(Client.java:1479)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy82.startContainers(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy83.startContainers(Unknown Source)
        at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
        at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=104.239.213.7/104.239.213.7:60640]
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
        at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
        at org.apache.hadoop.ipc.Client.call(Client.java:1451)
        ... 16 more
. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1468455280498
         final status: FAILED
         tracking URL: http://hadoop-master:8088/cluster/app/application_1468455134412_0001
         user: sclab
Community
  • 1
  • 1
npaluskar
  • 63
  • 9
  • you have to check memory configuration of your yarn cluster, how much is being allocated to your resource and node managers . – nat Jul 14 '16 at 17:47
  • Hi @nath , I am running only Spark Pi job and I am also making sure that no other job is running on the cluster. following is my yarn-site memory settings : yarn.nodemanager.resource.memory-mb 3072 yarn.scheduler.minimum-allocation-mb 512 yarn.scheduler.maximum-allocation-mb 3072 **One important thing that i have noticed is that i am not able to make connection with yarn through spark I tried `spark-shell --master yarn-client` but it gives same error as mentioned above.** – npaluskar Jul 14 '16 at 18:44
  • please check if your yarn cluster is up, check your yarn resource and node logs , it seems to be connectivity issue. – nat Jul 14 '16 at 21:21

0 Answers0