How to resolve a ConnectException when running a jar on Hadoop?

Question

I have written a simple map reduce job to perform KMeans clustering on some points.

However, when running the following command on Windows 10 cmd:

hadoop jar kmeans.jar KMeansJob /input /output

I get the following error:

21/04/08 22:26:14 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
21/04/08 22:26:14 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
21/04/08 22:26:17 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/????????/.staging/job_1617909910497_0001
Exception in thread "main" java.net.ConnectException: Call From PCNAME/XXX.XXX.X.X to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
        at org.apache.hadoop.ipc.Client.call(Client.java:1439)
        at org.apache.hadoop.ipc.Client.call(Client.java:1349)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
        at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1649)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1529)
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1526)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1526)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:327)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:237)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:106)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:70)
        at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:210)
        at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:128)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:101)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
        at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
        at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1889)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
        at clustering.KMeansJob.run(KMeansJob.java:43)
        at clustering.KMeansJob.main(KMeansJob.java:47)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.net.ConnectException: Connection refused: no further information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:687)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:790)
        at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:411)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1554)
        at org.apache.hadoop.ipc.Client.call(Client.java:1385)
        ... 43 more

In the above logs, I have hidden the IP the call was made from.

Running jps gives the following output:

4608 ResourceManager
13284 DataNode
7252 NameNode
10632 NodeManager
15436 Jps

What is the problem and is there any suggestion to cope with it?

This might be a longshot, but are the datanode and namenode processes for HDFS and/or YARN running? I don't know what the Win10-equivalent of `start-dfs.sh` or `start-yarn.sh` is, but this type of error has been brought to my system whenever I forgot to start the processes of Hadoop. — Coursal, Apr 08 '21 at 19:53
@Coursal yes, I have run both start-dfs.cmd and start-yarn.cmd before trying to run the jar. I have not run hadoop-env.sh though. Does this matter? — theodosis, Apr 08 '21 at 19:56
Does the `jps` command work in your shell? If so, do the two namenodes (primary and secondary) and one or several datanodes appear at the output? — Coursal, Apr 08 '21 at 19:57
@Coursal I have added the output of the jps command in my question details above! — theodosis, Apr 08 '21 at 20:02
That's interesting, everything seems to be ok? I tried to search the same error in SO and found this answer right here (https://stackoverflow.com/a/42281292/5644037) that suggests formatting the namenode. But it troubles me that all processes run just fine but the error persists, could this be an issue of Windows and Hadoop compatibility? — Coursal, Apr 08 '21 at 20:12
@Coursal In fact, I changed my core-site.xml configuration to `hdfs://localhost:9000` and this seems to work. Well, at the very least my jobs seem to start normally now. My previous config was: `hdfs://0.0.0.0:19000` as I had followed some tutorial steps on a blog. — theodosis, Apr 08 '21 at 20:16
Oh, that's lucky. Consider putting this solution of yours as an answer below, so people would try that out in case they ever stumble upon here. — Coursal, Apr 08 '21 at 20:20
@Coursal Do you have any idea as to why this works? I have just started using Hadoop, so I don't know much. It was indeed luck. — theodosis, Apr 08 '21 at 20:24
My best guess it that the `0.0.0.0` address doesn't really give the location of the HDFS, so the datanodes cannot access it even in a pseudo-distributed setup within a single PC. `localhost` clears things up by declaring that this machine is indeed the one with the HDFS services. — Coursal, Apr 08 '21 at 20:28

score 1 · Accepted Answer · answered Apr 08 '21 at 20:23

1

Changing the core-site.xml configuration seems to do the job:

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

For the record, my previous configuration had a value of:

<value>hdfs://0.0.0.0:19000</value>

answered Apr 08 '21 at 20:23

theodosis

894
5
15

In general, you should be using a remotely resolvable hostname or IP (which is what the official documentation for a pseudo distributed cluster says) and not localhost or 0.0.0.0 – OneCricketeer Apr 09 '21 at 05:12

How to resolve a ConnectException when running a jar on Hadoop?

1 Answers1