9

I am trying a write a file to hdfs using scala and I keep getting the following error 

Caused by: org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
at org.apache.hadoop.ipc.Client.call(Client.java:1113)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at bcomposes.twitter.Util$.<init>(TwitterStream.scala:39)
at bcomposes.twitter.Util$.<clinit>(TwitterStream.scala)
at bcomposes.twitter.StatusStreamer$.main(TwitterStream.scala:17)
at bcomposes.twitter.StatusStreamer.main(TwitterStream.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)

I installed hadoop following this tutorial. The code below is what I use to insert a sample file to hdfs.

val configuration = new Configuration();
val hdfs = FileSystem.get( new URI( "hdfs://192.168.11.153:54310" ), configuration );
val file = new Path("hdfs://192.168.11.153:54310/s2013/batch/table.html");
if ( hdfs.exists( file )) { hdfs.delete( file, true ); } 
val os = hdfs.create( file);
val br = new BufferedWriter( new OutputStreamWriter( os, "UTF-8" ) );
br.write("Hello World");
br.close();
hdfs.close();

The Hadoop version is 2.4.0 and hadoop library version I use is 1.2.1. What change should I do to make this work?

Rahul
  • 44,892
  • 25
  • 73
  • 103

3 Answers3

6

I had the same problem using Hadoop 2.3 and I've solved it adding the following lines to my build.sbt file :

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.3.0"

libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.3.0"

So I think in your case you case use the 2.4.0 version.

PS: It also worked on your code sample. I hope it will help

eliasah
  • 39,588
  • 11
  • 124
  • 154
  • please give the location of build.sbt file. – Manish Bhadani Feb 20 '15 at 17:28
  • Which build tool are you using? SBT by any luck? – eliasah Feb 20 '15 at 17:31
  • hadoop2.5.0 and eclipse kepler version. – Manish Bhadani Feb 20 '15 at 17:37
  • Maven, gradle or SBT? Or you don't know what I'm actually talking about? – eliasah Feb 20 '15 at 17:45
  • I am new in hadoop/hdfs. and also i have no idea about this three tool. but what i was trying to tell you is i want to run java program in eclipse which store data in hdfs but i am stuck in this error. Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4. – Manish Bhadani Feb 20 '15 at 17:57
  • Also i replaced hadoop-client.jar version but error still remain same. If you have any suggestions please help. – Manish Bhadani Feb 20 '15 at 17:58
  • Using a build automation tool is one of the basics of software development. I can't explain to you that here, not in a comment a least. For that you need to do some research over the Internet. Bottom line, to solve your issue you'll need to include the appropriate dependencies in your project. NB: I suppose that your Hadoop cluster in actually running normally. – eliasah Feb 20 '15 at 18:32
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/71393/discussion-between-manish-bhadani-and-eliasah). – Manish Bhadani Feb 21 '15 at 17:04
3

As said in error message Server IPC version 9 cannot communicate with client version 4 your server has slighly newer version, than your client. You have to either downgrade your hadoop cluster (most likely not an option) or upgrade your client library from 1.2.1 to 2.x version.

om-nom-nom
  • 62,329
  • 13
  • 183
  • 228
  • My Hadoop-core lib version is 1.2.1(latest: https://repository.apache.org/content/repositories/releases/org/apache/hadoop/hadoop-core/) and my hadoop-client lib version is 2.4.0. Still I get the same error. – Rahul May 13 '14 at 16:38
  • `hadoop-core` isn't what you want then. You want some combination of the 2.4.0 libraries which includes `hadoop-common`, `hadoop-client`, ... – Mike Park May 13 '14 at 18:06
  • Yep, they have [changed artifact name](http://stackoverflow.com/a/15202099/298389) – om-nom-nom May 13 '14 at 18:19
1

hadoop and spark versions should be in sync. (In my case, I am working with spark-1.2.0 and hadoop 2.2.0)

STEP 1 - goto $SPARK_HOME

STEP 2 - Simply mvn build spark with the version of hadoop client you want, 

mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests clean package

STEP 3 - Also spark project should have proper spark version, 

name := "smartad-spark-songplaycount"

version := "1.0"

scalaVersion := "2.10.4"

//libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.2.0"

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.2.0"

libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.2.0"

resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

References

Building apache spark with mvn

prayagupa
  • 30,204
  • 14
  • 155
  • 192