0

Case

Trying to build some code which requires Kafka libraries and will be deployed to an HDP platform. Build succeeds but at runtime complains that classes cannot be found. Looking up Kafka version on HDP platform reveals something like 0.10.0.2.5.3.16-1, a quick check on mvn reveals that no such Kafka library exists for this version. The nearest version is 0.10.0.2.5.3.42-1 which I'm using.

Symptoms

Runtime errors which say that classes or methods cannot be found, for instance kafka.javaapi.consumer.SimpleConsumer.<init>(Ljava/lang/String;IIILjava/lang/String;Ljava/lang/String;)V

TheRealJimShady
  • 3,815
  • 4
  • 21
  • 40
  • 1
    Please show us the **actual code** you tried. – Adam Arold Oct 18 '17 at 13:38
  • 1
    You need to use the same version to compile and to run. Otherwise weird things happen, like methods disappear etc. – Kayaman Oct 18 '17 at 13:39
  • @Kayaman can you see anything in particular which implies that runtime is different to what is being built? – TheRealJimShady Oct 18 '17 at 13:45
  • Yes. The `NoSuchMethodError`. – Kayaman Oct 18 '17 at 13:45
  • suppose I walked right into that one... – TheRealJimShady Oct 18 '17 at 13:47
  • It's pretty simple. You've compiled the code with a library that has the method (naturally, it wouldn't compile otherwise). At runtime the method is missing and the error is thrown. You could check for the differences in the constructors of `SimpleConsumer` and use one that exists, but a lot safer and smarter is to just use the proper version to compile. – Kayaman Oct 18 '17 at 13:48
  • yes I realise that... the kafka version I've specified which is on the cluster (0.10.0.2.5.3.16-1) does not exist as a maven artefact, so how can compile match runtime? – TheRealJimShady Oct 18 '17 at 13:53
  • I notice you have `provided` scope for your dependencies. This means they will not be part of your artifact, but expected to be provided by the runtime environment... If you want them bundled, use `compile` instead... – Per Huss Oct 18 '17 at 13:53
  • I initially set to compile but wanted to make sure I was using jars on the cluster rather than my own... in any case I get the same error. – TheRealJimShady Oct 18 '17 at 13:55
  • What's the nearest version in maven? Try that (older/newer). – Kayaman Oct 18 '17 at 14:01
  • nearest version is what I'm using (0.10.0.2.5.3.42-1). I'm now finding out if 0.10.0.2.5.3.16-1 is something weird HDP have come up with, and if so which version to build with. – TheRealJimShady Oct 18 '17 at 14:03
  • ok so, in kafka version 0.10.0.2.5.3.16-1, the 2.5.3 is actually the HDP version and nothing to do with kafka... the actual version of kafka is 0.10.0.1, but to use against spark 1.6.2 you need to use 0.8.2.1 – TheRealJimShady Oct 18 '17 at 16:09
  • @Kayaman I'd like to write up the answer to this but I don't seem to be able to, I think it might be because you've marked it as duplicate. Would you kindly unmark it as duplicate. Thank you – TheRealJimShady Oct 19 '17 at 08:47
  • Try making the title a bit more descriptive too, otherwise it'll just look like a generic version mismatch. – Kayaman Oct 19 '17 at 09:08
  • @Kayaman anything you would change? – TheRealJimShady Oct 24 '17 at 08:23
  • Yeah, that's better. – Kayaman Oct 24 '17 at 08:30
  • excellent, thanks for your input :) – TheRealJimShady Oct 24 '17 at 08:31

1 Answers1

0

Reason

The services provided in the HDP platform are suffixed with the HDP version, so Kafka 0.10.0.2.5.3.16-1 is actually Kafka version 0.10.0, and digging further reveals that it's version 0.10.0.1. The HDP version is 2.5.3.

Solution

Drop back ${kafka.version} in pom to a version which works, I found that 0.8.2.1 worked for me.

TheRealJimShady
  • 3,815
  • 4
  • 21
  • 40