0

I have a program of Kafka streams. It's Windows 64 bit machine and standalone Kafka server is running on it. Java version is java 8. In code pom has dependencies of Kafka client and streams APIs and the versions are latest i.e. 0.10.2. Whenever I am running the streams app, it is looking for rocksdb's dll file in my user home and failed. Anyone faced the same issue? Same code is running on different windows 64 machine.

pom

<dependencies>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.10.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-streams</artifactId>
        <version>0.10.2.1</version>
    </dependency>
</dependencies>

Stacktrace

Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError: C:\Users\abcd\AppData\Local\Temp\librocksdbjni8989756873626302713.dll: Can't find dependent libraries
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
at org.rocksdb.Options.<clinit>(Options.java:22)
at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:117)
at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:38)
at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:76)
at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:73)
at org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.put(ChangeLoggingSegmentedBytesStore.java:55)
at org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.put(MeteredSegmentedBytesStore.java:101)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:110)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:102)
at org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:65)
at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:48)
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:134)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:83)
at org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:44)
at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:48)
at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:134)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:83)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:70)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:197)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:627)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)
Mudit bhaintwal
  • 528
  • 1
  • 7
  • 21
  • See https://stackoverflow.com/questions/43742423. – miguno Jun 22 '17 at 11:59
  • but that is a mere hack. It's 2017 and Kafka doesn't have full windows support!!! – Mudit bhaintwal Jun 22 '17 at 15:45
  • Strictly speaking, it's a RocksDB issues -- as Streams API depends on RocksDB by default, we inherit this problem from there. You can also replace RocksDB with another storage engine that works on Windows. Mid to long term, we want to have better Windows support thought -- but we are not there atm. It does work for some Windows versions though... – Matthias J. Sax Jun 22 '17 at 16:29
  • Thanks Matthias. Any pointer how to replace and which DB perform good in place of Rocksdb. – Mudit bhaintwal Jun 22 '17 at 16:48
  • Mudit: Which Windows version are you using? My quick test with Kafka's Streams API on Windows 10 (w/ Oracle Java 8) worked out of the box. On the server side of Kafka, btw, it is not recommended to use Windows to run in production. – miguno Jun 23 '17 at 09:30
  • Michael - It's Win 7, service pack-1, 64 bit and java 1.8. My development machine is a windows one so no option of moving to Linux. Is there any way to change this default rocks db to something that works in windows. Moreover one of my friend having slimier config is able to run same code!!! I am totally confused now with this strange behavior. – Mudit bhaintwal Jun 23 '17 at 09:54
  • The main question is, if RocksDB support this version in the first place. You should talk to RocksDB folks: https://github.com/facebook/rocksdb/issues Nevertheless, you can plug in any store into Kafka Stream: You can implements the `StateStoreSupplier` and `StateStore` interface and hand the store to you operator via an according overload that does accept a store supplier. Thus, you can replace RocksDB with any other store you want to use. – Matthias J. Sax Jun 26 '17 at 23:09

0 Answers0