11

I am using openjdk:8-alpine for deploying Kafka Streams application. I am using Windowing and it crashes with below error:

Exception in thread "app-4a382bdc55ae-StreamThread-1" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni94709417646402513.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/librocksdbjni94709417646402513.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)
    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)
    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)
    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)
    at org.rocksdb.Options.<clinit>(Options.java:22)
    at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:116)
    at org.apache.kafka.streams.state.internals.Segment.openDB(Segment.java:43)
    at org.apache.kafka.streams.state.internals.Segments.getOrCreateSegment(Segments.java:91)
    at org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:100)
    at org.apache.kafka.streams.state.internals.RocksDBSessionStore.put(RocksDBSessionStore.java:122)
    at org.apache.kafka.streams.state.internals.ChangeLoggingSessionBytesStore.put(ChangeLoggingSessionBytesStore.java:78)
    at org.apache.kafka.streams.state.internals.ChangeLoggingSessionBytesStore.put(ChangeLoggingSessionBytesStore.java:33)
    at org.apache.kafka.streams.state.internals.CachingSessionStore.putAndMaybeForward(CachingSessionStore.java:177)
    at org.apache.kafka.streams.state.internals.CachingSessionStore.access$000(CachingSessionStore.java:38)
    at org.apache.kafka.streams.state.internals.CachingSessionStore$1.apply(CachingSessionStore.java:88)
    at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:142)
    at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:100)
    at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:127)
    at org.apache.kafka.streams.state.internals.CachingSessionStore.flush(CachingSessionStore.java:193)
    at org.apache.kafka.streams.state.internals.MeteredSessionStore.flush(MeteredSessionStore.java:169)
    at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:244)
    at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(AbstractTask.java:195)
    at org.apache.kafka.streams.processor.internals.StreamTask.flushState(StreamTask.java:332)
    at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:312)
    at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:208)
    at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:307)
    at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:297)
    at org.apache.kafka.streams.processor.internals.AssignedTasks$1.apply(AssignedTasks.java:67)
    at org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:357)
    at org.apache.kafka.streams.processor.internals.AssignedTasks.commit(AssignedTasks.java:347)
    at org.apache.kafka.streams.processor.internals.TaskManager.commitAll(TaskManager.java:403)
    at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:994)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:811)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)

Searching for the above issue, i came across https://issues.apache.org/jira/browse/KAFKA-4988. But it didn't helped.

So, Alpine uses musl-libc but it's not supported by RocksDB. The issue to add support for musl-libc to RocksDB: facebook/rocksdb#3143.

Question:Is there any openjdk docker image using which i can make my Kafka Stream application run and which won't give rocksdb issue?

Edit-1: I tried RUN apk add --no-cache bash libc6-compat, but it too fails with below error:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000000011e336, pid=1, tid=0x00007fc6a3cc8ae8
#
# JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.9.0
# Distribution: Custom build (Tue Oct 23 11:27:22 UTC 2018)
# Problematic frame:
# C  0x000000000011e336
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
bratkartoffel
  • 1,127
  • 1
  • 13
  • 37
mukesh210
  • 2,792
  • 2
  • 19
  • 41
  • have you tried this solution ? https://github.com/wurstmeister/kafka-docker/issues/435#issuecomment-444999853 – Mostafa Hussein Mar 04 '19 at 13:49
  • @MostafaHussein Yes. I tried it and it gave `Segmentation Fault` error. – mukesh210 Mar 04 '19 at 13:53
  • 1
    Have you tried using another base image? Like `adoptopenjdk/openjdk8:slim` – bratkartoffel Mar 04 '19 at 14:01
  • @bratkartoffel No, i haven't tried any other openjdk variant. Will it solve my problem? – mukesh210 Mar 04 '19 at 14:04
  • @Mukeshprajapati I have tried to install the package using: `apk add --no-cache bash libc6-compat` and it worked. Can you ensure that you have the latest version of the image ? image id `792ff45a2a17` – Mostafa Hussein Mar 04 '19 at 14:05
  • @Mukeshprajapati I think so as this will change the underlying libc from `musl` to `glibc`. This also means you're no longer using alpine though. – bratkartoffel Mar 04 '19 at 14:07
  • @bratkartoffel I will try it and will let you know. – mukesh210 Mar 04 '19 at 14:08
  • @MostafaHussein How can i find image id of alpine? – mukesh210 Mar 04 '19 at 14:10
  • @Mukeshprajapati execute `docker images` and check the output list. also you can run the following command to pull the latest image `docker pull openjdk:8-alpine` – Mostafa Hussein Mar 04 '19 at 14:11
  • @MostafaHussein In dockerFile, i am using `FROM openjdk:8-alpine`. Won't this pull latest alpine image? – mukesh210 Mar 04 '19 at 14:12
  • If the image already exist it won't updated. i am not sure when you installed it for the first time. that's why i asked you to run this command `docker pull openjdk:8-alpine` to ensure that you have the latest. On my machine i have pulled the image and installed `libc6-compat` without any issues – Mostafa Hussein Mar 04 '19 at 14:15
  • @MostafaHussein I will try pulling the latest image and adding `libc6-compat`. I will let you know if this works. – mukesh210 Mar 04 '19 at 14:16
  • The segfault after you've installed `libc6-compat` could indicate another kind of incompatiblity. Could you provide more context on your `Dockerfile` and application? Also, check whether a `hs_err_*.log` (JVM crash log) was created - it may provide more information. – valiano Mar 05 '19 at 06:48
  • @MostafaHussein Even after pulling latest alpine image and adding `libc6-compat`, it is giving the same error. – mukesh210 Mar 08 '19 at 06:01
  • @MostafaHussein Can you please post small snippet of your dockerfile. – mukesh210 Mar 08 '19 at 06:27
  • @Mukeshprajapati here you go, https://gist.github.com/mostafahussein/2113366788ba98b217d213bf4450051b Can you pelase remove the alpine image from your device then pull it again ? As there might be an old layer causing the failure. Also note that the current image is even **newer**: `e9ea51023687 built 6 hours ago` – Mostafa Hussein Mar 08 '19 at 08:27
  • Thanks @MostafaHussein. Currently i got it working with `adoptopenjdk/openjdk8:alpine-slim`. – mukesh210 Mar 08 '19 at 09:32
  • @Mukeshprajapati Should I submit my comment as an answer ? or you did not try it ? – Mostafa Hussein Mar 08 '19 at 09:37
  • @MostafaHussein I tried exact same thing but it didn't worked. Really appreciate your help. – mukesh210 Mar 08 '19 at 10:17

4 Answers4

15

The solution which worked for me was to change the docker image from openjdk:8-alpine to adoptopenjdk/openjdk8:alpine-slim.

adoptopenjdk/openjdk8:alpine-slim is glibc compatible.

I came to know about this image from http://blog.gilliard.lol/2018/11/05/alpine-jdk11-images.html.

Hope it helps someone.

mukesh210
  • 2,792
  • 2
  • 19
  • 41
1

The ticket you've linked, https://issues.apache.org/jira/browse/KAFKA-4988, gives a lot of insight into the issue.

As noted, it looks RocksDB is not compatible with musl libc, and that glibc would be required.

Installing libc6-compact may not do: it provides a compatiblity layer over musl libc that mimics glibc library structure and implements some missing functionatlies, but that's not the same as installing glibc per-se. glibc is a complex implementation, so there may not be a 1-on-1 correlation between the compatiblilty library and the actual glibc. See here for some subtle musl/glibc differences.

Reading the ticket comments, the faulty library may be librocksdbjni.so, which depends on libstdc++6.

Therefore, I would try the following (with openjdk:8-alpine as your base image):

valiano
  • 16,433
  • 7
  • 64
  • 79
  • I tried... but unfortunately it is failing with Segmentation Fault. – mukesh210 Mar 08 '19 at 06:27
  • @Mukeshprajapati I see. Could you try using `anapsix/alpine-java:8` as the base image? It is Alpine 3.8 + glibc + Oracle Java 8 (https://hub.docker.com/r/anapsix/alpine-java). It may be more compatible with Kafka. – valiano Mar 08 '19 at 07:52
  • @Mukeshprajapati Also, is JVM crash log (`hs_err*.log`) being generated? If it is, please post it - it could help diagnosing the exact issue. – valiano Mar 08 '19 at 07:54
  • 3
    I got it working with `adoptopenjdk/openjdk8:alpine-slim`. Thanks for helping. – mukesh210 Mar 08 '19 at 09:33
  • @Mukeshprajapati Good job! I see that image is glibc-enabled, too, using the ordinary (glibc compatible) OpenJDK Linux binaries. – valiano Mar 08 '19 at 10:55
  • Yupp... had to do lot of Google searching :) – mukesh210 Mar 08 '19 at 12:49
1

Rather then changing your default docker base image, you can build glibc for an Alpine distro. Even better than that, you can go and grab the pre-built apk from Sasha Gerrand's github page. Here's what we added to our Dockerfile to get this all working with his prebuilt apk:

# # GLIBC - Kafka Dependency (RocksDB)
# Used by Kafka for default State Stores.
# glibc's apk was built for Alpine Linux and added to our repository
# from this source: https://github.com/sgerrand/alpine-pkg-glibc/
ARG GLIBC_APK=glibc-2.30-r0.apk
COPY ${KAFKA_DIR}/${GLIBC_APK} opt/
RUN apk add --no-cache --allow-untrusted opt/${GLIBC_APK}

# C++ Std Lib - Kafka Dependency (RocksDB)
RUN apk add --no-cache libstdc++
stackunderflow
  • 10,122
  • 4
  • 20
  • 29
1

There is a known issue about an incompatibility of Kafka Streams and Alpine linux from https://issues.apache.org/jira/browse/KAFKA-4988. For those who use Java 11, adoptopenjdk/openjdk11:alpine-slim works fine for me. An other solution is to still use openjdk:11-jdk-alpine image as base one, but then install snappy-java lib manually

FROM openjdk:11-jdk-alpine
RUN apk update && apk add --no-cache gcompat
...
V-Q-A NGUYEN
  • 1,497
  • 16
  • 19