Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory"

Question

I am using the confluent-kafka Python client in my project. I'm trying to create a Docker image with this client.

I am facing the following error:-

#11 8.015 [pipenv.exceptions.InstallError]:       In file included from /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/Admin.c:17:
#11 8.015 [pipenv.exceptions.InstallError]:       /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
#11 8.015 [pipenv.exceptions.InstallError]:          23 | #include <librdkafka/rdkafka.h>
#11 8.015 [pipenv.exceptions.InstallError]:             |          ^~~~~~~~~~~~~~~~~~~~~~
#11 8.015 [pipenv.exceptions.InstallError]:       compilation terminated.
#11 8.015 [pipenv.exceptions.InstallError]:       error: command '/usr/bin/gcc' failed with exit code 1
#11 8.016 [pipenv.exceptions.InstallError]:       [end of output]

Based on my search it is related to Apple M1 build for librdkafka.

Please [edit] to post the relevant parts of your Dockerfile. Please include the base image `FROM`, and any other libraries/packages you tried to install, including confluent-kafka. — Gino Mempin, May 09 '22 at 05:27
If you're trying to build an ARM image, then yes, you'll probably have to use kafka-python instead — OneCricketeer, May 09 '22 at 05:35

score 24 · Answer 1 · answered Oct 10 '22 at 20:45

24

If you are installing confluent-kafka inside Docker container on M1, this helped me.

RUN apt-get update && \
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && \
  cd /tmp && git clone https://github.com/edenhill/librdkafka.git && \
  cd librdkafka && git checkout tags/v1.9.0 && \
  ./configure && make && make install && \
  cd ../ && rm -rf librdkafka

answered Oct 10 '22 at 20:45

y0j0

3,369
5
31
52

1

Specifically referring to the question about building the image in Docker - This works. I had to bump the tag to v2.02 but otherwise solves the problem. – Spanners Jan 25 '23 at 16:52

score 13 · Answer 2 · answered Jun 27 '22 at 11:37

13

Please refer to this Github issue. The problem with M1 is that Homebrew is installed in a different location and so these variables need to be added to the environment by including these lines in your .zshrc file

export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib

answered Jun 27 '22 at 11:37

Venkatesh Gopalakrishnan

311
2
8

6

why should it matter when building a docker image? the host platform should be completely independent right? – Yash Kumar Verma Sep 14 '22 at 11:27
1

The same problem occurs on a MacOS host. The title of the question doesn't refer to a docker. This answer is not about the docker. – Ophir Carmi Dec 14 '22 at 10:23
thank you so much, sir! your comment saved me – mr_bulrathi Jun 22 '23 at 18:08
Worked Flawlessly! – 5parkp1ug Jul 27 '23 at 16:48

score 7 · Answer 3 · answered Jun 06 '22 at 08:42

The Confluent engineers are obviously very focused on their paying customers, and many, many months after the release of Python 3.10, they still haven't released 3.10 wheels that include the binaries for the package.

https://github.com/confluentinc/confluent-kafka-python/issues/1219

You have two choices. 1, run Python 3.9; 2, install librdkafka-dev via apt before installing confluent-kafka (for a debian base image).

The issue is the same with Apple M1 - i.e, there are no pre-built wheels for the platform containing the binaries, but it is much easier to fix (by installing librdkafka-dev) if you are trying to build a amd64 Linux image.

Will the 1st choice work for Alphine images as well? – potterson11 Dec 13 '22 at 11:54 — potterson11, Dec 13 '22 at 11:54

Phoebe · Answer 4 · 2023-02-14T02:48:15.427

I added these to my docker-compose.yml (according to https://stackoverflow.com/a/74020511/20575677 and https://github.com/confluentinc/confluent-kafka-python/issues/65#issuecomment-269964346)

RUN apt-get update && \
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && \
  cd /tmp && git clone https://github.com/edenhill/librdkafka && \
  cd librdkafka && git checkout tags/v2.0.2 && \
  ./configure && make && make install && \
  ldconfig &&\
  cd ../ && rm -rf librdkafka

RUN pip install confluent-kafka==2.0.2

score 1 · Answer 5 · answered Jul 27 '22 at 23:47

1

On a non-M1 mac, Python 3.10.2, I solved this with brew install librdkafka.

answered Jul 27 '22 at 23:47

numbers are fun

423
1
7
12

score 0 · Answer 6 · answered May 11 '22 at 08:35

Prebuilt binary wheels of confluent-kafka-python for your Apple M1 seems to not exist.

Thus, in order to install the package, pip tries to build it itself from source. But librdkafka looks like to not be installed as per librdkafka/rdkafka.h: No such file or directory error you got.

You can find specific instructions to install from source for different platforms here.

score -1 · Answer 7 · answered Jul 11 '22 at 23:24

-1

I solved this problem trying with Python 3.9.13 on a virtual environment.

answered Jul 11 '22 at 23:24

Juan Mite

27
1
2

Could you add some detail about your virtual env set up? – PirateNinjas Jul 15 '22 at 16:23
Sure. 1 Navigate to the folder using cd 2 python3 -m venv env 3 source env/bin/activate Now you can install Python 3.9.13 and install confluent-kafka normally. Ctrl+D for quit from virtual env. – Juan Mite Aug 20 '22 at 03:10

Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory"

7 Answers7