2

I've been trying to build a simple consumer for Kafka in C++ using VsCode in MacOS (Catalina version 10.15.7) but I hit the same linkage error. I tried to find some answers around here, but the ones I found are not useful for me (as this one, for example) because I can build Kafka properly.

Here are the steps I tried up to now and the outputs of the compiler/linker in each case. I didn't include the code as it is a simple code and the errors come from the linker, although I will ad it if someone thinks it could help. Let me know if I should include anything else.

Installing with brew

First I installed kafka libraries using brew with the following command: brew install librdkafka After installation, the libraries were located in /usr/local/Cellar/librdkafka/1.7.0/lib/ and /usr/local/Cellar/librdkafka/1.7.0/include/, which were the paths I used in task.json for the default build in VsCode.

The error I got was this:

Undefined symbols for architecture x86_64:
  "RdKafka::get_debug_contexts()", referenced from:
      _main in read-topics-d64669.o
  "RdKafka::Conf::create(RdKafka::Conf::ConfType)", referenced from:
      _main in read-topics-d64669.o
  "RdKafka::Topic::OFFSET_BEGINNING", referenced from:
      _main in read-topics-d64669.o
  "RdKafka::Consumer::create(RdKafka::Conf const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&)", referenced from:
      _main in read-topics-d64669.o
...

(shortened for clarity). If I use IntelliSense to look for the definition of any Kafka function used in my code, it finds it properly opening the library file in the correct folder ('/usr/local/Cellar/librdkafka/1.7.0/include/librdkafka/rdkafkacpp.h' ) as added in tasks.json (Reference 1). However the linker seems to have problems finding the library file that is also added to the linker parameters.

Compiling from sources

Then I tried to compile Kafka directly from sources following the instructions provided in the repository's documentation. To be sure I was not missing anything, I added the option that automatically installs the dependencies needed: ./configure --install-deps.

The only suspicious message from that command was this one

WARNING: librdkafka-static.a: Not creating self-contained static library librdkafka-static.a: no static libraries available/enabled

but everything looked good and I don't think that's the real issue. Them, I installed them with make install and the output is available below (Ref 3).

Then I modified tasks.json to have the correct path to both include and lib folders (Reference 2). However, building still fails due to the linker and I have the same error as before:

Undefined symbols for architecture x86_64:
  "RdKafka::get_debug_contexts()", referenced from:
      _main in read-topics-991879.o
  "RdKafka::Conf::create(RdKafka::Conf::ConfType)", referenced from:
      _main in read-topics-991879.o
  "RdKafka::Topic::OFFSET_BEGINNING", referenced from:
      _main in read-topics-991879.o
 ...

Any idea of what I could try to solve the error? What I am doing wrong or what am I missing? Any help or suggestion will be appreciated.

References

  1. tasks.json with Kafka installed with brew
{
    "version": "2.0.0",
    "tasks": [
        {
            "type": "shell",
            "label": "C/C++: clang build active file",
            "command": "/usr/bin/clang",
            "args": [
                "-std=c++17",
                "-stdlib=libc++",
                "-g",
                "-v",
                "-I/usr/local/Cellar/librdkafka/1.7.0/include/librdkafka",
                "-L/usr/local/Cellar/librdkafka/1.7.0/lib/",
                "${file}",
                "-o",
                "${fileDirname}/${fileBasenameNoExtension}"
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            },
            "problemMatcher": [
                "$gcc"
            ],
            "group": {
                "kind": "build",
                "isDefault": true
            }
        }
    ]
}
  1. tasks.json with Kafka compiled from source (only the part that changed, everything else is the same)
...
"args": [
                "-std=c++17",
                "-stdlib=libc++",
                "-g",
                "-v",
                "-I/usr/local/include/librdkafka/",
                "-L/usr/local/lib",
                "${file}",
                "-o",
                "${fileDirname}/${fileBasenameNoExtension}"
            ],
...
  1. Kafka Build Output
Install librdkafka to /usr/local
install -d $DESTDIR/usr/local/include/librdkafka
install -d $DESTDIR/usr/local/lib
install rdkafka.h rdkafka_mock.h $DESTDIR/usr/local/include/librdkafka
install librdkafka.a $DESTDIR/usr/local/lib
[ ! -f librdkafka-static.a ] || install librdkafka-static.a $DESTDIR/usr/local/lib
install librdkafka.1.dylib $DESTDIR/usr/local/lib
[ -f "rdkafka.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka.pc $DESTDIR/usr/local/lib/pkgconfig \
    )
[ -f "rdkafka-static.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka-static.pc $DESTDIR/usr/local/lib/pkgconfig \
    )
(cd $DESTDIR/usr/local/lib && ln -sf librdkafka.1.dylib librdkafka.dylib)
Install librdkafka++ to /usr/local
install -d $DESTDIR/usr/local/include/librdkafka
install -d $DESTDIR/usr/local/lib
install rdkafkacpp.h $DESTDIR/usr/local/include/librdkafka
install librdkafka++.a $DESTDIR/usr/local/lib
[ ! -f librdkafka++-static.a ] || install librdkafka++-static.a $DESTDIR/usr/local/lib
install librdkafka++.1.dylib $DESTDIR/usr/local/lib
[ -f "rdkafka++.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka++.pc $DESTDIR/usr/local/lib/pkgconfig \
    )
[ -f "rdkafka++-static.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka++-static.pc $DESTDIR/usr/local/lib/pkgconfig \
    )
(cd $DESTDIR/usr/local/lib && ln -sf librdkafka++.1.dylib librdkafka++.dylib)
WARNING: librdkafka-static.a: Not creating self-contained static library librdkafka-static.a: no static libraries available/enabled
Generating pkg-config file rdkafka-static.pc
Checking librdkafka integrity
librdkafka.1.dylib             OK
librdkafka.a                   OK
Symbol visibility              OK
Generating pkg-config file rdkafka++-static.pc
Checking librdkafka++ integrity
librdkafka++.1.dylib           OK
librdkafka++.a                 OK
/Library/Developer/CommandLineTools/usr/bin/make -C examples
make[1]: Nothing to be done for `all'.
Updating CONFIGURATION.md
Installing documentation to /usr/local
install -d $DESTDIR/usr/local/share/doc/librdkafka
install LICENSE LICENSES.txt INTRODUCTION.md README.md CONFIGURATION.md STATISTICS.md CHANGELOG.md $DESTDIR/usr/local/share/doc/librdkafka
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
VictorHMartin
  • 325
  • 5
  • 20

2 Answers2

1

cmake solution CMakeLists.txt

include_directories(/usr/local/Cellar/librdkafka/1.8.2/include/) 

link_directories(/usr/local/Cellar/librdkafka/1.8.2/lib/) 

set(CMAKE_EXE_LINKER_FLAGS "-lrdkafka++ -lrdkafka")

option

$ pkg-config --libs rdkafka
$ pkg-config --libs rdkafka++
Jonas
  • 11
  • 2
0

Finally I got it working so I will write it here in case it helps someone else.

The way to solve it was with the second option, compiling Kafka library from sources. The thing that was missing was to add the link option for the library. What I mean is that I added the options to find the includes and binary library file

"-I/usr/local/Cellar/librdkafka/1.7.0/include/librdkafka",
"-L/usr/local/Cellar/librdkafka/1.7.0/lib/",

but that was for the compiler to find those. It needs the command to tell the linker to use that library which is "-lrdkafka++",.

The last thing that I got wrong was specifying to Clang that it should compile C++ code, and that's done changing this "command": "/usr/bin/clang++",. I hope this helps others that had the same problems I did.

VictorHMartin
  • 325
  • 5
  • 20