1

I want to run Kafka from IDEA and I am getting the following error:

> Task :core:Kafka.main()
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details.

> Task :core:Kafka.main() FAILED

Execution failed for task ':core:Kafka.main()'.
> Process 'command '/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

I am able to run Kafka from the terminal. I run Zookeeper from terminal and then Run Kafka from IDEA.

Steps that I follow to run Kafka

As usual, I run the command ./gradlew jar to build Kafka from the source, using the terminal. I open the project in IDEA using idea . from the root directory of the cloned repository.

I opened the file: core/src/main/scala/kafka/Kafka.scala

I then navigate to main() function and click on the green triangle.

This helps me get the Run configurations and it fails. I then add the config/server.properties in the Program Arguments to make the Run config look like this: enter image description here

Upon running with the above configurations, I get the aforementioned error.

I searched a bit and found that the same issue was resolved by adding dependencies, like mentioned here and here but I could not understand how I can add dependency as I do not use Maven and I cannot find the file pom.xml file as described here.

Update 1

I tried to add the exact dependency as stated from the terminal to the Run configurations, I am unsure whether that is added or not because the result is still the same: added dependency

This is what I find when I run Kafka from the terminal:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/core/build/dependant-libs-2.13.5/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/tools/build/dependant-libs-2.13.5/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/aviralsrivastava/dev/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2021-03-23 10:54:59,524] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)

From the second last line, I deduced the dependency name and added it as shown in the last image.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Aviral Srivastava
  • 4,058
  • 8
  • 29
  • 81
  • Add that dependency to the app CLASSPATH. – duffymo Mar 23 '21 at 17:50
  • Thanks @duffymo! How do I add dependency to an app? – Aviral Srivastava Mar 23 '21 at 17:50
  • Add the JAR to the classpath when you run. -d cp on the command line. – duffymo Mar 23 '21 at 17:51
  • In IDEA, I can select only one module using the dropdown. Can you tell me how do I add the JAR to the classpath? I also did not get "-d cp on the command line", sorry! – Aviral Srivastava Mar 23 '21 at 17:53
  • Also, should I add this dependency using the "+" sign in the "Code Coverage" section as shown in the image? Is that what you were saying? – Aviral Srivastava Mar 23 '21 at 17:54
  • Can you explain your use case for running Kafka server from source code? And coverage settings aren't going to fix your issue; it's definitely classpath related, which the bin scripts explicitly define – OneCricketeer Mar 24 '21 at 05:00
  • I want to contribute to Kafka and so, I think I should be able to run Kafka's source code in an IDE with a debugger ready so that I can experiment with it and carry out my coding. – Aviral Srivastava Mar 24 '21 at 22:15
  • You are running it, though, and your logger is working - `[2021-03-23 10:54:59,524] INFO Registered...` If you want to use a different log configuration (the one in the config folder), you must pass `-Dlog4j.configurationFile` in the program arguments – OneCricketeer Mar 27 '21 at 15:00

2 Answers2

0

Generally speaking, the classpath needs configured correctly, to isolate a single Slf4j implementation (the one in the core module, specifically), but messing with the build scripts should be avoided to fix your runtime dependencies within a module

To fix your logger, you need to pass in -Dlog4j.configuration=config/log4j.properties as a VM option (I think you'll have to toggle the drop-down where it says -cp kafka.core.main to get that input to show)

If you want to emulate actual runtime behavior of the server, and attach a debugger, setup your breakpoints, open a terminal (assuming you are using Zookeeper, and it is already running somewhere else, otherwise you need a new terminal for it)

export KAFKA_DEBUG=y
export DEBUG_SUSPEND_FLAG=y

bin/kafka-server-start config/server.properties

Then add a run configuration for a remote application and attach it to port 5005

Once it attaches, your breakpoint should take focus, and you can step through the code

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
0

I was also facing the same issue while trying to set up the Apache Kafka source code in Intellij IDEA and here's what I have tried to solve the SL4J issue.

Set the JVM arguments as highlighted in the picture below:

IDEA Screenshot

Also, in the build.gradle file, add the following dependency under section project(':core') after the line implementation libs.commonsCli

implementation libs.slf4jlog4j

Tested on Java 17, Apache Kafka 3.1.0 and Intellij IDEA 2021.3.1

A Thomas
  • 113
  • 1
  • 1
  • 7