4

I use Apache Beam vs Kafka Clients to process my data but the library Kafka Clients produces a lot of message so they do really mess in stack trace, how can I change log level for this package?

Versions of dependecies: slf4jVersion=1.7.29, apacheBeamVersion=2.18.0

I have added a file log4j.properties to src/main/resources/ with content below but it does not seem to be working

log4j.rootLogger=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
log4j.logger.org.apache.kafka.clients.consumer.internals=ERROR

build.gradle

dependencies {
    implementation "org.apache.beam:beam-sdks-java-core:${apacheBeamVersion}"
    implementation "org.apache.beam:beam-runners-google-cloud-dataflow-java:${apacheBeamVersion}"
    implementation "org.apache.beam:beam-sdks-java-io-google-cloud-platform:${apacheBeamVersion}"
    implementation "org.apache.beam:beam-sdks-java-io-kafka:${apacheBeamVersion}"
    runtimeOnly "org.slf4j:slf4j-jdk14:${slf4jVersion}"
    ...
}

Log message I want to avoid in my logs:

INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Seeking to LATEST offset of partition custom-group-0
Mar 16, 2020 1:55:34 PM org.apache.kafka.clients.consumer.internals.SubscriptionState maybeSeekUnvalidated
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Resetting offset for partition custom-group-0 to offset 9347.
Mar 16, 2020 1:55:35 PM org.apache.kafka.clients.consumer.internals.SubscriptionState lambda$requestOffsetReset$3
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Seeking to LATEST offset of partition custom-group-0
Mar 16, 2020 1:55:35 PM org.apache.kafka.clients.consumer.internals.SubscriptionState maybeSeekUnvalidated
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Resetting offset for partition custom-group-0 to offset 9347.
Mar 16, 2020 1:55:36 PM org.apache.kafka.clients.consumer.internals.SubscriptionState lambda$requestOffsetReset$3
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Seeking to LATEST offset of partition custom-group-0
Mar 16, 2020 1:55:36 PM org.apache.kafka.clients.consumer.internals.SubscriptionState maybeSeekUnvalidated
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Resetting offset for partition custom-group-0 to offset 9347.
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
boden
  • 1,621
  • 1
  • 21
  • 42
  • At the very least, match the log format of the other messages – OneCricketeer Mar 16 '20 at 13:07
  • @cricket_007 it seems that pipeline's logs and Kafka's logs are different – boden Mar 16 '20 at 14:31
  • True, but that doesn't mean you can't change the formats (neither of which match the file you've given). I'd also suggest using `logback.xml` or `log4j2.xml` since log4j 1.x has documented issues – OneCricketeer Mar 16 '20 at 16:40
  • @cricket_007 actually my attempt wasn't successful, none logback and log4j2 changed were able to change log level – boden Mar 29 '20 at 20:54
  • kafka-clients depends on log4j, you would have to exclude it from the dependencies. And like I said, the logs you see clearly are not using the file you've shown, based on the printed timestamp. – OneCricketeer Mar 29 '20 at 23:39

1 Answers1

0

You should not use slf4j-jdk14 as it has java.util.logging as the implementation and won't take log4j2 configuration into account. So remove this

runtimeOnly "org.slf4j:slf4j-jdk14:${slf4jVersion}"

In order to use log4j2 config, you will need a log4j2 implementation. I used

implementation "org.apache.logging.log4j:log4j-slf4j-impl:2.13.3"

If you want to use yaml config like me, just add these two dependencies

implementation "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.5"
implementation "com.fasterxml.jackson.core:jackson-databind:2.10.5"

In my case I had another slf4j implementation in one of my transitive dependencies that I had to exclude to make it work. Just so you know.

namp10010
  • 123
  • 1
  • 10