How to run springboot without kafka server

Question

I have a Spring boot project that has dependencies that use kafka for logging. I can't get rid of said dependency because my logic requires code from that dependency

I want to start said application locally without running kafka so I can test my simple db lookup code.

I've tried disabling autoconfiguration

spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration

I've tried setting missingTopicsFatal to false via bean configuration

@Bean(name = "kafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
    ConsumerFactory<Object, Object> kafkaConsumerFactor,
    ConcurrentKafkaListenerContainerFactoryConfigurer configurer) {

  ConcurrentKafkaListenerContainerFactory<Object, Object> factory =
      new ConcurrentKafkaListenerContainerFactory<>();
  configurer.configure(factory, kafkaConsumerFactor);

  ContainerProperties containerProperties = factory.getContainerProperties();
  containerProperties.setMissingTopicsFatal(false);

  ...
  return factory;
}

But evidently the setting is already false as my application runs without failing, just that tomcat doesn't want to open the listen port

All other solutions I've encountered involves changing the code where kafka is used; in this case, the dependency I am using. But I have no access to changing said dependency.

The application starts but spring refuses to open a listen port and keeps looping with the error

Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

The application runs fine with kafka running locally except now the mere act of typing is slow thanks to kafka's resource usage. Please help.

score 0 · Answer 1 · answered Apr 15 '20 at 11:40

Seems to me your main problem is the resource usage of your local kafka installation. Maybe first take a look at its configuration and data. Maybe start from fresh and purge all local data?

Alternatively, you could run KafkaServer in-process, something like this in your main class:

    Properties kafkaProperties = new Properties();
    kafkaProperties.setProperty("broker.id", "1");
    kafkaProperties.setProperty("host.name", "localhost");
    kafkaProperties.setProperty("listeners", "PLAINTEXT://localhost:" + kafkaPort);
    kafkaProperties.setProperty("log.dir", Files.createTempDir().getAbsolutePath());
    KafkaConfig kafkaBrokerConfig = new KafkaConfig(kafkaBrokerConfig);
    KafkaServer kafkaServer = new KafkaServer(kafkaBrokerConfig, Time.SYSTEM);
    kafkaServer.startup();

Or you could run Kafka from Docker. See e.g. the docker-compose file https://github.com/confluentinc/examples/tree/5.4.1-post/cp-all-in-one-community.

You'll probably only need the zookeeper and broker:

docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:5.4.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

Yes I'm aware I could spin up a server elsewhere to run kafka (missing the point of testing locally), or even run it as part of the application, but the point is kafka is completely not required so it shouldn't even run. I'm trying to figure out a way to have the application continue initialization without kafka. It even gets past flyway migration fine and instantiates all the controllers. Only doesn't start listening on tomcat. If it can be done for databases why not kafka: https://stackoverflow.com/questions/36387265/disable-all-database-related-auto-configuration-in-spring-boot — Isaac N, Apr 15 '20 at 14:20

score 0 · Answer 2 · answered Apr 16 '20 at 08:22

0

You can disable the whole kafkaListenerContainerFactory bean via a Spring profile:

@Profile("!nokafka")
@Bean(name = "kafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(){...}

And run your application with spring profile 'nokafka'.

E.g. using the latest spring-boot maven plugin:

mvn spring-boot:run -Dspring-boot.run.profiles=nokafka

Or via java system properties:

-Dspring.profiles.active=nokafka

answered Apr 16 '20 at 08:22

GeertPt

16,398
2
37
61

Deleting the whole bean doesn't even disable it. I added that in in hopes to override the bean from the dependency, but I have no clue what config properties to change. Disabling the bean just lets spring revert to whatever containerFactory the dependency has by default. I'd add that profile annotation to wherever kafka is actually configured but I have no access to said code – Isaac N Apr 16 '20 at 16:02

How to run springboot without kafka server

2 Answers2