60

Using Kafka as a messaging system in a microservice architecture what are the benefits of using spring-kafka vs. spring-cloud-stream + spring-cloud-starter-stream-kafka ?

The spring cloud stream framework supports more messaging systems and has therefore a more modular design. But what about the functionality ? Is there a gap between the functionality of spring-kafka and spring-cloud-stream + spring-cloud-starter-stream-kafka ? Which API is better designed?

Looking forward to read about your opinions

Eike Behrends
  • 855
  • 2
  • 7
  • 11
  • 1
    Maybe I haven't made myself clear, but wanting to know about a gap in terms of functionality is something you can't easily extract reading the docs. The foundation and api of the libraries / frameworks may differ but are they offering the same functionality? For example : is the spring-kafka API/functionality richer when using only kafka? – Eike Behrends Nov 15 '17 at 17:40
  • 1
    Well, each of them are a bit for different purposes. To hard to answer shortly. See their project pages for more info: https://projects.spring.io/spring-integration/, https://projects.spring.io/spring-kafka/, https://cloud.spring.io/spring-cloud-stream/ – Artem Bilan Nov 15 '17 at 17:43
  • I am aware of the advantages using the concept of binders but I am simply asking myself if there's a tradeoff, since it's build on top of spring-kafka and using it's own API. Every upcoming and new functionality in spring-kafka has somehow be "mapped" in the concept of spring cloud stream. Furthermore the API has to support multiple binder like rabbitmq and that's why the API has to be more abstract / generic. Please correct me if I am wrong but that's why I am asking myself if there is (or always will be) a gap in terms of functionality and if it is better to simply use spring-kafka – Eike Behrends Nov 15 '17 at 22:13
  • 5
    Well, that’s true. Since Binder API should be as generic as possible for any Binder implementation, there is definitely something missed from the target protocol specifics. If you really need something not implemented in the Kafka Binder, stay with just Spring Kafka and if you need more control over Consumer poll, go down to just raw Apache Kafka. In most streaming scenarios the auto-configuration from the Spring Cloud Stream is really enough – Artem Bilan Nov 15 '17 at 22:33
  • Thanks for your time and comments! – Eike Behrends Nov 16 '17 at 10:10

2 Answers2

26

Spring Cloud Stream with kafka binder rely on Spring-kafka. So the former has all functionalities supported by later, but the former will be more heavyweight. Below are some points help you make the choice:

  1. If you might change kafka into another message middleware in the future, then Spring Cloud stream should be your choice since it hides implementation details of kafka.
  2. If you want to integrate other message middle with kafka, then you should go for Spring Cloud stream, since its selling point is to make such integration easy.
  3. If you want to enjoy the simplicity and not accept performance overhead, then choose spring-kafka
  4. If you plan to migrate to public cloud service such as AWS Kensis, Azure EventHub, then use spring cloud stream which is part of spring cloud family.
Warren Zhu
  • 1,355
  • 11
  • 12
  • 1
    Can you elaborate more on #3 - "If you want to enjoy the simplicity and not accept performance overhead, then choose spring-kafka". I am looking for performance overhead with Spring cloud stream ? – R K Dec 15 '18 at 19:25
  • 1
    The performance overhead is caused by one extra layer of abstraction of receiving, sending and converting message formats. Since spring cloud stream is based on channel abstraction, there may be some channels could buffer and cache message. But the real overhead should be measured in your use case – Warren Zhu Dec 17 '18 at 02:12
  • I want to know the compatibility with spring batch with spring kafka vs spring cloud stream kafa. I have found way to use spring batch with spring kafka but not a way to spring batch stream kafka – Sajith Vijesekara Apr 13 '20 at 18:26
  • I will encourage you to go with Spring Cloud Stream 3.0 and above - all key points are validly summarized by @WarrenZhu above and should help. Personally, I really like Spring Cloud Stream as it decouples your dependency from underlying messaging platform and with mix of spring cloud functions, you can pretty much have your producers / consumers built out of config - so you can simply focus on writing business logic. Its bit evolving framework I guess, but has lots of goodies. – Deepak Chaudhary Aug 13 '20 at 14:55
16

Use Spring Cloud Stream when you are creating a system where one channel is used for input does some processing and sends it to one output channel. In other words it is more of an RPC system to replace say RESTful API calls.

If you plan to do an event sourcing system, use Spring-Kafka where you can publish and subscribe to the same stream. This is something that Spring Cloud Stream does not allow you do do easily as it disallows the following

public interface EventStream {
    String STREAM = "event_stream";

    @Output(EventStream.STREAM)
    MessageChannel publisher();

    @Input(EventStream.STREAM)
    SubscribableChannel stream();
}

A few things that Spring Cloud Stream helps you avoid doing are:

  • setting up the serializers and deserializers
Archimedes Trajano
  • 35,625
  • 19
  • 175
  • 265
  • 1
    What is your opinion about RPC systems vs event sourcing based, could you collaborate a bit please? Spring Cloud Stream is next level of abstraction for messaging, but the idea and usage is the same. – Oleksandr Yefymov Dec 22 '19 at 20:05
  • I'd like to know more about this as I mainly would use kafka to implement pub/sub event system. – Anddo Aug 10 '20 at 17:30