4

I am migrating from Spring XD to Spring Cloud Data Flow. When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow - One of them is KAFKA source.

My question is why KAFKA source is removed from standard sources list in spring cloud data flow ?

Viren Balaut
  • 55
  • 2
  • 6

1 Answers1

11

When I am looking for module list I realised that some of the sources are not listed in Spring Cloud Flow

Majority of the applications are ported over and the remaining are incrementally prioritized - you can keep track of the remaining subset in the backlog.

My question is why KAFKA source is removed from standard sources list in spring cloud data flow ?

Kafka is not removed and in fact, we are highly opinionated about Kafka in the context of streaming use-cases so much so that it is baked into the DSL directly. More details here.

For instance,

(i) if you've to consume from a Kafka topic (as a source), your stream definition would be:

stream create --definition ":someAwesomeTopic > log" --name subscribe_to_broker --deploy

(ii) if you've to write to a Kafka topic (as a sink), your stream definition would be:

stream create --definition "http --server.port=9001 > :someAwesomeTopic" --name publish_to_broker --deploy

(where *someAwesomeTopic* is the named destination, a topic name)

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
  • any working example of how to do this?, I wrote a very simple console app that uses a kafka producer to send the string "testme" and connect a stream to sand that to a file, and it is failing trying to convert the message: o.s.c.s.b.k.KafkaMessageChannelBinder : Could not convert message: 746573746D65 . This number is the ascii representation of the message "testme" – ka2 Feb 09 '17 at 17:03
  • I assume this DSL shortcut uses the same Kafka cluster that Data Flow Server itself uses as a broker for a stream. This is not exactly the same as consuming a message from some Kafka cluster. This is possible with RabbitMQ that has both source and sink streaming apps. But Kafka ones are surprisingly missing. – eduard.dudar Aug 19 '21 at 01:04