0

Is there any way that i can use kafka stream api with predefined users topic but able to call functions like through() & to(), per my understanding these functions will generate intermediate topics for caching purposes.

Henry Leung
  • 1
  • 1
  • 1
    Check out the docs: http://kafka.apache.org/24/documentation/streams/developer-guide/manage-topics.html -- topic uses in `through()` and `to()` should be pre-created. – Matthias J. Sax Jan 01 '20 at 19:01

1 Answers1

1

You require a pre-defined topic as your input source.

through() is the only action you listed that makes an "intermediate" topic.
to() makes a terminal topic.

If you don't do stateful operations or repartitioning (by altering the message key), there are no intermediate topics.

More info - What are internal topics used in Kafka?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks for the reply, I have read these articles earlier and the only thing i can't figure out is how do we create some of the topics ein advance. – Henry Leung Jan 02 '20 at 02:24
  • Have you never created a topic? https://kafka.apache.org/documentation/#quickstart_createtopic Or https://stackoverflow.com/a/53851426/2308683 – OneCricketeer Jan 02 '20 at 04:59
  • I am referring to internal topics – Henry Leung Jan 03 '20 at 08:42
  • Internal topics are created when the broker starts. And they're made using the above methods. I don't understand. There is no internal users topic – OneCricketeer Jan 03 '20 at 08:51
  • i would like to perform joining and aggregations via stream apis with "users topics" alone (with no privilege to be granted for creating internal topics), however i realised the followings.It'll be helpful if you have any other better ways to bypass generation of internal topics. Changelog topics are created when there are join/aggregation operations on the stream. Actually the result of aggregation call creates a state store and for fault-tolerance the state store is backed up by a Kafka Changelog topic. The aggregation results are stored into this internal topic. – Henry Leung Jan 04 '20 at 15:43
  • Again, there is no "users topic" that exists, so I'm not sure I understand what you mean – OneCricketeer Jan 04 '20 at 16:21
  • you can refer to this site regarding definitions of user topics https://kafka.apache.org/10/documentation/streams/developer-guide/manage-topics.html – Henry Leung Jan 04 '20 at 16:28
  • Okay, that just means a topic created by a user, following the above methods I already mentioned. – OneCricketeer Jan 04 '20 at 16:42