How to programmatically create a Topic in Kafka through spark structured streaming

Question

I want to create multiple kafka Topics run time in my Spark Structured Streaming application. I found that there are various methods available in Java API. But I couldn't find any with Spark Structured Streaming. Please let me know if there is any way available or I need to use java library

My apache Spark version is 2.4.4 and Kafka library dependency is spark-sql-kafka-0-10_2.12

@Ahmed Does the broker need to have auto topic creation enabled? If not, how would Spark create a topic without setting replication or partition settings? — OneCricketeer, Nov 28 '19 at 22:04
With auto creation off, I'd expect an exception. Same without proper credentials when using a secure Kafka cluster. — Jacek Laskowski, Nov 29 '19 at 16:41

OneCricketeer · Answer 1 · 2019-12-31T03:48:41.640

-1

AFAIK, Spark doesn't create topics.

You can use the same Java APIs you've found before initializing your SparkSession

spark-sql-kafka includes kafka-clients, so you have the AdminClient class available

How to create a Topic in Kafka through Java

edited Dec 31 '19 at 03:48

answered Nov 28 '19 at 22:05

OneCricketeer

179,855
19
132
245

It does. And more specifically it's the kafka data source that does the writing part. Sorry. And please don't use `SparkContext` (Spark Core). It's `SparkSession` time (Spark SQL). – Jacek Laskowski Nov 29 '19 at 14:16
@Jacek Can you answer my comment above then? – OneCricketeer Nov 29 '19 at 14:45

How to programmatically create a Topic in Kafka through spark structured streaming

1 Answers1