-2

I want to push multiple json files (around 200k files) to kafka topic for testing purpose. I want to push each file as a message to kafka topic. What is the approach ?

sparker
  • 1,666
  • 4
  • 21
  • 35
  • [How to write a file to Kafka Producer](https://stackoverflow.com/questions/33273587/how-to-write-a-file-to-kafka-producer) – Erwin Bolwidt Apr 03 '18 at 02:26
  • @ErwinBolwidt The thread you mentioned is to use command line and that too for single file. I want it for pushing around 200k files from a directory programatically – sparker Apr 03 '18 at 05:17
  • 1
    Where did you write requirement that in your question? Right now it says "I want to push whole file as **a** message". Also, please show the code that you have written as a [mcve] and indicate where the problem is. – Erwin Bolwidt Apr 03 '18 at 05:49
  • Sorry @ErwinBolwidt its my mistake. Edited the question now. – sparker Apr 03 '18 at 06:31
  • Ideally, you push JSON objects from the file as individual messages. Don't attempt to push whole files – OneCricketeer Apr 03 '18 at 13:24

1 Answers1

1

Use Kafka Connect with the kafka-connect-spooldir connector. You can see it in action here.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92