-3

I'm pretty new to kakfa/spark-streaming. My use case is to send a image through kafka producer and capture it from kafka consumer. I did some research and found out that image must be converted to binary or byte format, but couldn't find any proper sources to continue with it.

What i have done so far:

1.Started zookeeper
2.Started server
3.created a new topic
4.started a producer to send(eg: JSON) - kafka-console-producer.sh --broker-list localhost:9092 --topic send_json < materials.json
5.started a consumer to recieve

How can i send a .jpg or .png file in a similar way? Lang : scala or pyspark

eliasah
  • 39,588
  • 11
  • 124
  • 154

1 Answers1

1

Minimal example (scala):

val props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put("key.serializer", "org.apache.kafka.common.serialization.LongSerializer")
props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer")

val producer = new KafkaProducer(props)
val data = Files.readAllBytes(Paths.get("/path/to/file"))

val record = new ProducerRecord[Long, Array[Byte]]("yourtopic" , data)
producer.send(record)
producer.close

If your images get's bigger keep this one in mind How can I send large messages with Kafka (over 15MB)?

If you want to use the kafka-console-producer:

kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic < my_file.txt
TobiSH
  • 2,833
  • 3
  • 23
  • 33
  • Should I implement this in producer console –  Apr 08 '19 at 12:24
  • Sorry for the confusion. You can see two independent examples. Use the scala snippet in your application or put it into the main function of an object to run it standalone. The console consumer can be run as it is. – TobiSH Apr 08 '19 at 12:48
  • Sorry @Anthony what exactly are you looking for? A command-line tool? Or a scala code snippet? Both are present. Pick whatever you want – TobiSH Apr 08 '19 at 13:00
  • There's no LongSerializer under org.apache.kafka.common.serialization? –  Apr 09 '19 at 12:33
  • Yes only stringSerializer and stringdeserializer , serializer and deserilazer, bytearrayserializer and bytearraydeserializer. –  Apr 09 '19 at 16:38
  • It is part of the kafka-client lib – TobiSH Apr 10 '19 at 07:27
  • https://imgur.com/a/H0megxj –  Apr 10 '19 at 10:08
  • How to conert that byteArray back to image? –  Apr 10 '19 at 12:17
  • How does serialisation helps in this use case? –  Apr 10 '19 at 16:39