How to produce audio (.wav) to kafka in one message

Question

In order to apply real time speech analytics using big data technologies, I'm trying to use at beginning KAFKA. So at first I convert .wav file to bytes using WAVIO API and then send messages containing [data(type of nparray), rate (integer) and sampwidth (integer)] to kafka after these messages will be consumed by a Consumer who will convert them to .wav file again.

The problem is how can I send and receive these [data, rate, sampwidth] to and from kafka in one message (each message represent .wav file)?

For the Producer:

    producer = KafkaProducer(bootstrap_servers='localhost:9092')
    x = wav2bytes("bush_read") # return tuple containing(data, rate, sampwidth)
    #here I'm sending 3 messages
    producer.send("TestTopic", key=b'data', value=b'%s' % (x[0])) # data -> nparray
    producer.send("TestTopic", key=b'rate', value=b'%d' % (x[1]))   # rate -> int
    producer.send("TestTopic", key=b'sampwidth', value=b'%d' % (x[2]))  #sampwidth -> int
    send("TestTopic","bush_read")

For the consumer:

    for message in consumer:
        msg = message     # I want somthing like this
        file = bytes2wav("name", msg.data, msg.rate, msg.sampwidth )

Which kafka client do you use? – Reznik Nov 02 '19 at 14:44 — Reznik, Nov 02 '19 at 14:44
why can't you write a custom ser/deser? – senseiwu Nov 02 '19 at 23:34 — senseiwu, Nov 02 '19 at 23:34
did you ever find a solution? – frlzjosh Jun 08 '20 at 09:37 — frlzjosh, Jun 08 '20 at 09:37
Yes I find it, its the answer bellow. – John Smith Jun 08 '20 at 10:05 — John Smith, Jun 08 '20 at 10:05

score 0 · Accepted Answer · answered Nov 02 '19 at 14:42

0

You can send it as json (or any other serializion) if you want, create a json like

{'data' : data, 'rate': rate, 'sampwidth': sampwidth}

and you can deserializion it in the consumer

answered Nov 02 '19 at 14:42

Reznik

2,663
1
11
31

1

I don't think you can just put np.array into json – OneCricketeer Nov 02 '19 at 15:27
@cricket_007 you can searlize it into json https://stackoverflow.com/questions/26646362/numpy-array-is-not-json-serializable – Reznik Nov 02 '19 at 15:36

score 0 · Answer 2 · answered Nov 03 '19 at 06:04

Just another thought!!

If .wav file quite large it could put load on broker which could slow down cluster. It can be avoided by publishing reference messages beside full large file.

Store large file in some where external storage
Publish meta reference of file to topic pointing to stored file location
Consumer may consume reference and locate file into external storage.

How to produce audio (.wav) to kafka in one message

For the Producer:

For the consumer:

2 Answers2