1

I have a Flask API, which when hit by a client, will subscribe to a Kafka topic and start consuming (polling) data from the topic and write it to a database.

Currently the consumer function runs an infinite loop to poll the data and write it to the database. But the problem here is, the client doesn't receive a success response here. The further client processes happens after the server timeout.

But the objective is, client has to receive the response right after the consumer is initialized and is ready to consume.

How to implement it in a proper way?

Edit:

I came across this thread - Flask end response and continue processing

Seems that using Python's Thread Library won't work here according to the comments on the top voted answer.

And the Accepted Answer suggested to use WSGI middleware that adds a hook to the close method of the response iterator. But is it advisable in this case where the consumer will be running for an infinite amount of time?

Is there an alternative to these?

Gokul Raam
  • 31
  • 6
  • 1
    Kafka Consumer blocks thread, so you should run it in another thread (or proccess) – sudden_appearance Jun 07 '22 at 20:09
  • You should ideally use Kafka Connect to write to the database rather than do that all from Python and your web server. – OneCricketeer Jun 07 '22 at 20:15
  • @sudden_appearance I have edited my answer based on another solution that I read about using threads. Can you please suggest the right way to run it on another thread? – Gokul Raam Jun 07 '22 at 20:41
  • Why don't you want an infinite consumer? Why can't you use Kafka Connect? – OneCricketeer Jun 08 '22 at 14:47
  • 1
    Any of the service bus or queuing or message queues allows you to decouple producer from consumer. Since i dont know much of your usecase i.e what is the consumer doing.. Still..one of the approach could be utilized is have two topics (input/output). Flow: Service(Fast/Flask) publish message(producer) to input topic and the response is job-id. Consumer service consume the topic writes to database and publishes to output topic. The calling program can query the job-id for result in the output topic. – teedak8s Jun 18 '22 at 23:35

0 Answers0