1

In my application, there are multiple enterprises. each enterprise login and do some action like upload the data, then Kafka producer takes the data and sends to the topic. Another side Kafka consumer consumes data from the topic and performs business logic. and persists into the database. In this case, everything is perfect when a single enterprise login. but when multiple enterprise logins then Kafka consuming in sequentially. i.e., how can I make the process parallel? on multiple client requests. thanks in advance.

kumar
  • 71
  • 1
  • 12
  • 1
    You need to [partition the topic](https://stackoverflow.com/questions/38024514) according to the number of consumers, and then each instance of your consumer will be assigned one or more partitions, and they will consume independently in parallel. If necessary, you can use the partition key to ensure correlated messages arrive at the same consumer. – StuartLC Feb 12 '19 at 13:58

2 Answers2

1

As mentioned in previous Answers you can use multiple partitions . Another option is you get advantage of threading(Threadpoolexecutor) so follow will be like : receive message -> create parallel thread to do the required logic --> ack message . Please ensure you have throttling (using thread pool executors) application perforamance .

ahmed metwaly
  • 31
  • 3
  • Actually what happens when an enterprise login and uploads data is spring app get the uploaded data and Kafka producer to send data to the topic and another side consumer takes the data and process it. consider the time taken to process data through Kafka producer, Kafka consumer and persist into the database is 3min.mean while another enterprise comes into the picture before completing the first enterprise. second enterprise request is waiting until the first request is completed.here every enterprise request acting on the same topic having multiple partitions. – kumar Feb 13 '19 at 10:42
  • You can create multiple partitions in that topic and use enterprise Id as partitioning key when you write to Kafka so consumers can run in parallel and receive messages in order. A consumer can cater to one or more partitions but to gain max parallelism you can run one consumer per partition.What kind of a database are you using to store this data ?.When you write your own consumer there are few things you would need to handle like re balances,some offfset management if you have stricter delivery semantics . You should look at a kafka connect option if available which has inbuilt – rookie Feb 14 '19 at 20:49
0

If that topic only has one partition, it's sequential on the consumer side. Multiple producers for one partition has no guarantees on ordering.

Consumers and producers will batch messages and process them in chunks.

Another side Kafka consumer consumes data from the topic and performs business logic. and persists into the database.

I suggest not using a regular consumer for this. Please research Kafka Connect and see if your database is supported

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245