2

We have multiple consumers that are subscribed to one topic (for now) with 10 partitions and are part of one consumer group.

Since these consumers are constantly listening to messages, we decided to use the worker service template in .NET Core 3.1. And since the idea is to have one consumer per process, each consumer has its own dedicated worker service. They have been running for a couple of weeks in prod, and they seem to be consuming and processing messages as expected.

Was this an overkill approach? And would it be more efficient to deploy one service and have it create 10 consumer instances and start them on separate threads?

For example:

public class ConsumerGroupBuilder : BackgroundService
{
    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
      for(var i = 0; i < 10; i++)
        Task.Run(() => BuildConsumer(stoppingToken));

      return Task.CompletedTask;
    }

   ...
}

Your contribution is greatly appreciated.

asdbabil
  • 91
  • 8
  • `and would it be more efficient to deploy one service and have it create 10 consumer instances and and start them on separate threads?` What happened when you tried it? – mjwills Jul 13 '21 at 23:18
  • @mjwills I tried it while I was still developing the consumer and that worked as well, but I really didn't do any heavy testing. – asdbabil Jul 14 '21 at 14:35
  • 1
    I created multiple instances of consumers on separate threads for similar use case and it worked well. – Bibek Shah Aug 19 '21 at 11:00

0 Answers0