2
  1. If I set 10 video taskqueue in temporal matching, if we have 5 matching services, temporal will assign 2 video taskqueue for each matching service?
  2. If I set 10 video taskqueue in temporal matching, if we have 50 workers for them, how are they assigned to which taskqueue to poll? 5 workers poll for each queue? How do we divide which worker poll which video taskqueue? Can anyone explained the principle a little bit?
Sharan Foga
  • 135
  • 8
zheyi yi
  • 129
  • 10

1 Answers1

2

If I set 10 video taskqueue in temporal matching, if we have 5 matching services, temporal will assign 2 video taskqueue for each matching service?

By default, a Temporal Task Queue is configured with 4 partitions. So 10 task queues are going to have 40 partitions total. Temporal uses consistent hashing to place partitions to matching hosts. Note that this algorithm doesn't guarantee exact distribution. But on average each host will end up with 8 partitions.

If I set 10 video taskqueue in temporal matching, if we have 50 workers for them, how are they assigned to which taskqueue to poll? 5 workers poll for each queue? How do we divide which worker poll which video taskqueue? Can anyone explained the principle a little bit?

Temporal doesn't assign workers to task queues. Your code does that. When a worker is created a task queue name is a required parameter. In the majority of cases, you don't need to use multiple task queues. A single queue can support almost any throughput if configured with the appropriate number of partitions.

The reasons for using more than one task queue for the given application:

  • To route requests to separate pools of workers or specific processes
  • To rate limit a certain type of requests
  • To specify per worker limits (rate and number of parallel tasks) for certain type of request
Maxim Fateev
  • 6,458
  • 3
  • 20
  • 35
  • Thanks for your answer. Can I also ask that if 1 taskqueue with 4 partitions. For a worker with the task queue name, how does the work know polling from which partition? Let see we have foo taskqueue with 4 partitions and 10 workers for foo taskqueue. How does these workers are assigned to poll the 4 partitions? – zheyi yi Aug 06 '22 at 17:37
  • Workers are not assigned to partitions. Each poll request is randomly routed to one of the partitions. Each worker usually has multiple polling threads. So the polls are distributed more or less evenly. – Maxim Fateev Aug 06 '22 at 23:54