2

I am getting {"code": "Too many requests", "message": "Request is denied due to throttling."} from ADX when I run some batch ADF pipelines. I have came across this document on workload group. I have a cluster where we did not configured work load groups. Now i assume all the queries will be managed by default workload group. I found that MaxConcurrentRequests property is 20. I have following doubts.

  1. Does it mean that this is the maximum concurrent requests my cluster can handle?

  2. If I create a rest API which provides data from ADX will it support only 20 requests at a given time?

  3. How to find the maximum concurrent requests an ADX cluster can handle?

Justin Mathew
  • 950
  • 8
  • 34
  • could you please include the full error message? it should include the origin of the throttling (i.e. which policy has this limit defined in it) – Yoni L. Jun 14 '21 at 14:17
  • 'TableSetOrAppend', Capacity: 6, Origin: 'CapacityPolicy/Ingestion'.", "@context": { "timestamp": "2021-06-14T07:55:41.3803061Z", "serviceAlias": "XXXXXX", "machineName": "KEngine000000", "processName": "Kusto.WinSvc.Svc", "processId": 9544, "threadId": 7080, "appDomainName": "Kusto.WinSvc.Svc.exe", "clientRequestId": "ADF.ControlCommand;", "activityId": " ", "subActivityId": " – Justin Mathew Jun 14 '21 at 14:26

1 Answers1

4

for understanding the reason your command is throttled, the key element in the error message is this: Capacity: 6, Origin: 'CapacityPolicy/Ingestion'.

this means - the number of concurrent ingestion operations your cluster can run is 6. this is calculated based on the cluster's ingestion capacity, which is part of the cluster's capacity policy.

it is impacted by the total number of cores/nodes the cluster has. Generally, you could:

  • scale up/out in order to reach greater capacity, and/or
  • reduce the parallelism of your ingestion commands, so that only up to 6 are being run concurrently, and/or
  • add logic to the client application to retry on such throttling errors, after some backoff.

additional reference: Control commands throttling

Yoni L.
  • 22,627
  • 2
  • 29
  • 48
  • Where can I find the: "Number of nodes in cluster" and "Core count per node"? – Ace Aug 12 '21 at 08:45
  • you can use `.show cluster` for that: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/diagnostics#show-cluster – Yoni L. Aug 12 '21 at 17:43
  • Thank you. I increased it to 12 by scaling out and there are still several errors (much less than before) - Is there a way to predict how much do I have to scale or is it a trial and error? – Ace Aug 14 '21 at 19:51