Frequent RESOURCE_EXHAUSTED errors

Question

I'm getting into the Google Cloud Deployment Manager lately but struggling with some strange errors. Maybe someone here will have some idea as to the cause.

Typically I will "create" a deployment in preview mode, then once I see it in the GUI I will click the "Deploy" button and have it do it's thing. But when it finishes I am greeted with the following error message:

{
    "ResourceType": "deploymentmanager.v2.virtual.enableService",
    "ResourceErrorCode": "429",
    "ResourceErrorMessage": {
        "code": 429,
        "message": "Insufficient tokens for quota 'DefaultGroup' and limit 'CLIENT_PROJECT-100s' of service 'servicemanagement.googleapis.com' for consumer 'project_number:794362051698'.",
        "status": "RESOURCE_EXHAUSTED",
        "details": [{
            "@type": "type.googleapis.com/google.rpc.Help",
            "links": [{
                "description": "Google developer console API key",
                "url": "https://console.developers.google.com/project/794362051698/apiui/credential"
            }]
        }],
        "statusMessage": "Too Many Requests",
        "requestPath": "https://servicemanagement.googleapis.com/v1/operations/projectSettings.d51a6a47-7a15-42bb-90a7-70c3dd7c25e3"
    }
}

One might think this is due to a quota being exceeded, but after checking the quotas in IAM->Quotas I can't find anything over quota.

There are now 3 deployments piled up in my Deployment Manager in a similar sorry state. The weird thing is that my project was created and when I look at the enabled APIs for that project I find everything that I specified in my DM configuration.

Edit: The problem seems to go away when I remove some of the APIs which I enable (using the apis: block in my resource: block). I'm trying to enable 23 but the problem goes away when I cut that number in half.

score 2 · Answer 1 · answered Jun 27 '17 at 00:57

2

This issue is discussed in https://github.com/GoogleCloudPlatform/google-cloud-node/issues/1756 . The upshot is:

I was giving each subscription a unique subscription name (uuid) to ensure messages arrive to a single subscriber at a time, so one way to reduce usage is to avoid that and instead ignore irrelevant messages. But the main thing that solved it was increasing the quota in the developer console if I remember correctly the default quota is 1K, I took it up to the maximum (10K); it wasn't clear you could actually do that from the console. Finally, I'm now using a different messaging queue for few reasons one of which I I thought 10K was too little - that did it ;)

answered Jun 27 '17 at 00:57

mikep

3,841
8
21

It's unclear what a messaging queue has to do with the problem. It is also unclear what quota to bump up from 1000 to 10000. I did in fact bump the quota for "Administrative Operations" in the "Pub/Sub Quotas" section. Still no luck. Having read through the link, I'm not at all sure how I can "generate random subscription ids to ensure messages are received by all subscribers" in the context of Deployment Manager. – Randy L Jun 27 '17 at 15:05
And I've grepped the entire contents of the [Deployment Manager Examples](https://github.com/GoogleCloudPlatform/deploymentmanager-samples) and found no instances of "uuid" anywhere. – Randy L Jun 27 '17 at 15:34

score 2 · Answer 2 · answered Jun 29 '17 at 13:29

You are likely running into the quotas imposed by Deployment Manager - https://cloud.google.com/deployment-manager/pricing-and-quotas.

Deployment Manager

Deployment Manager enforces the following API limits and quotas:

Unlimited read requests, which include any requests made through gcloud or Google Cloud Platform Console. Requests such as get and list would be considered read requests.

Unlimited delete requests to delete deployments.

1,000 API write requests per day, including requests made through gcloud. This includes requests to change or create deployments and resources, such as insert and update.

20 API requests per second.

1,000 deployments per project.

1 MB limit on user-provided configurations, including the YAML configuration and any imports.

1 MB limit on expanded configurations. When fully-expanded, your configuration cannot exceed this size.

I had initially marked this as the correct answer but I'm not too convinced and even if I were, there's no obvious solution for the problem. — Randy L, Jul 20 '17 at 20:19

Frequent RESOURCE_EXHAUSTED errors

2 Answers2

Deployment Manager