1

I try to follow the below guide from GCP to create a data quality task. https://cloud.google.com/dataplex/docs/check-data-quality?&_ga=2.139058355.-143783534.1669209779#before_you_begin when I try to run the task I get the error

Error

  • As per my understanding Dataplex is calling dataproc internally for running the spark jobs due to which we get the error message. But we dont have the option to adjust the number of CPUs that can be used.
  • I tried to increase the CPU quota but GCP does not allow me( My account is enabled for billing for a long time) Any suggestion to correct this error will be appreciated.
user546298
  • 41
  • 4
  • 1
    Is it a personal or a business account (bind to your company organisation)? – guillaume blaquiere Apr 03 '23 at 08:06
  • its a personal account. But like I mention I have enabled for billing. – user546298 Apr 03 '23 at 09:26
  • What do you mean when you say `we don't have option to adjust CPUs`? Are you facing any issue [changing](https://cloud.google.com/dataproc/docs/concepts/compute/secondary-vms#using_secondary_workers) the no. of workers through Dataproc console/CLI? – Sakshi Gatyan Apr 03 '23 at 10:40
  • The quota is similar in concept to a "credit limit" on a credit card. For personal accounts, sometimes you can get the quota increased but not always. – John Hanley Apr 03 '23 at 21:41

2 Answers2

1

It won't help you, but even if you have activated the billing, as personal user you have a very limited number of allowed CPU (here my CPU quotas for all region; and I'm a GDE! Check it in your project.) enter image description here

And here the quotas with my corporate sandbox project enter image description here

If you have the opportunity to test it in a corporate account, or if you have a Google Cloud contact, you could ask for a quota increase.

guillaume blaquiere
  • 66,369
  • 2
  • 47
  • 76
  • Hi Guillaume, I checked this already and mine is 8. And as you said it is not allowing me to increase the quota. – user546298 Apr 04 '23 at 07:11
1

As mentioned by @guillaume, there is set a limit on the number of CPUs you can use for a region (and all regions) in GCP. It is different for individuals and organizations. You can check this under IAM -> Quotas-> Compute Engine API CPU quota.

Solutions to your error:

  • You can request a quota increase on the number of CPUs for a region/all regions for your project.

  • Try changing the configuration of your Dataproc Cluster by decreasing the number of secondary workers. Reference

Sakshi Gatyan
  • 1,903
  • 7
  • 13