I am new in GCP platform and trying to create a simple Data Fusion workflow to load a BigQuery table from a text file that resides in GCS bucket.
The workflow has been deployed successfully. However, while running the workflow, it is failing in step 2 with an error statement
PROVISION task failed in REQUESTING_CREATE state for program run program_run:default.testing_df.-SNAPSHOT.workflow.DataPipelineWorkflow.5ff986e9-7241-11e9-af46-56bbe5c7844b.
In detailed log below, we have "Required 'compute.firewalls.list' permission for 'projects/mylab-gcp" statement
Below is complete error log for the execution.
2019-05-09 10:01:13,945 - DEBUG [provisioning-service-7:i.c.c.i.p.t.ProvisioningTask@121] - Executing PROVISION subtask REQUESTING_CREATE for program run program_run:default.testing_df.-SNAPSHOT.workflow.DataPipelineWorkflow.5ff986e9-7241-11e9-af46-56bbe5c7844b. 2019-05-09 10:01:16,839 - INFO [provisioning-service-7:i.c.c.r.s.p.d.DataprocProvisioner@171] - Creating Dataproc cluster cdap-testingdf-5ff986e9-7241-11e9-af46-56bbe5c7844b with system labels {goog-datafusion-version=6_0, cdap-version=6_0_0-1555624816640, goog-datafusion-edition=basic} 2019-05-09 10:01:17,162 - ERROR [provisioning-service-7:i.c.c.i.p.t.ProvisioningTask@151] - PROVISION task failed in REQUESTING_CREATE state for program run program_run:default.testing_df.-SNAPSHOT.workflow.DataPipelineWorkflow.5ff986e9-7241-11e9-af46-56bbe5c7844b. com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Required 'compute.firewalls.list' permission for 'projects/mylab-gcp'", "reason" : "forbidden" } ], "message" : "Required 'compute.firewalls.list' permission for 'projects/mylab-gcp'" } at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146) ~[na:na] at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) ~[na:na] at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) ~[na:na] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321) ~[na:na] at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065) ~[na:na] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[na:na] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[na:na] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[na:na] at io.cdap.cdap.runtime.spi.provisioner.dataproc.DataprocClient.getFirewallTargetTags(DataprocClient.java:310) ~[na:na] at io.cdap.cdap.runtime.spi.provisioner.dataproc.DataprocClient.createCluster(DataprocClient.java:155) ~[na:na] at io.cdap.cdap.runtime.spi.provisioner.dataproc.DataprocProvisioner.createCluster(DataprocProvisioner.java:172) ~[na:na] at io.cdap.cdap.internal.provision.task.ClusterCreateSubtask.execute(ClusterCreateSubtask.java:43) ~[na:na] at io.cdap.cdap.internal.provision.task.ProvisioningSubtask.execute(ProvisioningSubtask.java:54) ~[na:na] at io.cdap.cdap.internal.provision.task.ProvisioningTask.lambda$executeOnce$0(ProvisioningTask.java:123) ~[na:na] at io.cdap.cdap.common.service.Retries.callWithRetries(Retries.java:183) ~[na:na] at io.cdap.cdap.common.service.Retries.callWithInterruptibleRetries(Retries.java:257) ~[na:na] at io.cdap.cdap.internal.provision.task.ProvisioningTask.executeOnce(ProvisioningTask.java:123) ~[na:na] at io.cdap.cdap.internal.provision.ProvisioningService.lambda$null$16(ProvisioningService.java:559) [na:na] at io.cdap.cdap.internal.provision.ProvisioningService.callWithProgramLogging(ProvisioningService.java:772) [na:na] at io.cdap.cdap.internal.provision.ProvisioningService.lambda$null$17(ProvisioningService.java:557) [na:na] at io.cdap.cdap.common.async.KeyedExecutor$2.run(KeyedExecutor.java:98) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_212] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212] 2019-05-09 10:01:17,180 - DEBUG [provisioning-service-7:i.c.c.i.p.t.ProvisioningTask@159] - Terminated PROVISION task for program run program_run:default.testing_df.-SNAPSHOT.workflow.DataPipelineWorkflow.5ff986e9-7241-11e9-af46-56bbe5c7844b due to exception.
I can see my root GCP console user along with a Service Account "mylab-bigquery@mylab-gcp.iam.gserviceaccount.com" in IAM console. I have assigned both root account and service account as Owner along with below additional roles in both account.
BigQuery Admin Compute Instance Admin (v1) Compute Network Admin Compute OS Admin Login Compute Security Admin Cloud Data Fusion Admin Cloud Data Fusion API Service Agent Owner
However, after doing all these also, still I am getting the same error message "Required 'compute.firewalls.list' permission for 'projects/mylab-gcp"
