I am using the AWS Glue connector for BigQuery. My glue jobs were running fine in multiple AWS accounts but suddenly it started failing with the below response in all the accounts together:
Access Denied: Table common-infra-services:detailedcost.gcp_billing_export_resource_v1_01E8AD_3E792E_BB0E5D: Permission bigquery.tables.get denied on table common-infra-services:detailedcost.gcp_billing_export_resource_v1_01E8AD_3E792E_BB0E5D (or it may not exist).", "reason": "accessDenied"
Please review and let me know what could be the issue of this problem.
I am using the GCP IAM service account role to run queries using Glue to BigQuery with the following set of permissions:
bigquery.jobs.create
bigquery.tables.getData
bigquery.tables.list
And with these permissions, all jobs were running fine till yesterday.