1

I'm trying to use the AWS Glue connector to BigQuery following the tutorial in https://aws.amazon.com/blogs/big-data/migrating-data-from-google-bigquery-to-amazon-s3-using-aws-glue-custom-connectors/ but after following all steps I get a:

: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment.  Please set a project ID using the builder.

The Python exception shows that:

Traceback (most recent call last):
  File "/tmp/ETHBlockchainExport.py", line 20, in <module>
    DataSource0 = glueContext.create_dynamic_frame.from_options(connection_type = "marketplace.spark", connection_options = 
{
    "parentProject": "MYGOOGLE_PROJECT_ID",
    "connectionName": "BigQuery",
    "table": "MYPROJECT.DATASET.TABLE"
}

So everything seems provided but still complains about Project Id. How can I provide that info to the connector?

tonicebrian
  • 4,715
  • 5
  • 41
  • 65

1 Answers1

1

You specify in the data source connector option when you create the Glue job, as key-value pair.

from your log it seems that you included the project id in the table field as well, should be dataset.table

Also another possibility is that you didn't specify the values of the project ID and table values etc in environment variable (this seems to be more likely based on the error shown)

Example enter image description here

Reference

unacorn
  • 827
  • 10
  • 27
  • i am currently have the same problem. i already add the connection options like above, but still get the same error. btw i am running it from aws GlueStudio using Job, do i still need to set the environment variable in my case? – victorxu2 Sep 14 '21 at 08:58
  • 1
    @victorxu2 what i did via gluestudio and it works is just keying in the value, like what is shown in the image. Do you have a cloudwatch log that display the error? – unacorn Sep 14 '21 at 10:00
  • thankyou for replying, i got the problem solved now. i think i got the wrong google credential key in my secret, when i try the new one and use it the problem fixed – victorxu2 Sep 16 '21 at 08:06
  • 1
    great, good to know you got it resolved! – unacorn Sep 16 '21 at 09:03