4

Is there a method to add/modify google oauth scopes to a Cloud Composer 2.0 environment?
When installing Composer 2.0 there is no option to modify the oauth scopes from the UI or command line. I need to add google drive to the oauth scopes on the cluster so that my existing Python Operators will be able to access sheets in a google drive.

My understanding is that Composer 2.0 installs GKE with AutoPilot mode and I have also been looking for a way to add the oauth scopes to a GKE AutoPilot cluster as well with no success. Any ideas?

Is there alternately a way to specify scopes for the Python operator execution?

Ron Scott
  • 41
  • 1
  • good question, would love to know if you can alter API scopes after the fact of creation too (from my understanding you cannot) – the pillow Jan 21 '22 at 01:20
  • accessing to Google Sheets is so common that I don't understand how there are such a few operators. Some ideas: GoogleSheetsToBigQueryOperator, GoogleSheetsReadOperator, GoogleSheetsWriteOperator. – Mike Mar 29 '22 at 14:31

1 Answers1

1

Came across this same issue and finally figured it out.

  1. Create a Google Cloud Airflow Connection
  2. Within your DAG, where you call on the operator to build the task, include the parameter:
    • gcp_conn_id = '< name of connection >'

Cloud Composer v2 doesn't support OAuth on the Environment Level anymore because it now uses GKE Autopilot. GKE Autopilot does not support OAuth since this is a legacy method of GCE. For the connections that still need OAuth you will need to define these in an Airflow Connection and use the connection in your task. The scope of the connection seems to override and preexisting scopes that may be included by default, like the full cloud-platform scope. I had to specify both spreadsheets and cloud-platform in my connection for the sheets_to_gcs operator.

alexandrakayy
  • 91
  • 7
  • I like this solution... I have not tried it yet. I ultimately ended up writing python tasks and establishing the scope by initializing the cloud platform and spreadsheets python sdk classes using a credentials file. Not a great secure solution if you have multiple individuals accessing a single composer instance. But you can limit the access of the credentials to only the services necessary. Might be better to add the credentials to google secrets manager for use in the tasks when initiating the python client libraries. No great answers yet. – Ron Scott Apr 06 '22 at 14:00