3

I'm trying to trigger Airflow DAG inside of a composer environment with cloud functions. In order to do that I need to get the client id as described here. I've tried with curl command but it doesn't return any value. With a python script I keep getting this error:

Traceback (most recent call last):
  File "get_client_id.py", line 55, in <module>
    get_client_id(
  File "get_client_id.py", line 40, in get_client_id
    print(query_string['client_id'][0])
KeyError: 'client_id'

Python code:

import google.auth
import google.auth.transport.requests
import requests
import six.moves.urllib.parse

# Authenticate with Google Cloud.
# See: https://cloud.google.com/docs/authentication/getting-started
credentials, _ = google.auth.default(
    scopes=['https://www.googleapis.com/auth/cloud-platform'])
authed_session = google.auth.transport.requests.AuthorizedSession(
    credentials)

project_id = 'my-project'
location = 'my-region'
composer_environment = 'my-env'

environment_url = (
    'https://composer.googleapis.com/v1beta1/projects/{}/locations/{}'
    '/environments/{}').format(project_id, location, composer_environment)
composer_response = authed_session.request('GET', environment_url)
environment_data = composer_response.json()
airflow_uri = environment_data['config']['airflowUri']



# The Composer environment response does not include the IAP client ID.
# Make a second, unauthenticated HTTP request to the web server to get the
# redirect URI.
redirect_response = requests.get(airflow_uri, allow_redirects=False)
redirect_location = redirect_response.headers['location']

# Extract the client_id query parameter from the redirect.
parsed = six.moves.urllib.parse.urlparse(redirect_location)
query_string = six.moves.urllib.parse.parse_qs(parsed.query)
print(query_string['client_id'][0])

cURL command:

curl -v my_airflow_url 2>&1 >/dev/null | grep -o "client_id\=[A-Za-z0-9-]*\.apps\.googleusercontent\.com"

Does anybody have an idea how to get the Get the client_id of the IAM proxy?

Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
Dudes
  • 75
  • 6
  • 1
    Could you clarify if you are asking for `IAM` or `IAP`? Your title is poining to IAM but in code you have `...the IAP client ID`. If you are asking for `IAP`, did you see thread [How to authenticate programmatically to a Cloud Identity-Aware Proxy (Cloud IAP)-secured resource using user default credentials?](https://stackoverflow.com/questions/49071544/)? Also there is a docs in GCP [here](https://cloud.google.com/iap/docs/authentication-howto). Is that what you are looking for? – PjoterS Sep 22 '21 at 12:48
  • @PjoterS I'm asking for IAM. The code was provided by google in the documentation https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf#python – Dudes Sep 22 '21 at 13:01
  • 1
    Hi @Dudes, are you using Composer v2? This code was written for Composer v1 and I think just needs an update to work for Composer v2, which is still in preview. I work on these tutorials and will raise a bug to address this. – LEC Sep 23 '21 at 17:11
  • Is indeed Composer v2. – Dudes Sep 24 '21 at 11:03
  • @Dudes did you try to rewrite this to V2 or have you used Composer V1 to run this code? – PjoterS Sep 28 '21 at 08:55
  • I've tried with V1 and it works @PjoterS – Dudes Sep 28 '21 at 09:48

2 Answers2

1

Posting this Community Wiki for better visibility.

As mentioned in the comment section by @LEC this configuration is compatible with Cloud Composer V1 which can be found in GCP Documentation Triggering DAGs with Cloud Functions.

At the moment there can be found two tabs Cloud Composer 1 Guides and Cloud Composer 2 Guides. Under Cloud Composer 1 is code used by the OP, but if you will check Cloud Composer 2 under Manage DAGs > Triggering DAGs with Cloud Functions you will get information that there is not proper documentation yet.

This documentation page for Cloud Composer 2 is not yet available. Please use the page for Cloud Composer 1.

As solution, please use Cloud Composer V1.

PjoterS
  • 12,841
  • 1
  • 22
  • 54
  • The guide how to trigger a DAG using Stable API in Composer 2 was recently published: https://cloud.google.com/composer/docs/composer-2/access-airflow-api – rafalbiegacz Oct 15 '21 at 14:53
0

Here is the guide with instructions describing how to trigger a DAG in Composer 2/Airflow - https://cloud.google.com/composer/docs/composer-2/access-airflow-api