0

I've seen the post below but I am trying to set up this for airflow 1.7.1.3

Airflow Remote logging not working

Does anyone have a specific example of the format required in connections to specify the key file for a service account to access relevant storage bucket of a project? {"project":"","key_path":""} This is what I've tried.

Paul Atkinson
  • 1
  • 2
  • Your question is not clear. If you generate a key for a service account in GCP there's no need to modify it's format, therefore I don't understand what you are trying to do here. Also, there's no information on what kind of "connections" you're writing about. Please make the question more descriptive, the [Stack Overflow question checklist](https://meta.stackoverflow.com/questions/260648/stack-overflow-question-checklist) may help. – arudzinska Mar 28 '18 at 16:19
  • Remote Log Connection is using the default user ID of the server eg root. I need it to use a service account rather than a server ID, it does not pick up the ID within the bash profile. So I need to alter the connection for GCS to identify the service account key location as per another example I've seen but I do not know the correct label to use as key_path does not work. – Paul Atkinson Apr 12 '18 at 17:01

1 Answers1

1

It seems like your having trouble with Google credentilas. There are a lot of ways to solve it and I'll just explain my way of doing this. You must first create connection id in Airflow. You can either do it programmatically or by using Airflow web UI.

Here are instructions for doing this by UI:

Airflow web UI -> Admin -> Connections -> Create -> insert credential informations -> Save

example: enter image description here

To use GCS, you must indicate it on the "Scopes" of the connection. For the Keyfile Path, you must get a json file from your project. To get this you must:

Google Console → API & Services -> Credentials(Key shaped) -> API Credentials -> Create credentials -> Service account key -> choose your account key -> Create

Then you will have your json file downloaded.

The 'Conn Id' you put is the one that goes in "remote_log_conn_id" of your airflow.cfg file.

example: enter image description here

OGCheeze
  • 74
  • 1
  • 11