3

I have a up and running Apache - Airflow 1.8.1 instance.

I got a working connection (and it's ID) to write to Google Cloud Storage and my airflow user has the permission to write to the bucket.

I try to use the remote log storage functionality by adding

remote_base_log_folder = 'gs://my-bucket/log'

remote_log_conn_id = 'my_working_conn_id'

And that's all (I didn't touch any configuration but that)

I restarted all the services but the log aren't uploading to gcs (my bucket it's still empty) and my filesystem space is still decreasing.

Have you enabled successfully remote log with gcs? If yes, what did you change / do?

Pierre CORBEL
  • 713
  • 1
  • 6
  • 14

1 Answers1

5

I manage to get the remote log to GCS. First, you need to give the service account permission to write to GCS bucket.

This is my GCP connection set up:

GCP connection setup

Then, edit the airflow.cfg file:

remote_base_log_folder = gs://my-backup/airflow_logs
remote_log_conn_id = my_gcp_conn

After editing the config file, you need to re-initialize it again:

airflow initdb

# start the web server, default port is 8080
airflow webserver -p 8080

Testing by turning on the "tutorial" DAG, you should be able to see the logs both locally and remotely in GCS:

enter image description here

Tuan Vu
  • 708
  • 7
  • 15
  • Thanks to your picture, I understood what was wrong . 1°) You need an airflow.cfg without quote . 2°) You need at least the https://www.googleapis.com/auth/devstorage.full_control scope. I only had the https://www.googleapis.com/auth/devstorage.read_write scope which wasn't enough – Pierre CORBEL Sep 21 '17 at 11:51
  • @PierreCORBEL How do you enable/add/set the required scope? – Kasa Oct 31 '17 at 02:28
  • @Kasa You can set the scope when you define your connection of type Google Cloud Platform. The last field is the field scope – Pierre CORBEL Oct 31 '17 at 06:55
  • @PierreCORBEL Thanks this worked. How were you able to tell you needed that specific scope? Trial and error; official docs; something else? – Kasa Nov 01 '17 at 01:30
  • @Kasa Official doc from here https://cloud.google.com/storage/docs/authentication#oauth-scopes – Pierre CORBEL Nov 01 '17 at 09:25