0

I have following pods:

NAME                                                        READY     STATUS      RESTARTS   AGE
airflow-database-init-job-ggk95                             0/1       Completed   0          3h
airflow-redis-0                                             1/1       Running     0          3h
airflow-scheduler-7594cd584-mlfrt                           2/2       Running     9          3h
airflow-sqlproxy-74f64b8b97-csl8h                           1/1       Running     0          3h
airflow-worker-5fcd4fffff-7w2sg                             2/2       Running     0          3h
airflow-worker-5fcd4fffff-m44bs                             2/2       Running     0          3h
airflow-worker-5fcd4fffff-mm55s                             2/2       Running     0          3h
composer-agent-0034135a-3fed-49a6-b173-9d3f9d0569db-ktwwt   0/1       Completed   0          3h
composer-agent-0034135a-3fed-49a6-b173-9d3f9d0569db-nmjvw   0/1       Error       0          3h
composer-agent-d043348f-025a-4aa1-89b4-d4a5fae91653-8zdwk   0/1       Completed   0          3h
composer-fluentd-daemon-grwsp                               1/1       Running     0          3h
composer-fluentd-daemon-rxhjc                               1/1       Running     0          3h
composer-fluentd-daemon-xxrmr                               1/1       Running     0          3h

I don't know which of them are webserver pods. airflow-worker is probably not webserver, righ? I want to poke it to check if it is working properly, because it seems not to.

Bunyk
  • 7,635
  • 8
  • 47
  • 79

2 Answers2

1

As explained in the documentation about Cloud Composer's architecture, the Airflow webserver is running in an App Engine flexible environment hosted in a Google-managed tenant project to which users do not have access.

Unfortunately, the Webserver logs are not forwarded to the Composer's main project (i.e. your project), although there is an open Feature Request in the Public Issue Tracker, so feel free to click on the star icon and comment on it in order to let the Composer engineering know about the importance of this feature and your use case too. Therefore, if you believe you have any other similar issue regarding the webserver itself, I recommend you to either contact support (if you are eligible to do so) or open an issue in the corresponding Public Issue Tracker so that your issue can be investigated by the GCP Support Team.

In case you want to know more about the Airflow Webserver, you can find some additional information in its documentation page too.

dsesto
  • 7,864
  • 2
  • 33
  • 50
  • I'm trying to debug this issue: https://stackoverflow.com/q/41560614/816449 (or this: https://stackoverflow.com/q/51218314/816449) Not sure where to continue with it. It seems to happen when I do datastore query. – Bunyk Dec 13 '18 at 17:43
  • 1
    Based on my experience, when a DAG is not clickable in the Airflow webserver, this is because something failed on the processing of the DAG (not its execution), such as a permissions issue, errors in the code, etc. In any case, given that this question is specifically about where to find the Webserver and not about the issue you are experiencing, I would recommend you to create a new post for your issue, providing more details about your scenario and the errors you are getting. Also, I think it may be worth contacting support through the channels I mentioned, too. – dsesto Dec 14 '18 at 08:51
1

With regards to logs of Airflow Webserver - these logs are visible in Stackdriver logging.

  1. Navigate to GCP Menu -> Logging -> Logs Viewer

  2. If you are using classical Stackdriver UI then select "Cloud Composer Environment" in the "resource" dropdown and then select "airflow-webserver" in the second drop down as it was shown in this picture

  3. If you are using new Stackdriver menu then put the following query into query box: 

query:resource.type="cloud_composer_environment"

logName="projects/<your project name>/logs/airflow-webserver"

... and you will get the logs generated by airflow-webserver.

rafalbiegacz
  • 96
  • 5