0

I have task that is set up as in this question.

Based on the UI it looks like the dependencies are well-defined:

enter image description here

I have tested individual tasks as follows: airflow test capone_dash_preproc AAAG5608078M2 2017-07-25. This works great, updating the target database entries corresponding to that user. When I try to run the full task, however, it seems like it's getting hung up. python3 dash_dag.py basically results in the following terminal output on the web server, repeated endlessly. All CPUs are quiet, so it doesn't seem like there's much calculation happening: 

/usr/local/lib/python3.6/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  .format(x=modname), ExtDeprecationWarning
[2017-07-25 16:48:23,266] [58627] {models.py:167} INFO - Filling up the DagBag from /Users/aaronpolhamus/airflow/dags
[2017-07-25 16:48:25 -0500] [58487] [INFO] Handling signal: ttou
[2017-07-25 16:49:13 -0500] [58557] [INFO] Worker exiting (pid: 58557)
[2017-07-25 16:49:44 -0500] [58487] [INFO] Handling signal: ttin
[2017-07-25 16:49:44 -0500] [58642] [INFO] Booting worker with pid: 58642
/usr/local/lib/python3.6/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  .format(x=modname), ExtDeprecationWarning
[2017-07-25 16:49:44,607] [58642] {models.py:167} INFO - Filling up the DagBag from /Users/aaronpolhamus/airflow/dags
[2017-07-25 16:49:46 -0500] [58487] [INFO] Handling signal: ttou
[2017-07-25 16:50:21 -0500] [58568] [INFO] Worker exiting (pid: 58568)
[2017-07-25 16:50:51 -0500] [58487] [INFO] Handling signal: ttin
[2017-07-25 16:50:51 -0500] [58661] [INFO] Booting worker with pid: 58661
/usr/local/lib/python3.6/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  .format(x=modname), ExtDeprecationWarning
[2017-07-25 16:50:52,324] [58661] {models.py:167} INFO - Filling up the DagBag from /Users/aaronpolhamus/airflow/dags
[2017-07-25 16:50:54 -0500] [58487] [INFO] Handling signal: ttou
[2017-07-25 16:51:20 -0500] [58596] [INFO] Worker exiting (pid: 58596)
[2017-07-25 16:51:50 -0500] [58487] [INFO] Handling signal: ttin
[2017-07-25 16:51:50 -0500] [58677] [INFO] Booting worker with pid: 58677
...

My confusion arises from the fact that the individual tests run just fine and populate the database. It is the entire run that hangs and fails. Anything obvious here?

aaron
  • 6,339
  • 12
  • 54
  • 80
  • 1
    Have you tried something like `airflow backfill capone_dash_preproc -s YYYYMMDD -e YYYYMMDD` – Chengzhi Jul 26 '17 at 00:23
  • Hi @Chengzhi, not yet. Why would that be likely to fix the problem? Should I do this for dates that I have not yet attempted runs for? – aaron Jul 26 '17 at 00:51
  • Hey, @Aaron, it won't depends on scheduler so you can test if all the tasks can run. You can clear the tasks that you have run then backfill as well. – Chengzhi Jul 26 '17 at 14:08

1 Answers1

1

Airflow exists of multiple components. The most important being the scheduler, webserver, and worker (in case of horizontal scaling). The scheduler always* needs to be running to have Airflow execute tasks. The webserver is only there to provide a nice UI on top of what the scheduler is doing and for the Rest API. All components use the backing database.

You are showing the output of the webserver, while I would have expected to see the output of the scheduler. From the screenshot you posted and the way you try running your DAG my assumption is that you did not start it.

So start the scheduler by issuing airflow scheduler on the command line and your DAG will probably kick off.

*backfills are an exception

Bolke de Bruin
  • 690
  • 4
  • 8
  • Makes great sense. I will try this suggestion in a couple hours and return to select your answer. Cheers! – aaron Jul 26 '17 at 17:29
  • That was it, thank you so much. Quick question (which I know I could get from the docs): How do I control how many tasks execute in parallel at one time? I almost blew up my computer when I did this, haha – aaron Jul 27 '17 at 00:29