2

Airflow v1.9.0 docker deployment based on puckel just broke for me with this error:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 27, in <module>
    args.func(args)
  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 891, in worker
    worker.run(**options)
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/worker.py", line 255, in run
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 99, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 122, in setup_instance
    self.should_use_eventloop() if use_eventloop is None
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 241, in should_use_eventloop
    self._conninfo.transport.implements.async and
  File "/usr/local/lib/python2.7/dist-packages/kombu/transport/base.py", line 125, in __getattr__
    raise AttributeError(key)
AttributeError: async
Cobman
  • 375
  • 1
  • 3
  • 12

2 Answers2

3

It is related to Celery AttributeError: async error error just reported where kombu was updated from 4.1.0 to 4.2.0. I fixed it by switching the install order per below:

&& pip install kombu==4.1.0 \
&& pip install celery[redis]==4.0.2 \
&& pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,s3]==$AIRFLOW_VERSION \

Seems like kombu needs to pinned to this version in source...

Cobman
  • 375
  • 1
  • 3
  • 12
3

The Cobman solution works, just FYI you can fix it too upgrading the Celery version (as they recommend in their repository)

&& pip install celery[redis]==4.1.1

skozz
  • 2,662
  • 3
  • 26
  • 37