0

I am trying to scale airflow using celery and rabbitMq on EC2.

I am following following code: http://site.clairvoyantsoft.com/setting-apache-airflow-cluster/

Following is code in master node.

sql_alchemy_conn = postgresql+psycopg2://user:gues@localhost:5432/airflow
executor = CeleryExecutor
broker_url = amqp://user:gues@ip-11-222-12-117:5672
celery_result_backend = db+postgresql://user:gues@localhost:5432/airflow

Following is code for salve node:

sql_alchemy_conn = postgresql+psycopg2://user:gues@ip-11-222-12-117:5432/airflow
executor = CeleryExecutor
broker_url = amqp://user:gues@ip-11-222-12-117:5672
celery_result_backend = db+postgresql://user:gues@localhost:5432/airflow

When I run airflow scheduler, it works fine. But on slave node I am getting following error:

[2017-05-23 21:47:44,385: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@ip-11-222-12-117:5672//: Couldn't log in: a socket error occurred.
Trying again in 2.00 seconds..

However I am able to see both nodes connected using rabbitMq on rabbitMQ UI.

What I am doing wrong?

Kush Patel
  • 3,685
  • 5
  • 42
  • 65

1 Answers1

1

Have you checked that the amqp server is allowed to listen to anything other than the loopback? Please check this answer: Can't access RabbitMQ web management interface after fresh install

Steen
  • 6,573
  • 3
  • 39
  • 56