Error with celery worker in elastic beanstalk (using django and SQS) [ImportError: The curl client requires the pycurl library.]

Question

I'm trying to deploy to elastic beanstalk a django project that uses celery periodic tasks, using SQS.

I've been more or less following the instructions here:

How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?

When I deploy to eb, the periodic tasks are not being executed.

Checking the celery-beat log, everything seems right:

celery beat v4.2.1 (windowlicker) is starting.
__    -    ... __   -        _
LocalTime -> 2019-01-27 09:48:16
Configuration ->
    . broker -> sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-01-27 09:48:44,659: INFO/MainProcess] beat: Starting...
[2019-01-27 09:48:44,660: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:44,809: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2019-01-27 09:48:44,809: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:49,865: INFO/MainProcess] Writing entries...
[2019-01-27 09:49:00,409: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:50:00,050: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:50,543: INFO/MainProcess] Writing entries...
[2019-01-27 09:52:00,048: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:53:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)

but there seems to be some problem with the celery worker. This is the error I get in celery-worker.log:

/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-01-27 09:49:12,513: INFO/MainProcess] Connected to sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
[2019-01-27 09:49:12,950: WARNING/MainProcess] /opt/python/run/venv/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-01-27 09:49:12,951: INFO/MainProcess] celery@ip-172-31-23-20 ready.
[2019-01-27 09:49:12,952: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 291, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 163, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 160, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 316, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 332, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 342, in _get_async
    qname, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
    region=self.region
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
    **kwargs
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 178, in __init__
    **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

 -------------- celery@ip-172-31-23-20 v4.2.1 (windowlicker)
---- **** -----
--- * ***  * -- Linux-4.14.88-72.76.amzn1.x86_64-x86_64-with-glibc2.3.4 2019-01-27 09:49:12
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         raiseflags:0x7fbf06309f28
- ** ---------- .> transport:   sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . check_users_inboxes
  . periodic_send
  . raiseflags.celery.debug_task
  . scrapeURLs
  . spreadsheets_update
  . sum_two_numbers

I've been searching for answers and some people seem to have the same problem. I've tried deploying different versions of celery and kombu but can't solve it.

This is my requirements.txt file:

amqp==2.4.0
anyjson==0.3.3
beautifulsoup4==4.7.1
billiard==3.5.0.5
boto==2.49.0
bs4==0.0.1
cachetools==3.0.0
celery==4.2.1
certifi==2018.11.29
chardet==3.0.4
Django==2.1.5
django-celery-beat==1.4.0
django-celery-results==1.0.4
django-timezone-field==3.0
google-api-python-client==1.7.7
google-auth==1.6.2
google-auth-httplib2==0.0.3
httplib2==0.12.0
idna==2.8
kombu==4.2.2.post1
oauth2client==4.1.3
psycopg2==2.7.7
pyasn1==0.4.5
pyasn1-modules==0.2.3
python-dateutil==2.7.5
pycurl==7.43.0 --global-option="--with-nss"
python-crontab==2.3.6
pytz==2018.9
requests==2.21.0
rsa==4.0
six==1.12.0
soupsieve==1.7.3
stripe==2.18.1
uritemplate==3.0.0
urllib3==1.24.1
vine==1.2.0

If I ssh connect to the instance and type "python -c 'import pycurl'" everything seems right, no problem there.

Does someone know how to solve it? Any help would be appreciated.

Thank you very much.

score 2 · Accepted Answer · answered Jan 30 '19 at 12:33

2

when you do

python -c 'import pycurl'

make sure that the python version is the same as python3.6, to be sure, try:

python3.6 -m pip install pycurl

answered Jan 30 '19 at 12:33

holalluis

108
2
10

1

as holalluis suggested, the problem was that the package was being installed in python2.7 instead of python3.6. On top of that, --global-option="--with-nss" had to be specified for the package to be found. Solved by doing `export PYCURL_SSL_LIBRARY=nss` and then `python3.6 -m pip install pycurl --global-option="--with-nss" --target=/opt/python/run/venv/lib/python3.6/site-packages --no-cache-dir --upgrade` – jaume Feb 12 '19 at 10:17
@jaume - How did you get permissions to run these commands? I tried to use `eb ssh` then run the command with no luck. I then tried `sudo -i` and reran your commands. That seemed to succeed, but I still get an error from the venv python3.6 install stating libcurl link-time backend is different from compile-time. Thanks! – vitale232 May 14 '19 at 18:04
hey @vitale232. I did it the way you suggest. Everything worked fine but then I deleted all the environment. When some months later I tried to recreate everything, I got the same error you say. So I was mistaken: this is not how I solved it. And I don't know how I solved it!!! I posted another question but I'm getting no answers: https://stackoverflow.com/questions/56043178/how-to-install-libcurl-with-nss-backend-in-aws-ec2-python-3-6-64bit-amazon-lin – jaume May 15 '19 at 20:55
:-( Thanks for the response. I'll chime in on your new question if I figure it out. It's surprising to me how easy it is to configure everything in eb... except Celery. – vitale232 May 17 '19 at 11:29
hey @vitale232 any luck? I had also posted a question in aws forums and someone with the same problem seemed to solve it but it's not working for me. I just replied to him for further help: https://forums.aws.amazon.com/thread.jspa?messageID=900056#900056 – jaume May 27 '19 at 18:27
@jaume - No luck yet. I've started to dockerize my app. I'm hopeful I can still use EB but avoid the dependency issues. – vitale232 May 27 '19 at 20:23
1

I ended up using OpenSSL and it works. It didn't at first but when I changed kombu's and celery's version it worked. So, in requirements.txt: pycurl==7.43.0 --global-option="--with-openssl" celery==4.1.1 kombu==4.5.0 – jaume May 29 '19 at 15:35

score 0 · Answer 2 · answered Nov 06 '20 at 16:29

For those trying to resolve the following issues while trying to deploy a Django application that uses Celery :

'Hub' object has no attribute '_current_http_client' ...

ImportError: The curl client requires the pycurl library.

I am using the following Packages :

celery==4.4.7
django-celery-beat==2.1.0
django-celery-results==1.2.1

Solution :-

Use the Elastic Beanstalk's config file to install pycurl, Refer my config file below:

packages:
  yum:
    gcc: []
    gcc-c++: []
    graphviz-devel: []
    libjpeg-turbo-devel: []
    libcurl-devel: []
    openssl-static.x86_64: []

container_commands:
  00_1_pip_upgrade:
    command: /opt/python/run/venv/bin/pip install --upgrade pip
    ignoreErrors: false
  00_2_install_pycurl:
    command: /opt/python/run/venv/bin/pip install pycurl --global-option="--with-openssl" --upgrade
    ignoreErrors: false
    leader_only: true
    .
    .
    .
  xx_add_rest_of_your_commands:

The second container command in the above code block should help resolve the issues.

Error with celery worker in elastic beanstalk (using django and SQS) [ImportError: The curl client requires the pycurl library.]

2 Answers2