4

Trying to run a few simple tasks via celery. The header of the workerfile looks like 

from celery import Celery, group
from time import sleep
celery = Celery('workerprocess', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')

After passing the jobs, I am trying to read the results like this. 

jobresult=group(signatureList).apply_async()
while not jobresult.ready():sleep(30) #Line 6

The code is running perfectly in my desktop. The configuration is Python 3.6.7 and 4.15.0-20-generic #21-Ubuntu SMP.

When I try to run the same thing on my staging server (with the worker node running there too in the background), #Line 6 above throws the following error. 

kombu.exceptions.DecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

From the log, it appears the task is sent to the redis queue, is executed correctly, but the main process can not perform any operation on the jobresult variable to retrieve the output of the calculations. I have checked by pinging the redis-cli, and it echoes PONG. The server configuration (the ones which I think are relevant) are Python 3.5.2 and 4.4.0-89-generic #112-Ubuntu.

Since the task is already running in my local desktop, I guess it is a matter of dependency. But the error does not give any clue about what other libraries should I install, whether using PIP or apt-get. I looked up, Kombu (don't know what it does, but guess something important) and it is already installed. So how to resolve this?

This is the output from the redis-cli from a comment to the question. I am not sure what it means, though. 

127.0.0.1:6379> lrange celery 0 0
(empty list or set)
127.0.0.1:6379>
Della
  • 1,264
  • 2
  • 15
  • 32

2 Answers2

3

If your workerprocess is supposed to return utf8-encoded responses then this likely is not a missing dependency, but either a different library version or something wrong with the celery workerprocess as set up on your server.

There is a known problem with celery returning error messages that are not compatible with utf-8 encoding, though the specs say they should be. There are also multiple documented bugs in older versions (fixed in newer versions) that used wrong or mismatched encodings, especially in handling json.

The unfortunate result is that you are seeing a report complaining that the first character of the result (0x80) is invalid, rather than seeing the actual error (or mis-encoded data) being returned.

To debug this, activate enough logging to see the actual data or error result being returned, and work from there.

Alternatively, you may be able to treat the inbound data as binary, rather than utf8, which would allow bytes to come through unscathed. They still won't be readable as utf8 or ascii characters, but at least you will receive them.

You can see a number of ways in which other folks have handled unexpectedly non-utf8 data here.

Craig.Feied
  • 2,617
  • 2
  • 16
  • 25
1

One possibility is that default encoding for python is different between your local and server.

You can get default encoding by doing

python -c 'import sys; print(sys.getdefaultencoding())'

on your local and server.

If both are not same, most general way to change default encoding is the environment variable

export PYTHONIOENCODING=new_encoding

But it depends on the environment.

Below link explains more ideas to change encoding.

c.f. Changing default encoding of Python?

taka
  • 1,407
  • 5
  • 7
  • Thanks. But does not the shebang reading '#!/usr/bin/env python3;#encoding:utf-8' take care of that? – Della Jan 28 '19 at 10:10
  • Do you mean https://www.python.org/dev/peps/pep-0263/ ? This is for the raw strings in the source, not default encoding of file or network read, I think. You can confirm if it works by executing small script, which contains shebang and print code above. – taka Jan 28 '19 at 14:36