1

I recently was forced to switch databases to mysql from sqlite3 due to high concurrent write problems when running my celery tasks (these worked fine with sqlite3). Now that I have the mysql database configured, It throws a 

C:\Sites\deltafy-massive-feed-backend>celery -A backend worker -l info

-------------- celery@Robert-PC v3.1.11 (Cipater)
---- **** -----
--- * ***  * -- Windows-7-6.1.7601-SP1
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         backend:0x2d69b10
- ** ---------- .> transport:   django://localhost//
- ** ---------- .> results:     djcelery.backends.cache:CacheBackend
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
--------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . backend.celery.debug_task
  . private_api.tasks.gilt_script
  . private_api.tasks.nordstrom_script
  . private_api.tasks.zappos_script

[2014-06-14 01:37:13,625: ERROR/MainProcess] Unrecoverable error: WindowsError(123,             'The filename, directory name, or volume label syntax is incorrect')
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)
  File "C:\Python27\lib\site-packages\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery\bootsteps.py", line 373, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 925, in __init__
    self._setup_queues()
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1267, in _setup_queues
    self._inqueue = SimpleQueue()
  File "C:\Python27\lib\site-packages\billiard\queues.py", line 369, in __init__
    self._reader, self._writer = Pipe(duplex=False)
  File "C:\Python27\lib\site-packages\billiard\__init__.py", line 97, in Pipe
    return Pipe(duplex, rnonblock, wnonblock)
  File "C:\Python27\lib\site-packages\billiard\py2\connection.py", line 233, in Pipe
    1, obsize, ibsize, win32.NMPWAIT_WAIT_FOREVER, win32.NULL
WindowsError: [Error 123] The filename, directory name, or volume label syntax is     incorrect`

Also, I have a hunch that the address is the problem and contains invalid characters, so I went into connection.py and printed it. The address is:

c:\\.\pipe\pyc-5532-0-7ix5od

Any ideas as to what this is doing, or any solution to the problem is much appreciated! I haven't a clue even from my hours of searching.

robert
  • 819
  • 1
  • 10
  • 24
  • after testing by substituting the sqlite3 backend back into the settings file it still throws it, so it must be something else that's effecting it because it used to work just fine. – robert Jun 14 '14 at 06:54
  • I have encountered the same error. I have yet to find a solution though I have come accross this : http://support.microsoft.com/kb/318746 – Dawson Jun 19 '14 at 19:12

1 Answers1

2

This issue is a bug in billiard, the multiprocessing module fork used by Celery. Here is a link to the issue.

You may confirm this by running python -c "from billiard.connection import Pipe; p = Pipe(duplex=False)" which should produce the same error.

Cogniva has just pushed a fixed of this to their fork of billiard. At the time of writing this post this fix has been submitted though not merged into celery/billiard. This issue will be resolved in version 3.4.0.0 of billiard.

To install the fix : 

$ pip uninstall billiard
$ git clone git@github.com:cogniva/billiard.git
$ cd billiard 
$ git checkout bb6169ff1bdd8bc7c5520f7c47085063bc51d4c7
$ python setup.py install

To confirm this fix works run python -c "from billiard.connection import Pipe; p = Pipe(duplex=False)". This time you should not see an error message.

Dawson
  • 4,391
  • 2
  • 24
  • 33