1

Context:

I am trying to launch graphite with pypy interpreter.

Error:

launching graphite(*) leads to

ImportError: cannot import name 'threadpool',

even though launching a python (pypy) interpreter and typing from twisted.python import threadpool works.

Full stacktrace:

15/09/2014 13:29:09 ::   File "app_main.py", line 75, in run_toplevel
15/09/2014 13:29:09 ::   File "/opt/graphite/bin/carbon-cache.py", line 30, in <module>
15/09/2014 13:29:09 ::     run_twistd_plugin(__file__)
15/09/2014 13:29:09 ::   File "/opt/graphite/lib/carbon/util.py", line 93, in run_twistd_plugin
15/09/2014 13:29:09 ::     runApp(config)
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/scripts/twistd.py", line 23, in runApp
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/application/app.py", line 380, in run
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/scripts/_twistd_unix.py", line 193, in postApplication
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/scripts/_twistd_unix.py", line 390, in startApplication
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/application/app.py", line 658, in startApplication
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/application/service.py", line 282, in startService
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/application/service.py", line 282, in startService
15/09/2014 13:29:09 ::   File "/opt/graphite/lib/carbon/writer.py", line 191, in startService
15/09/2014 13:29:09 ::     reactor.callInThread(writeForever)
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/internet/base.py", line 997, in callInThread
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/internet/base.py", line 989, in getThreadPool
15/09/2014 13:29:09 ::   File "/home/vagrant/test2/site-packages/twisted/internet/base.py", line 954, in _initThreadPool
15/09/2014 13:29:09 :: ImportError: cannot import name 'threadpool'

I am using pypy (2.3.1-linux_x86_64-portable) on centos 6.5 and have run pip to install twisted, whisper (and applied an additional patch)

(*) test2/bin/python /opt/graphite/bin/carbon-cache.py --instance=a start

Edit:

  • the virtualenv is called test2.

  • test2/site-packages/twisted/python/threadpool.py{,c} shows twisted has threadpool

  • test2/bin/pypy --info shows [usemodules] thread = True

  • 'twisted.python.threadpool' in sys.modules returns false just before the failing import

Edit2:

adding from twisted.python import threadpool earlier in the call stack (in /opt/graphite/lib/carbon/util.py for instance) works and make graphite work.

Community
  • 1
  • 1
kamaradclimber
  • 2,479
  • 1
  • 26
  • 45

2 Answers2

2

Does your installation of Twisted have the threadpool module? Look for a file named threadpool.py in the twisted/python/ directory. If this is missing, your installation of Twisted has been corrupted somehow. You might be able to fix it by re-installing Twisted (perhaps destroying your ... virtualenv? and creating a new one).

Does it have the stdlib threading module? Twisted's threading support only works if the underlying Python runtime supports threads. If this is missing you may need a different Python runtime. PyPy supports threads but perhaps you got a build with threads disabled somehow.

If neither of these is the problem, you might learn more by running Python with import debugging enabled. python -v enables minimal import debugging and python -vv enables more verbose import debugging. I'm not sure whether these behave the same way on PyPy as they do on CPython. Hopefully they do or things are a bit more difficult.

If that doesn't help then you might also try adding a breakpoint with pdb just before the import of the twisted.python.threadpool module and then stepping carefully through, inspecting state as you go. One thing to check could be whether sys.modules has a 'twisted.python.threadpool' item in it already, and if so, what value is associated with it. None will prevent the import system from even looking for a module implementation on disk and just fail.

Jean-Paul Calderone
  • 47,755
  • 6
  • 94
  • 122
2

I had a similar problem. The problem was that in carbon.conf's USER setting, I specified my own user "graphite". When you do this, the carbon-cache daemon will run as this user. The graphite user did not have permissions to the twisted modules. Solution was chown -R graphite /usr/lib64/python2.6/site-packages/twisted

mediastream
  • 253
  • 3
  • 8