8

I am trying to use Scrapy on Mac OS X El Capitan. I have zsh installed and I have tried everything that I could find online to fix this issue. I have also looked at Scrapy throws ImportError: cannot import name xmlrpc_client and could not resolve my problem!

Python installed via brew and added "pip install scrapy":

➜  DriverEBV  which python
/usr/local/bin/python

My .zshrc has the following line:

export PATH=/usr/local/bin:$PATH
export PYTHONPATH="/Library/Python/2.7/site-packages"

This is the error I get:

➜  DriverEBV  scrapy runspider DriverEBV.py
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line 12, in <module>
    from scrapy.http.request.rpc import XmlRpcRequest
  File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py", line 7, in <module>
    from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

When I run "pip install scrapy" now this is what I see:

➜  DriverEBV  pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /Library/Python/2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.8.0 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.7 in /usr/local/lib/python2.7/site-packages (from pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.1.0 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/site-packages (from cffi>=1.1.0->cryptography>=0.7->pyOpenSSL->scrape)

Anybody able to help me?

Community
  • 1
  • 1
Dominik
  • 4,718
  • 13
  • 44
  • 58
  • Possible duplicate of [Scrapy throws ImportError: cannot import name xmlrpc\_client](http://stackoverflow.com/questions/30964836/scrapy-throws-importerror-cannot-import-name-xmlrpc-client) – eLRuLL Nov 26 '15 at 02:35
  • 2
    Unfortunately not a duplicate. I have tried everything discussed in the article you have posted. So have I tried pretty much everything else that was available online going from least intrusive to most intrusive. – Dominik Nov 26 '15 at 11:24
  • why do you set the python path here? – cel Nov 26 '15 at 11:27
  • The idea there was to add Python with pip in my user to be separate from Mac OS native pip and to install scrapy with --user. – Dominik Nov 26 '15 at 15:48
  • Are you sure your .zshrc got sourced? If you specifically type: `export PYTHONPATH="/Library/Python/2.7/site-packages"` in your terminal followed by a simple `scrapy shell http://stackoverflow.com` do you still get the error? – vrleboss Jan 23 '16 at 05:13
  • @JackMurphy I had to reinstall lxml, six, and setuptools to get everything working properly – Benjamin James May 07 '16 at 23:09

4 Answers4

2

I've had nothing but pain mucking with the Mac OS X system Python libraries installed in the /Library/Python directory. What has worked well for me is a combination of MacPorts and virtualenv:

  1. Install MacPorts

  2. Install Python, pip, and virtualenv from MacPorts:

    /opt/local/bin/port install python27
/opt/local/bin/port install py27-pip
/opt/local/bin/port install py27-virtualenv

  3. Setup virtualenv:

    /opt/local/bin/virtualenv-2.7 myenv

  4. Activate virtualenv (don't forget the dot!)

    . myenv/bin/activate

  5. Install scrapy

    pip install scrapy

This way, the system Python libraries are untouched and you can install whatever packages you like without having to remove or upgrade existing packages.

Mogsdad
  • 44,709
  • 21
  • 151
  • 275
Lex Scarisbrick
  • 1,540
  • 1
  • 24
  • 31
  • I think that solved my problem when running scrapy on the command line. when running in PyCharm, then I get this error. Any suggestions to solve? – Guy Daher May 10 '16 at 17:52
  • PyCharm appears to have support for virtualenv now. You should be able to [add an existing virtualenv](https://www.jetbrains.com/help/pycharm/2016.1/adding-existing-virtual-environment.html) via the [Project Interpreter](https://www.jetbrains.com/help/pycharm/2016.1/project-interpreter.html). Hope this helps. – Lex Scarisbrick May 10 '16 at 19:53
  • Yeah I figured that after posting this comment. Thanks for the information above, awarded you bounty – Guy Daher May 14 '16 at 16:54
1

What helped me was to uninstall six and scrapy and then install again:

pip uninstall six
pip uninstall scrapy

pip install six
pip install scrapy

Run with sudo if necessary.

Or, you can also try upgrading six and scrapy:

pip install --upgrade scrapy
pip install --upgrade six
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • I think that solved my problem when running scrapy on the command line. when running in PyCharm, then I get this error. Any suggestions to solve? – Guy Daher May 10 '16 at 17:52
  • @GuyDaher Is the Project Interpreter in the settings the correct one? (I believe in this example it should be the default python installation) – mkaran May 13 '16 at 13:09
0

Try uninstalling via pip and then reinstall using easy_install command. I had the same trouble with another python module and doing it this way fixed the issue for me on Mac OS X El Capitan.

Craig
  • 259
  • 1
  • 5
  • 14
-1

I believe the best solution on OS X should be "Don’t use system python". It will make life easier. This link shows how to do this.

There’s a known issue that prevents pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:

(Recommended) Don’t use system python, install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:

  1. Install homebrew following the instructions in http://brew.sh/
  2. Update your PATH variable to state that homebrew packages should be used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell):

echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc

  1. Reload .bashrc to ensure the changes have taken place:

source ~/.bashrc

  1. Install python:

brew install python

  1. Latest versions of python have pip bundled with them so you won’t need to install it separately. If this is not the case, upgrade python:

brew update; brew upgrade python

Community
  • 1
  • 1
Shupeng Xu
  • 31
  • 7