Installing Python module with pip

Question

I'm trying to install a module called Scrapy. I installed it using

pip install Scrapy

I see the 'scrapy' folder in my /usr/local/lib/python2.7/site-packages, but when I try to import it in a Python program, is says there is no module by that name. Any ideas as to why this might be happening?

EDIT: Here is the output of the pip command:

    Downloading/unpacking Scrapy
  Downloading Scrapy-0.20.0.tar.gz (745kB): 745kB downloaded
  Running setup.py egg_info for package Scrapy

    no previously-included directories found matching 'docs/build'
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/local/lib/python2.7/site-packages (from w3lib>=1.2->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Scrapy
  Running setup.py install for Scrapy
    changing mode of build/scripts-2.7/scrapy from 644 to 755

    no previously-included directories found matching 'docs/build'
    changing mode of /usr/local/bin/scrapy to 755
Successfully installed Scrapy
Cleaning up...

When I run /usr/local/bin/scrapy I get the usage for the command and the available commands. I noticed that I have a python2.7 and python2.7-32 in my /usr/local/bin, and I remember installing the 32 bit version because of a problem with Mavericks.

Here is the output of python /usr/local/bin/scrapy:

Traceback (most recent call last): File "/usr/local/bin/scrapy", line 3, in <module> from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline

And head /usr/local/bin/scrapy:

#!/usr/local/opt/python/bin/python2.7 from scrapy.cmdline import execute execute()

@bheklilr I think so? Here is the output of ls -ld /usr/local: drwxrwxr-x 17 root admin 578 Oct 30 19:08 /usr/local — Mike, Nov 12 '13 at 21:03
Are you sure you are running the same python installation. Maybe you are running different version? — scriptmonster, Nov 12 '13 at 21:05
I'm not totally positive, but the output of "which python" is: /Library/Frameworks/Python.framework/Versions/2.7/bin/python — Mike, Nov 12 '13 at 21:07
@J.F.Sebastian No, there isn't. Should I append '/usr/local' to sys.path? — Mike, Nov 12 '13 at 23:57
what happens if you run `/usr/local/bin/scrapy`? Can you install any module? For example, `pip install pip-tools` and run `pip-review` — jfs, Nov 13 '13 at 00:16
When I run /usr/local/bin/scrapy I get the usage for the command and the available commands. I noticed that I have a python2.7 and python2.7-32 in my /usr/local/bin, and I remember installing the 32 bit version because of a problem with Mavericks. Could this be the conflict? — Mike, Nov 13 '13 at 04:05
What happens if you run `python /usr/local/bin/scrapy`? (put the command literally into your terminal) — jfs, Nov 13 '13 at 16:06
@J.F.Sebastian `Traceback (most recent call last): File "/usr/local/bin/scrapy", line 3, in from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline` — Mike, Nov 14 '13 at 00:00
@Miles: If `/path/to/scrapy` shows usage info but `python /path/to/scrapy` raises `ImportError` then there are at least two `python` executables on your machine and `pip` doesn't install into the default one that is available as `python`. What `head /usr/local/bin/scrapy` shows? — jfs, Nov 14 '13 at 00:03
@J.F.Sebastian Okay, that makes sense. Here is the output: `#!/usr/local/opt/python/bin/python2.7 from scrapy.cmdline import execute execute()` — Mike, Nov 14 '13 at 03:11
Do you have a version of Python installed at /usr/local/opt/python/bin/python2.7? If so, that's clearly what scrapy is trying to use, even though the package itself is not installed there. Not sure how that could happen. Some kind of install mix up. — Jud, Nov 18 '13 at 05:23
You could also try manually editing `/usr/local/bin/scrapy` to point it to your regular python install folder. — Jud, Nov 18 '13 at 05:29
@Jud: "even though the package itself is not installed there" is incorrect. `scrapy` *is* installed into `/usr/local/opt/...` otherwise `scrapy` command would produce `ImportError` instead of the usage info. [@cm2's answer](http://stackoverflow.com/a/20040531/4279) suggests a simple way to access particular python executable. If `python2.7` alias is defined to point to `#!/usr/local/opt/python/bin/python2.7` then it is enough to add `#!/usr/bin/env python2.7` shebang to any Python script that has `import scrapy` in it or just start `python2.7` from the command-line. — jfs, Nov 21 '13 at 01:33

score 10 · Accepted Answer · answered Nov 18 '13 at 05:12

Are you using Homebrew or MacPorts or something? As @J.F.Sebastian said, it sounds like you are having issues mixing the default python that comes with OS X, and one that is installed via a package manager... Try /usr/local/opt/python/bin/python2.7 -m scrapy and see if that throws an ImportError.

If that works, then you may want to consider making that python executable your default. Something like alias python2.7=/usr/local/opt/python/bin/python2.7 and then always use python2.7 instead of the default python. You can likewise just point python to the /urs/local... bit, but then you won't have easy access to the system (OS X-supplied) python if you ever needed it for some reason.

It turns out that the version of Python I had installed via homebrew was conflicting with the Python that I manually installed. — Mike, Nov 21 '13 at 02:11
Glad you got it figure out. I think all of us were saying very similar things, RE making sure scrapy is using the *right* python. — cm2, Nov 21 '13 at 05:35

score 2 · Answer 2 · edited May 23 '17 at 12:07

EDIT: You can force pip to install to an alternate location. The details are here: Install a Python package into a different directory using pip?. If you do indeed have extra Python folders on your system, maybe you can try directing scrapy to those, even if just for a temporary solution.

Can you post the output of the pip command? Perhaps it is failing somewhere?

Also, is it possible you have two versions of Python on your machine? Pip only installs to one location, but perhaps the version of Python on your path is different.

Finally, sometimes package names given to pip are not exactly the same as the name used to import. Check the documentation of the package. I took a quick look and the import should be lowercase:

import scrapy

I added the output of the pip command to my post. As far as I'm aware I only have python 2.7.5. I also tried import scrapy to no avail. — Mike, Nov 12 '13 at 21:01

score 1 · Answer 3 · edited May 23 '17 at 12:31

1

When all else fails you can always set the environment variable PYTHONPATH (see Permanently add a directory to PYTHONPATH for help) to the path where you installed Scrapy. (pending you're not using virtualenv -- and if you are please specify so we can help, it's generally a good idea to provide OS too)

edited May 23 '17 at 12:31

Community

1
1

answered Nov 20 '13 at 18:48

hharnisc

937
7
11

score 0 · Answer 4 · answered Nov 18 '13 at 05:59

if you run on Ubuntu:

use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.

Optionally, even if it solves your problem, it is always better to install python libraries on a virtual environment, using virtualenvwrapper to keep the libraries separated, try to examine the apt-get installation log to find out what tools where added, then remove scrapy python library and reinstall it in the virtual env. using pip

score 0 · Answer 5 · answered Nov 21 '13 at 02:45

It appears that the scrapy module that is installed on the Python path is an executable file that will bootstrap a Scrapy project directory for you.

The Python code in the scrapy executable looks like this:

   #!/usr/bin/env python

   from scrapy.cmdline import execute
   execute()

That's intended to be run from the command line rather than imported into your own Python project module.

According to the documentation for the project, running the scrapy executable with this syntax:

scrapy startproject <your-project-name>

will bootstrap a Scrapy project that has the following directory structure:

your-project-name/
scrapy.cfg
tutorial/
    __init__.py
    items.py
    pipelines.py
    settings.py
    spiders/
        __init__.py
        ...

There are a number of examples in the documentation that demonstrate how you create and run your own spiders, link extractors, etc., and how to manipulate the data that you retrieve with the application. They each demonstrate the appropriate Python imports from subdirectories in the scrapy package to get you up and running.

Hope that this helps.

Installing Python module with pip

5 Answers5

Linked