I created a new environment with
conda create -n scraping python=2.7
I activate this environment with
source activate scraping
Then I proceed to install scrapy with (-n scraping doesn't seem necessary anymore, but I added it just in case)
conda install scrapy -n scraping
I now have the following installed
% conda list
# packages in environment at /Users/alexis/anaconda3/envs/scraping:
#
cffi 1.1.2 py27_0
cryptography 0.9.2 py27_0
cssselect 0.9.1 py27_0
enum34 1.0.4 py27_0
idna 2.0 py27_0
ipaddress 1.0.7 py27_0
ipython 3.2.1 py27_0
libxml2 2.9.2 0
libxslt 1.1.28 2
lxml 3.4.4 py27_0
nose 1.3.7 py27_0
openssl 1.0.1k 1
pip 7.1.0 py27_0
pyasn1 0.1.7 py27_0
pycparser 2.14 py27_0
pyopenssl 0.14 py27_0
python 2.7.10 0
python.app 1.2 py27_4
queuelib 1.2.2 py27_0
readline 6.2 2
scrapy 0.24.4 py27_0
setuptools 18.0.1 py27_0
six 1.9.0 py27_0
sqlite 3.8.4.1 1
tk 8.5.18 0
twisted 15.2.1 py27_0
w3lib 1.8.1 py27_1
zlib 1.2.8 0
zope.interface 4.1.2 py27_1
(I also installed ipython)
Now when I try to start a project I get
% scrapy startproject megadeluxe
Traceback (most recent call last):
File "/Users/alexis/anaconda3/envs/scraping/bin/scrapy", line 4, in <module>
from scrapy.cmdline import execute
File "/Users/alexis/anaconda3/envs/scraping/lib/python2.7/site- packages/scrapy/__init__.py", line 48, in <module>
from scrapy.spiders import Spider
ImportError: No module named spiders
Looking at the content of scraping/bin/scrapy I see no file named spiders.py and if I grep for class Spider I get no such class anywhere.
Did I do something wrong with the conda install at first? I install python 2.7, which is the requirement.
which openssl returns (I think that it's fine)
/Users/alexis/anaconda3/envs/scraping/bin/openssl
lxml is installed as we can see from conda list
I'm a little clueless here.
[Edit]
As mentioned in the comments, I was not using the latest version of scrap. This does not explain the error, but I proceeded to install the latest version (1.0) with pip (as of this writing conda only install version 0.24).
After doing so, starting a project would spit something out, ending with Library not loaded: libssl.1.0.0.dylib
Looking this up on stack overflow returned me this link Cannot Set Up a Scrapy Project
The answer given by Joe Hooper solved my problem https://stackoverflow.com/a/26961576/2883980
After all this I now have a solution, but have no clue as to the root reason.
I will put this here so other may find this useful (because I had no idea what DYLD_LIBRARY_PATH was used for)
man dyld
DESCRIPTION The dynamic linker uses the following environment variables.
They affect any program that uses the dynamic linker.DYLD_FRAMEWORK_PATH This is a colon separated list of directories that contain frameworks. The dynamic linker searches these directories before it searches for the framework by its install name. It allows you to test new versions of existing frameworks. (A framework is a library install name that ends in the form XXX.framework/Versions/YYY/XXX or XXX.framework/XXX, where XXX and YYY are any name.) For each framework that a program uses, the dynamic linker looks for the framework in each directory in DYLD_FRAMEWORK_PATH in turn. If it looks in all the directories and can't find the framework, it searches the directories in DYLD_LIBRARY_PATH in turn. If it still can't find the framework, it then searches DYLD_FALLBACK_FRAMEWORK_PATH and DYLD_FALL- BACK_LIBRARY_PATH in turn. Use the -L option to otool(1). to discover the frameworks and shared libraries that the executable is linked against.
