"OSError: [Errno 1] Operation not permitted" when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)

Question

I'm trying to install Scrapy Python framework in OSX 10.11 (El Capitan) via pip. The installation script downloads the required modules and at some point returns the following error:

OSError: [Errno 1] Operation not permitted: '/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

I've tried to deactivate the rootless feature in OSX 10.11 with the command:

sudo nvram boot-args="rootless=0";sudo reboot

but I still get the same error when the machine reboots.

Any clue or idea from my fellow StackExchangers?

If it helps, the full script output is the following:

sudo -s pip install scrapy
Collecting scrapy
  Downloading Scrapy-1.0.2-py2-none-any.whl (290kB)
    100% |████████████████████████████████| 290kB 345kB/s 
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from scrapy)
Collecting w3lib>=1.8.0 (from scrapy)
  Downloading w3lib-1.12.0-py2.py3-none-any.whl
Collecting lxml (from scrapy)
  Downloading lxml-3.4.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 112kB/s 
Collecting Twisted>=10.0.0 (from scrapy)
  Downloading Twisted-15.3.0.tar.bz2 (4.4MB)
    100% |████████████████████████████████| 4.4MB 94kB/s 
Collecting six>=1.5.2 (from scrapy)
  Downloading six-1.9.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from zope.interface>=3.6.0->Twisted>=10.0.0->scrapy)
Installing collected packages: six, w3lib, lxml, Twisted, scrapy
  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/basecommand.py", line 223, in main
status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/commands/install.py", line 299, in run
root=options.root_path,
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_install.py", line 726, in uninstall
paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-7.1.0-py2.7.egg/pip/utils/__init__.py", line 314, in renames
shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

Possible duplicate of [Unable to install nltk on Mac OS El Capitan](http://stackoverflow.com/questions/32898583/unable-to-install-nltk-on-mac-os-el-capitan) — hectorpal, Oct 07 '15 at 20:31
The questions is being discussed here: http://apple.stackexchange.com/questions/209572/how-to-use-pip-after-the-el-capitan-max-os-x-upgrade — hectorpal, Oct 07 '15 at 20:32
Maybe you it helps you without disable SIP: http://stackoverflow.com/a/32930419/3090526 — nvg58, Jan 08 '16 at 15:43
I had the exact same problem when I tried installing "virtualenvwrapper" using pip but this command worked for me: "sudo pip install --ignore-installed virtualenvwrapper" So it might be apply to your case as well without having to go through a painful round-about — Arabasta, Jan 15 '16 at 01:34

score 401 · Answer 1 · edited Dec 28 '15 at 00:33

401

pip install --ignore-installed six

Would do the trick.

Source: github.com/pypa/pip/issues/3165

edited Dec 28 '15 at 00:33

Luke Singham

1,536
2
20
38

answered Oct 14 '15 at 22:12

Sath

4,043
2
10
2

51

This worked well for me when installing awscli on El Capitan without messing with the python installation: `sudo pip install --ignore-installed awscli` – Luke Nov 06 '15 at 16:10
3

couldn't import scrapy when installed with this method `ImportError: cannot import name xmlrpc_client` – harinsa Nov 20 '15 at 09:23
3

`--ignore-installed` flag did the trick for the AWS Elastic Beanstalk Cli for me as well. Finally... – Friendly King Dec 04 '15 at 04:33
1

this worked well for me as well. I think this should be the correct answer. – Paul Carlton Dec 13 '15 at 18:55
This worked for me. Thank you for saving me from going into the OS tweaks snake pit! – Verticon Jan 13 '16 at 16:04
1

This worked for me in `sudo pip install --ignore-installed scrapy`. – Sophia Feng Apr 13 '16 at 05:32
This didn't work for me for Numpy/Matplotlib. I'm trying to upgrade Matplotlib, but can't seem to uninstall Numpy 1.8.0rc1 and replace it with Numpy 1.11.0. – Translunar Apr 14 '16 at 13:14
still needs sudo tho! – olleh May 10 '16 at 13:57
Worked for me installing TensorFlow – dmgig Jun 21 '16 at 18:40
Amazing, this finally solved my problem on El Capitan! My temporary solution was to use `virtualenv` that worked as well, but for a specific environment I needed packages on whole system to be installed. Now it worked like a charm: `sudo pip install --ignore-installed --upgrade -r requirements.txt ` – loretoparisi Aug 16 '16 at 20:23
1

@Luke I had the same issue yesterday when I tried to install the latest version of numpy. Your suggestion worked and saved me all kinds of time. Thank you. – Ramesh Ramasubramanian Nov 29 '16 at 22:49
This worked for me.. i think this is a better solution – QADeveloper Feb 08 '18 at 15:51
worked for me. I faced the same issue while I was installing virtualenvwrapper on HighSierra. – HibernatedGuy Mar 29 '18 at 07:02

score 153 · Accepted Answer · answered Oct 20 '15 at 19:40

153

I also think it's absolutely not necessary to start hacking OS X.

I was able to solve it doing a

brew install python

It seems that using the python / pip that comes with new El Capitan has some issues.

answered Oct 20 '15 at 19:40

Mario

2,619
1
24
22

2

Worked for me too, and got me the next Python 2.7.X. This answer should win – Russell_A Mar 09 '16 at 23:16
3

This is the best answer for El Capitan Mac users – zzzzzzzzzzzzzzzzzzzzzzzzzzzzzz Sep 01 '16 at 05:28
You have pointed it out correctly ! This should be accepted answer! – Muaaz salagar Apr 15 '17 at 00:37
that did it for me. ( Sierra 10.13.6 ) – Michael Guild Sep 20 '18 at 15:51
@codersofthedark maybe `brew install python3` could solve this as on newer systems python3 is now the default python. – Mario Mar 16 '19 at 18:50

Fabio · Answer 3 · 2017-06-26T15:28:07.193

89

As the other answers said, it's because of the new System Integrity Protection, but I believe the other answers are overcomplicated.

If you're only gonna use that package in the current user, you should be able to install it just fine, without the need to disable the SIP, by using the --user flag. Like this:

sudo pip install --user packagename

edited Jun 26 '17 at 15:28

answered Oct 07 '16 at 02:47

Fabio

3,015
2
29
49

2

I'm unclear on precisely how these different flags work, but this answer seems to make the most sense to me. (Also, it worked :) – xdhmoore Apr 07 '17 at 18:35
@xdhmoore Thanks, I think so too haha. No need to disable SIP or to install a new python or mess with dependencies. – Fabio Apr 10 '17 at 12:22
For mkvirtualenv, this did not install the package in the expected location /usr/local/bin. Instead, `sudo pip install mkvirtualenv --ignore-installed six` works as expected, putting virtualenvwrapper.sh in /usr/local/bin – Rob May 17 '17 at 20:33
@Rob It really doesn't. The --user flag only installs for current user, so it makes sense that it wouldn't install in a folder that is not user-specific. For me, this command installs my packages in "/Users/myusername/Library/Python/2.7/lib/python/site-packag‌es" for .py files and "/Users/myusername/Library/Python/2.7/bin" for .sh files – Fabio Jun 26 '17 at 15:25
@dietbacon I think we are in agreement? The command I shared put it in a system-wide available location, yours is for installing it for a given user. – Rob Jun 28 '17 at 23:57
@Rob Yeah, sort of. But /usr/local/bin shouldn't be the expected location if you're installing something only for the current user, as I wrote that was the case in the answer. – Fabio Jun 29 '17 at 12:17
This appeared to install in the user space for me, but then wasn't linked. What is the step to make sure that it is linked? OS X El Capitan 10.11.6 – Timothy Swan Sep 28 '17 at 15:11
@TimothySwan That's weird. If it's in "/Users/myusername/Library/Python/2.7/lib/python/site-packag‌‌es" or "/Users/myusername/Library/Python/2.7/bin", python should find it automatically, unless you're using another version of python you manually installed I think. You can see where python is loading packages from by opening the python shell, then running `import sys` and `sys.path`. That should show you the folders where python is looking for packages. – Fabio Sep 28 '17 at 22:01
1

This worked for me for a rasa-nlu . sudo pip install --user rasa_nlu – nantitv Aug 07 '18 at 20:23
Worked for me as well. – Thomas Crawford Sep 30 '19 at 08:49

score 42 · Answer 4 · edited Oct 22 '18 at 04:16

42

The high voted answers didn't work for me, it seems to work for El Capitan users. But for MacOS Sierra users try the following steps

brew install python
sudo pip install --user <package name>

edited Oct 22 '18 at 04:16

Cœur

37,241
25
195
267

answered Sep 27 '17 at 21:19

LYu

2,316
4
21
38

1

`sudo pip install --user ` worked for me, without the need to reinstall the python – Datageek Mar 28 '18 at 14:51

rholmes · Answer 5 · 2015-10-23T09:34:53.303

Warnings

I would suggest very strongly against modifying the system Python on Mac; there are numerous issues that can occur.

Your particular error shows that the installer has issues resolving the dependencies for Scrapy without impacting the current Python installation. The system uses Python for a number of essential tasks, so it's important to keep the system installation stable and as originally installed by Apple.

I would also exhaust all other possibilities before bypassing built in security.

Package Manager Solutions:

Please look into a Python virtualization tool such as virtualenv first; this will allow you to experiment safely.

Another useful tool to use languages and software without conflicting with your Mac OS is Homebrew. Like MacPorts or Fink, Homebrew is a package manager for Mac, and is useful for safely trying lots of other languages and tools.

"Roll your own" Software Installs:

If you don't like the package manager approach, you could use the /usr/local path or create an /opt/local directory for installing an alternate Python installation and fix up your paths in your .bashrc. Note that you'll have to enable root for these solutions.

How to do it anyway:

If you absolutely must disable the security check (and I sincerely hope it's for something other than messing with the system languages and resources), you can disable it temporarily and re-enable it using some of the techniques in this post on how to Disable System Integrity-Protection.

It's a bit of an investment in time and requires some command line proficiency, but the result is well worth it - I use similar virtual environments for Ruby (RVM), Perl, etc. Homebrew is useful for safely trying lots of other languages and tools as well. — rholmes, Aug 09 '15 at 02:06

score 16 · Answer 6 · answered Jun 14 '16 at 17:36

16

This did the trick for me:

   sudo pip install scrapy --ignore-installed six

answered Jun 14 '16 at 17:36

Roger

2,823
3
25
32

Yes, this works for `sudo pip install virtualenvwrapper --ignore-installed six` as well, avoiding this error: `OSError: [Errno 1] Operation not permitted: '/tmp/..../System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'` – Rob May 17 '17 at 20:34

score 12 · Answer 7 · answered Apr 28 '16 at 17:40

I tried to install AWS via pip in El Capitan but this error appear

OSError: [Errno 1] Operation not permitted: '/var/folders/wm/jhnj0g_s16gb36y8kwvrgm7h0000gp/T/pip-wTnb_D-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

I found the answer here

sudo -H pip install awscli --upgrade --ignore-installed six

It works for me :)

Ergun Ozyurt · Answer 8 · 2015-10-21T07:01:55.780

11

You should disable "System Integrity Protection" which is a new feature in El Capitan.

First, you should run the command for rootless config on your terminal

# nvram boot-args="rootless=0"
# reboot

Then, you should run the command below on recovery partition's terminal (Recovery OS)

# csrutil disable
# reboot

I've just solved my problem like that. I'm not sure that the first part is necessary. Try as you like.

--WARNING

You should enable SIP again after everything works;

Simply reboot again into Recovery Mode and run in terminal

# csrutil enable

csrutil: Configuring System Integrity Protection

edited Oct 21 '15 at 07:01

answered Sep 22 '15 at 17:29

Ergun Ozyurt

512
3
17

10

no, no, no. This is a util running in user space. Please see http://stackoverflow.com/a/33024464/743638 for a relatively simple way around this. Disabling protection from 1. unsigned kernel extensions 2. modify systems binaries 3. inspecting and mounting over system paths etc. etc. is just foolhardy. At least recommend TEMPORARILY removing SIP until after install, and then turning it back on. – Sacrilicious Oct 08 '15 at 19:35
6

I would not do this. This is asking for trouble. For a more graceful solution approach see my answer below. – Mario Oct 20 '15 at 19:41
1

the question was unanswered more than a month. i think, this was the only "findable" solution at that time. i will edit my answer with some warning to enable SIP after everyting has been installed. – Ergun Ozyurt Oct 21 '15 at 06:56

score 10 · Answer 9 · answered Jul 13 '17 at 14:40

I was getting the same error on on my MacOS Sierra. I followed these steps and successfully able to install scarpy package.

1. sudo pip install --ignore-installed six
2. sudo pip install --ignore-installed scrapy

MacBook-Air:~ shree$ scrapy version
Scrapy 1.4.0

score 6 · Answer 10 · answered Jun 26 '16 at 10:28

6

This did the trick for me.

sudo pip install --ignore-installed scrapy

answered Jun 26 '16 at 10:28

Miss Haha

69
1
1

score 5 · Answer 11 · answered Aug 28 '17 at 01:21

5

Tried a combination of some answers and this eventually worked:

sudo -H pip install --upgrade --ignore-installed awsebcli

Cheers

answered Aug 28 '17 at 01:21

saza

460
6
7

score 2 · Answer 12 · answered Dec 27 '16 at 04:09

2

install python again:

brew install python

try it again:

sudo pip install scrapy

works for me, hope it can help

answered Dec 27 '16 at 04:09

dominicli

29
1

score 1 · Answer 13 · answered Oct 19 '15 at 19:02

1

Restart Mac -> hold down "Command + R" after the startup chime -> Opens OS X Utilities -> Open Terminal and type "csrutil disable" -> Reboot OS X -> Open Terminal and check "csrutil status"

answered Oct 19 '15 at 19:02

Sandeep

346
5
8

score 1 · Answer 14 · answered May 24 '17 at 07:41

1

This command would work perfectly fine :D

sudo -H pip install --upgrade package_name --ignore-installed six

answered May 24 '17 at 07:41

Anmol Shrivastava

96
1
3

score 1 · Answer 15 · answered Nov 24 '17 at 08:25

1

Sometimes such behavior may be achieved if you try to install python3 lib in python2 folder using pip instead of pip3.

answered Nov 24 '17 at 08:25

user3860984

21
2

score 0 · Answer 16 · answered Jan 11 '16 at 12:19

-- close SIP(system Integrity Protection) -- then reboot, use command +R to enter debug mode, then select terminal: csrutil disable reboot

2.

sudo C_INCLUDE_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2 :/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2/libxml :/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include pip install scrapy --ignore-installed six

3. -- then remove old six, install it again sudo rm -rf /Library/Python/2.7/site-packages/six* sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six* sudo pip install six

4. -- then set it back csrutil enable reboot

-- crappy works now

score -1 · Answer 17 · answered Mar 07 '17 at 03:28

-1

it work for me:

pip install scrapy --user -U

answered Mar 07 '17 at 03:28

DearX

11
1

premonition · Answer 18 · 2017-06-01T20:31:40.150

-1

I was missing a dependency somewhere else along the line, so I installed the other requirements for the project like this:

pip install --user -r requirements.txt

edited Jun 01 '17 at 20:31

answered May 18 '17 at 15:59

premonition

289
4
7

"OSError: [Errno 1] Operation not permitted" when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)

18 Answers18

Linked

Related