8

I am trying to install PyCrypto on OS X 10.11.3 (El Capitan). I am using Python 3.5.1. I downloaded the gzip file from https://pypi.python.org/pypi/pycrypto and decompressed it. Then I ran python setup.py build like the instructions said and it appeared to do something, then it produced this output:

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -fwrapv -Wall -Wstrict-prototypes -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/MD2.c -o build/temp.macosx-10.6-intel-2.7/src/MD2.o
src/MD2.c:30:10: fatal error: 'string.h' file not found
#include <string.h>
         ^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

I tried python3 setup.py build and got some very similar output:

/usr/bin/clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -fno-common -dynamic -fwrapv -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c src/MD2.c -o build/temp.macosx-10.6-intel-3.5/src/MD2.o
src/MD2.c:30:10: fatal error: 'string.h' file not found
#include <string.h>
         ^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

I tried Googling to figure out what to do, but I couldn't find anything useful. How can I install PyCrypto?

EDIT: I also tried several other things like pip install pycrypto and sudo pip3 install pycrypto and they didn't work. @l'L'l helped me get it to work by doing several strange, complex things that I never would have though have myself. They are summarized in the answer below.

Elias Zamaria
  • 96,623
  • 33
  • 114
  • 148
  • 1
    Do you have xcode or the command line tools installed? You need that to be able to use the standard libraries, llvm etc. – adarsh Feb 18 '16 at 01:17
  • I have Xcode version 7.2. I tried typing `make` and `gcc` into the terminal, as advised by [this question](https://stackoverflow.com/questions/15371925/how-to-check-if-command-line-tools-is-installed) to test if I have the XCode command line tools installed and they told me `clang: error: no input files` and `make: *** No targets specified and no makefile found. Stop.` respectively, which would imply that I have them installed. I tried running `xcode-select -p; echo $?` and it told me `0`, which also must mean that they are installed. – Elias Zamaria Feb 18 '16 at 02:03
  • 1
    Perhaps creating this symlink works, assuming same issue exists with OS X 10.11 and Xcode 7.x? https://gist.github.com/chrisyip/5b3cdb6a916f55354201#file-stdlib-h-md – adarsh Feb 18 '16 at 02:10
  • 1
    In fact someone else seems to have had a very similar issue with a solution http://stackoverflow.com/questions/13588518/fatal-error-string-h-file-not-found-after-upgrading-to-python-3-2-and-install – adarsh Feb 18 '16 at 02:17
  • I looked at that question and ran `sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer /Developer` then `sudo ln -s /Developer/SDKs/MacOSX10.11.sdk MacOSX10.6.sdk`. I am getting the same error when I run `python3 setup.py build`. – Elias Zamaria Feb 18 '16 at 02:28
  • 1
    Did you check `ls -al /usr/include`? If that doesn't look right perhaps try `sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include /usr/include`? – adarsh Feb 18 '16 at 02:33
  • I tried `ls -al /usr/include` and got `ls: /usr/include: No such file or directory`. I tried `sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/S‌​DKs/MacOSX10.11.sdk/usr/include /usr/include`, not expecting it to work but just wanting to see what would happen, and it told me `ln: /usr/include: Operation not permitted`. – Elias Zamaria Feb 18 '16 at 02:37
  • That is peculiar that `/usr/include` doesn't exist. Someone [here](http://stackoverflow.com/a/27328921/1068887) seems to have had a similar issue and was solved by using `xcode-select --install`. Hopefully this works? – adarsh Feb 18 '16 at 02:42
  • I removed the symlinks I added, ran `xcode-select --install`, then ran `python3 setup.py build`, and it told me `clang: error: unable to find utility "clang", not a developer tool or in PATH`, then `error: command '/usr/bin/clang' failed with exit status 72`. – Elias Zamaria Feb 18 '16 at 02:49
  • 1
    Try `unset DYLD_LIBRARY_PATH`? – adarsh Feb 18 '16 at 02:55
  • I tried `echo $DYLD_LIBRARY_PATH` and didn't see anything, so I doubt that unsetting it will fix anything. But I tried running `unset DYLD_LIBRARY_PATH`, then `python3 setup.py build`, and got the same `clang: error: unable to find utility "clang", not a developer tool or in PATH` error. – Elias Zamaria Feb 18 '16 at 02:57
  • Perhaps try this https://forums.developer.apple.com/thread/27453. Hope this works. I apologize that I haven't been able to help much and I am out of ideas now :/ – adarsh Feb 18 '16 at 03:04
  • 1
    I will look at that when I get a chance. At least you tried to help me. Thanks. – Elias Zamaria Feb 18 '16 at 03:05
  • I am trying to start Xcode and it is crashing immediately. I am seeing a dialog box saying Xcode cannot be opened because of a problem. There is a long error message involving `Dyld Error Message`. I think I messed up something, but I don't know what, and I haven't yet had a chance to figure it out. **If anyone is reading this, I would recommend not trying any of the things mentioned in these comments.** – Elias Zamaria Feb 18 '16 at 05:52
  • I deleted Xcode and re-downloaded it. I now able to run Xcode again. But I still can't run `python3 setup.py build` without getting the `string.h` error. – Elias Zamaria Feb 20 '16 at 01:52
  • You are on 10.11. But the console out `/Developer/SDKs/MacOSX10.6.sdk` suggest that you are using 10.6 sdk? – ljk321 Feb 21 '16 at 02:50
  • @skyline75489, I noticed that. But I don't know what it means or what I am supposed to do about it. I am running `python3 setup.py build`. I don't know why it is trying to use the 10.6 SDK. BTW, I tried running `ls /Developer/SDKs/` and saw `MacOSX10.5.sdk` and `MacOSX10.6.sdk` and nothing else. – Elias Zamaria Feb 21 '16 at 04:28
  • I'm also using 10.11. And tell you what, I don't even have `/Developer/`! What's the output of `xcode-select -p`? – ljk321 Feb 21 '16 at 05:20
  • I ran `xcode-select -p` and got `/Applications/Xcode.app/Contents/Developer`. – Elias Zamaria Feb 21 '16 at 05:24
  • From my chat with @l'L'l, it seems like my problems were caused by an obscure bug in PyCrypto. But it also seems like PyCrypto is no longer being actively maintained. I found this: https://github.com/dlitz/pycrypto/issues/173 – Elias Zamaria Feb 22 '16 at 02:35
  • @EliasZamaria Yeah... but it's still weird that most people including me can install it without problem on 10.11. – ljk321 Feb 22 '16 at 02:40
  • @skyline75489, I agree. I think I have spent so much time on this that I have little interest in spending even more time figuring out why this is happening. I think I will try other cryptographic modules like PyCryptodome. – Elias Zamaria Feb 22 '16 at 02:45

2 Answers2

7

Overview:

The manual build you're trying looks like it might be failing because it's referencing the OS X 10.6 SDK, which you likely don't have, and is outdated for the most part. Also, SDKs are now stored in a completely different location than when the 10.6 SDK was in it's prime.

New SDKs location:

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/

Old SDKs location:

/Developer/SDKs/

Non-existent / outdated SDK:

Since it appears that when trying to build PyCrypto it's referencing the MacOSX10.6.sdk there are several things to consider:

  1. Why does it reference an outdated SDK
  2. Where is the SDK it's referencing set
  3. What should be done to correct the issue

Unless we audit the source code carefully we might not know exactly where the incorrect flags are set, but we can do our best to work with the information we have. From the error we can see that there are several instances where the 10.6 SDK's name pops up:

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -fwrapv -Wall -Wstrict-prototypes -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library

Building from source:

/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/MD2.c -o build/temp.macosx-10.6-intel-2.7/src/MD2.o
src/MD2.c:30:10: fatal error: 'string.h' file not found
#include <string.h>
                 ^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

Analyzing this we can see that the PyCrypto's MD2.c file is trying to be built using the flag -isysroot /Developer/SDKs/MacOSX10.6.sdk. It might be worth trying pip instead:

Installing with pip:

...
fatal error: 'string.h' file not found #include <string.h>
...

Same error; we should probably find out if the <string.h> header even exists on the system — Let's make a quick test C application to find out:

Testing the C headers:

$ echo "#include <string.h>
#include <stdio.h>
int main() { printf(\"TEST\n\"); return 0; }" > t.c
$ clang t.c -o t
$ ./t
TEST

It's apparent the header does exist because the test worked fine. This tells us that the problem is more likely related directly to the 10.6 SDK (which doesn't seem to exist on the system).

Symlinking (non-existing) 10.6 SDK to 10.11 SDK:

Since we haven't determined where the SDK is actually getting set we'll go ahead and try to create symlinks so that any reference of the old 10.6 SDK links to the latest SDK (10.11 at this time):

$ cd /Developer/SDKs
$ sudo ln -s /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk MacOSX10.6.sdk

We can verify the symlink by issuing the following command:

$ ls -lat
total 8
drwxr-xr-x  3 root  wheel  102 Feb 21 15:54 .
lrwxr-xr-x  1 root  wheel   99 Feb 21 15:54 MacOSX10.6.sdk -> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
drwxr-xr-x  3 root  wheel  102 Feb 21 15:52 ..

Now that we've successfully created the symlink let's try installing PyCrypto with pip once more:

$ sudo pip install pycrypto
Collecting pycrypto
  Downloading pycrypto-2.6.1.tar.gz (446kB)
    100% |████████████████████████████████| 446kB 1.2GB/s 
Installing collected packages: pycrypto
  Running setup.py install for pycrypto ... done
Successfully installed pycrypto-2.6.1

No errors! It looks like our problem is solved! Well, almost...

We still need to figure out what is responsible for setting the wrong (10.6) SDK during builds. Let's use the xcrun tool to see what the defaults are set at:

$ xcrun --show-sdk-version
10.11

The system default SDK is set to 10.11, so it must be get set incorrectly to 10.6 by Python, PyCrypto, or some other anomaly we might not have considered.

UPDATE:

After doing some recon it was discovered that Python 3 appears to be built with the OS X 10.6 SDK. In addition it's also setting the SDK to 10.6 and setting the (outdated) path in numerous places throughout the Python_Framework. There are so many references I won't bother listing them all, although here's an example:

Python_Framework Folder/Versions/3.5/lib/python3.5/config-3.5m/Makefile:79:CONFIGURE_CFLAGS= -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk

I can only assume the developers were trying to be as backward compatible as possible, however, it's unfortunately breaking forward compatibility in the process.


Notes:

Installing Python packages with pip can make life much easier in a lot of ways (package management, updating, uninstalling, etc.). For example installing PyCrypto should just be a matter of issuing the command:

$ sudo pip install pycrypto

If you have multiple Python's you can use the version number to install for that Python accordingly:

$ sudo pip3.5 install pycrypto

https://pip.pypa.io/en/stable/installing/

Community
  • 1
  • 1
l'L'l
  • 44,951
  • 10
  • 95
  • 146
  • Argh... I thought about using pip. The only reason I didn't is because the documentation I found didn't mention it. Anyway, I tried `pip install pycrypto`, `sudo pip install pycrypto`, and `sudo pip3.5 install pycrypto`, and each time, I got the same kind of `fatal error: 'string.h' file not found` error. – Elias Zamaria Feb 21 '16 at 18:04
  • Try downloading the command-line tools from Apple again, and then install them: http://adcdownload.apple.com/Developer_Tools/Command_Line_Tools_OS_X_10.11_for_Xcode_7.2/Command_Line_Tools_OS_X_10.11_for_Xcode_7.2.dmg. (you might need to be signed in to download). If things don't work after that then there is likely a path problem which still needs to be resolved. – l'L'l Feb 21 '16 at 18:28
  • You can also install via Xcode: https://developer.apple.com/library/prerelease/mac/recipes/xcode_help-locations_preferences/SpecifyingCommandLineTools/SpecifyingCommandLineTools.html#//apple_ref/doc/uid/TP40010523-CH7-SW1 – l'L'l Feb 21 '16 at 18:57
  • @I'L'l, the first link told me "This page no longer exists or the Apple ID you signed in with does not have permission to view this page.". – Elias Zamaria Feb 21 '16 at 19:35
  • @I'L'l, I tried the second link and the Command Line Tools pop-up menu only gave me one choice: "XCode 7.2 (7C68)". Under that, it said "/Applications/Xcode.app". I tried the "More Developer Tools..." menu option and saw a page asking for my Apple ID and password. I logged in, agreed to their developer agreement, and saw a download named "Command Line Tools OS X 10.11 for Xcode 7.2.dmg". I downloaded it, installed it, and ran `sudo pip3.5 install pycrypto`, and got the same `string.h` error. Same for `pip install pycrypto`. – Elias Zamaria Feb 21 '16 at 19:48
  • I ran `xcrun --show-sdk-version` and got this result: dyld: Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest Referenced from: /Applications/Xcode.app/Contents/Frameworks/IDEFoundation.framework/Versions/A/IDEFoundation Reason: image not found xcrun: error: unable to lookup item 'SDKVersion' in SDK '/' – Elias Zamaria Feb 22 '16 at 03:36
  • Also, I am now unable to run git. If I type `git` at the command line, I see this: sh: line 1: 72906 Trace/BPT trap: 5 /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk / -find git 2> /dev/null git: error: unable to find utility "git", not a developer tool or in PATH I don't know for sure if this is related to the stuff we did involving XCode, but I can swear I remember git working a few weeks ago, and maybe later than that. I would notice that it is broken. – Elias Zamaria Feb 22 '16 at 03:36
  • You've definitely got Xcode/System components that apparently have gone completely missing (I think it's mostly Xcode related). I would recommend trashing Xcode completely and redownload it again. We didn't change anything in your PATH, so the problems you mention are likely caused by the same issue that might've deleted the entire developer folder out of Xcode.app. If you continue to have issues then it might be worth considering reinstalling the OS just to get everything back that is mysteriously absent. – l'L'l Feb 22 '16 at 03:45
  • I am about to delete, redownload, and reinstall XCode. Reinstalling the OS sounds like a PITA so I probably won't do it unless something is truly messed up in a way I can't work around. – Elias Zamaria Feb 22 '16 at 03:50
  • I reinstalled XCode and I am now able to run `git` again. – Elias Zamaria Feb 22 '16 at 07:25
  • 1
    Thank you!! I had to delete/rename the existing `MacOSX10.6.sdk` before it would work. Verifying the link with `ls -lat` was key for me. Thanks for your great instructions! – Heather Piwowar Oct 01 '19 at 12:51
1

I had what might be a related issue on OSX El Capitan when I would run pip install pycrypto. I was seeing RuntimeError: autoconf error. All I had to do was run sudo xcodebuild -license and type agree after reviewing the licence agreement. Afterward I was able to use pip to install pycrypto.

jfunk
  • 7,176
  • 4
  • 37
  • 38