Problem
I didn't knowingly make any significant changes, but suddenly, when I try to run my django app locally, I get the following error:
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ foreman start
03:55:28 web.1 | started with pid 38101
03:55:29 web.1 | Traceback (most recent call last):
03:55:29 web.1 | File "/Users/macuser/Dropbox/code/heroku/awe01/bin/gunicorn", line 5, in <module>
03:55:29 web.1 | from pkg_resources import load_entry_point
03:55:29 web.1 | File "/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages/pkg_resources.py", line 22, in <module>
03:55:29 web.1 | import zipfile
03:55:29 web.1 | File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 6, in <module>
03:55:29 web.1 | import io
03:55:29 web.1 | File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
03:55:29 web.1 | import _io
03:55:29 web.1 | ImportError: dlopen(/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
03:55:29 web.1 | Referenced from: /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
03:55:29 web.1 | Expected in: flat namespace
03:55:29 web.1 | in /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
03:55:29 web.1 | exited with code 1
03:55:29 system | sending SIGTERM to all processes
SIGTERM received
(awe01)MoriartyMacBookAir13:awesomeapp macuser$
That foreman start command is how I always run the app locally (based on heroku instructions), instead of python manage.py runserver (so I don't know if python manage.py runserver was even working at all recently) but now the error messages (above) are the same for both foreman start and python manage.py runserver anyway.
I also get the same error even if I revert back to a much earlier git commit, a commit for which foreman start worked at the time, so I don't think it's a new line added recently to my code. Rather it seems something's changed about the way python is set up (if that's possible?)
Additional info
This all happened as suddenly as described in 31164285
I always run the project in a virtual environment as you can see from (awe01) in brackets
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ which python
/Users/macuser/Dropbox/code/heroku/awe01/bin/python
Based on this thread (31106958), the following output might be informative to somebody?
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ nm /System/Library/Frameworks/Python.framework/Versions/2.7/Python | grep PyErr_ReplaceException
0000000000092b82 T __PyErr_ReplaceException
Note that if, in this venv, I type "python", it tells me I'm using 2.7.6
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages/feedparser-5.1.2-py2.7.egg', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python27.zip', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/plat-darwin', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/plat-mac', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/macuser/Dropbox/code/heroku/awe01/Extras/lib/python', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-tk', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-old', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages']
Output of brew config
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ brew config
HOMEBREW_VERSION: 0.9.5
ORIGIN: https://github.com/Homebrew/homebrew.git
HEAD: 02d219b58d254c0e8bd77a73d901b0e03af13818
Last commit: 5 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: quad-core 64-bit sandybridge
OS X: 10.10.5-x86_64
Xcode: 6.3.2
CLT: N/A
Clang: 6.1 build 602
X11: N/A
System Ruby: 2.0.0-p481
Perl: /usr/bin/perl
Python: /Users/macuser/Dropbox/code/heroku/awe01/bin/python
Ruby: /usr/bin/ruby
Java: 1.8.0_40
(awe01)MoriartyMacBookAir13:awesomeapp macuser$
Output of brew doctor
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry and just ignore them. Thanks!
Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected header files:
/usr/local/include/node/ares.h
/usr/local/include/node/ares_version.h
/usr/local/include/node/nameser.h
/usr/local/include/node/node.h
/usr/local/include/node/node_buffer.h
/usr/local/include/node/node_internals.h
/usr/local/include/node/node_object_wrap.h
/usr/local/include/node/node_version.h
/usr/local/include/node/openssl/opensslconf.h
/usr/local/include/node/uv-private/ngx-queue.h
/usr/local/include/node/uv-private/stdint-msvc2008.h
/usr/local/include/node/uv-private/tree.h
/usr/local/include/node/uv-private/uv-bsd.h
/usr/local/include/node/uv-private/uv-darwin.h
/usr/local/include/node/uv-private/uv-linux.h
/usr/local/include/node/uv-private/uv-sunos.h
/usr/local/include/node/uv-private/uv-unix.h
/usr/local/include/node/uv-private/uv-win.h
/usr/local/include/node/uv.h
/usr/local/include/node/v8-debug.h
/usr/local/include/node/v8-preparser.h
/usr/local/include/node/v8-profiler.h
/usr/local/include/node/v8-testing.h
/usr/local/include/node/v8.h
/usr/local/include/node/v8stdint.h
/usr/local/include/node/zconf.h
/usr/local/include/node/zlib.h
Warning: Your Xcode (6.3.2) is outdated
Please update to Xcode 6.4.
Xcode can be updated from the App Store.
Warning: Some keg-only formula are linked into the Cellar.
Linking a keg-only formula, such as gettext, into the cellar with
`brew link <formula>` will cause other formulae to detect them during
the `./configure` step. This may cause problems when compiling those
other formulae.
Binaries provided by keg-only formulae may override system binaries
with other strange results.
You may wish to `brew unlink` these brews:
openssl
The command python manage.py shell fails with the same error, but when I jump into the python console directly I can see what imports work or don't work
(This thread recommended checking urllib2)
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 94, in <module>
import httplib
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 80, in <module>
import mimetools
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/mimetools.py", line 6, in <module>
import tempfile
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
Referenced from: /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
Attempts so far (no luck yet)
1. Re-install python
That was recommended here: https://github.com/SirVer/ultisnips/issues/508 and on question 31900156
So I ran the following:
brew uninstall python
brew update
brew install python
It didn't solve the issue.
This tells me it's not a django or app-specific issue; it's a python problem.
Here's the output after uninstalling and re-installing python; I get the same error for running pip command:
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ brew uninstall python
Uninstalling /usr/local/Cellar/python/2.7.10_2... (6194 files, 96M)
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ brew update
Already up-to-date.
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ brew install python
==> Downloading https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
Already downloaded: /Library/Caches/Homebrew/python-2.7.10.tgz
==> Downloading https://bugs.python.org/file30805/issue10910-workaround.txt
Already downloaded: /Library/Caches/Homebrew/python--patch-c075353337f9ff3ccf8091693d278782fcdff62c113245d8de43c5c7acc57daf.txt
==> Patching
patching file Include/pyport.h
Hunk #1 succeeded at 713 (offset 14 lines).
Hunk #2 succeeded at 736 (offset 14 lines).
==> ./configure --prefix=/usr/local/Cellar/python/2.7.10_2 --enable-ipv6 --datarootdir=/usr/local/Cellar/python/2.7.10_2/share --datadir=/usr/local/Cellar/python/
==> make
==> make install PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.10_2
==> make frameworkinstallextras PYTHONAPPSDIR=/usr/local/Cellar/python/2.7.10_2/share/python
==> Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-18.0.1.tar.gz
Already downloaded: /Library/Caches/Homebrew/python--setuptools-18.0.1.tar.gz
==> Downloading https://pypi.python.org/packages/source/p/pip/pip-7.1.0.tar.gz
Already downloaded: /Library/Caches/Homebrew/python--pip-7.1.0.tar.gz
==> Downloading https://pypi.python.org/packages/source/w/wheel/wheel-0.24.0.tar.gz
Already downloaded: /Library/Caches/Homebrew/python--wheel-0.24.0.tar.gz
==> /usr/local/Cellar/python/2.7.10_2/bin/python -s setup.py --no-user-cfg install --force --verbose --single-version-externally-managed --record=installed.txt --
==> /usr/local/Cellar/python/2.7.10_2/bin/python -s setup.py --no-user-cfg install --force --verbose --single-version-externally-managed --record=installed.txt --
==> /usr/local/Cellar/python/2.7.10_2/bin/python -s setup.py --no-user-cfg install --force --verbose --single-version-externally-managed --record=installed.txt --
==> Caveats
Pip and setuptools have been installed. To update them
pip install --upgrade pip setuptools
You can install Python packages with
pip install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Homebrew-and-Python.md
.app bundles were installed.
Run `brew linkapps python` to symlink these to /Applications.
==> Summary
/usr/local/Cellar/python/2.7.10_2: 6194 files, 96M, built in 3.4 minutes
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ pip install --upgrade
Traceback (most recent call last):
File "/Users/macuser/Dropbox/code/heroku/awe01/bin/pip", line 7, in <module>
from pip import main
File "/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages/pip/__init__.py", line 13, in <module>
from pip.utils import get_installed_distributions, get_prog
File "/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages/pip/utils/__init__.py", line 15, in <module>
import zipfile
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 6, in <module>
import io
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
Referenced from: /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ brew linkapps python
Linking /usr/local/opt/python/IDLE.app to /Applications.
Linking /usr/local/opt/python/Python Launcher.app to /Applications.
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ pip install --upgrade
Traceback (most recent call last):
File "/Users/macuser/Dropbox/code/heroku/awe01/bin/pip", line 7, in <module>
from pip import main
File "/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages/pip/__init__.py", line 13, in <module>
from pip.utils import get_installed_distributions, get_prog
File "/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/site-packages/pip/utils/__init__.py", line 15, in <module>
import zipfile
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 6, in <module>
import io
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyErr_ReplaceException
Referenced from: /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /Users/macuser/Dropbox/code/heroku/awe01/lib/python2.7/lib-dynload/_io.so
(awe01)MoriartyMacBookAir13:awesomeapp macuser$
The reason I checked urllib2 specifically was based on this comment on issue-1384
Note at this point the python installed is version 2.7.10_2 but when I just type python in the virtual env awe01 it still brings me to a shell where version is 2.7.6
(awe01)MoriartyMacBookAir13:awesomeapp macuser$ python
Python 2.7.6 (default, Sep 9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
2. Trying foreman start outside of virtual env
This means that python is the system python, 2.7.10
MoriartyMacBookAir13:awesomeapp macuser$ which python
/usr/bin/python
This also fails, but I get a different error message
MoriartyMacBookAir13:awesomeapp macuser$ foreman start
04:44:30 web.1 | started with pid 50330
04:44:30 web.1 | /usr/local/foreman/bin/foreman-runner: line 41: exec: gunicorn: not found
04:44:30 web.1 | exited with code 127
04:44:30 system | sending SIGTERM to all processes
SIGTERM received
MoriartyMacBookAir13:awesomeapp macuser$
Interestingly now, that foreman start error differs to this python manage.py runserver error:
MoriartyMacBookAir13:awesomeapp macuser$ python manage.py runserver
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
utility.execute()
File "/Library/Python/2.7/site-packages/django/core/management/__init__.py", line 345, in execute
settings.INSTALLED_APPS
File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 46, in __getattr__
self._setup(name)
File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 42, in _setup
self._wrapped = Settings(settings_module)
File "/Library/Python/2.7/site-packages/django/conf/__init__.py", line 98, in __init__
% (self.SETTINGS_MODULE, e)
ImportError: Could not import settings 'shareducate.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named dj_database_url
MoriartyMacBookAir13:awesomeapp macuser$
In other words, my problem now looks just like question 10812651, "suddenly I'm getting DJANGO_SETTINGS_MODULE is not defined when using runserver". Response to that question blamed context_processors, but I've made no deliberate adjustments to context_processors.
Note that while python manage.py shell still fails, if I just type python I can use the python shell... and the version is 2.7.10 now (not 2.7.6 as in venv above).
Also note in case urllib2 is relevant (and yes, this is used in my code also), now I CAN import it without any error:
MoriartyMacBookAir13:awesomeapp macuser$ python
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>>
The output: MoriartyMacBookAir13:awesomeapp macuser$ nm /System/Library/Frameworks/Python.framework/Versions/2.7/Python | grep PyErr_ReplaceException 0000000000092b82 T __PyErr_ReplaceException is exactly the same as when run inside the virtual env.
3. openssl
Hence why brew doctor shows a special note on openssl above: I had tried the following 3 lines of code from question 31888394
brew install openssl
brew link --force openssl
brew install python
(I think the openssl line of investigation follows from the urllib2 checking above)
Seems to have made no difference.
4. Attempt downgrade from 2.7.10 to 2.7.9
Not sure if this would make a difference since the venv seems to be 2.7.6 anyway, but when I do try [as recommended on github (issue-1367 and issue-1384 and in question 31888479)
brew switch python 2.7.9 I get error,
Error: python does not have a version "2.7.9" in the Cellar.
Versions available: 2.7.10_2
[EDIT: Steps 5-7 added based on comments]
EDIT: 5. Check/amend PATH
Advice such as23783054 and issue-40241 suggested amending PATH. I ran this single command from 5846204:
export PATH=/usr/local/bin:$PATH
Since the whole setup was working and then suddenly stopped working, I'm reluctant to make too many system changes that might complicate the picture...
EDIT: 6. virtualenv --clear [following @Maikflow's comment]
Resulted in this error
(awe01)MoriartyMacBookAir13:heroku macuser$ deactivate
MoriartyMacBookAir13:heroku macuser$ virtualenv --clear awe01
Deleting tree awe01/lib/python2.7
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 11, in <module>
sys.exit(main())
File "/Library/Python/2.7/site-packages/virtualenv.py", line 824, in main
symlink=options.symlink)
File "/Library/Python/2.7/site-packages/virtualenv.py", line 984, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/Library/Python/2.7/site-packages/virtualenv.py", line 1145, in install_python
rmtree(lib_dir)
File "/Library/Python/2.7/site-packages/virtualenv.py", line 508, in rmtree
shutil.rmtree(dir)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: 'awe01/lib/python2.7/site-packages/mutagen-1.22-py2.7.egg-info/dependency_links.txt'
MoriartyMacBookAir13:heroku macuser$
EDIT: Success
7. Create a new virtualenv with the system's python 2.7.10
[Following @Evert's comment]
(awe01)MoriartyMacBookAir13:awe01 macuser$ deactivate
MoriartyMacBookAir13:awe01 macuser$ cd ..
MoriartyMacBookAir13:heroku macuser$ virtualenv awe02 --no-site-packages
New python executable in awe02/bin/python
Installing setuptools, pip...done.
MoriartyMacBookAir13:heroku macuser$ cd awe02
MoriartyMacBookAir13:awe02 macuser$ source bin/activate
(awe02)MoriartyMacBookAir13:awe02 macuser$
At this point I copied the app folder, awesomeapp to be inside the folder awe02 (alongside bin, lib and include folders)
(awe02)MoriartyMacBookAir13:awe02 macuser$ cd awesomeapp/
(awe02)MoriartyMacBookAir13:awesomeapp macuser$ pip install -r requirements.txt
When I did pip install -r requirements.txt I got a bunch of errors for psycopg2
warning: unused function 'Dprintf' and
warning: implicit conversion loses integer precision: 'long' to 'int'
but right before I gave up programming for ever, I tried
foreman start
and it works!
(awe02)MoriartyMacBookAir13:awesomeapp macuser$ foreman start
16:28:16 web.1 | started with pid 14174
16:28:16 web.1 | 2015-08-22 16:28:16 [14174] [INFO] Starting gunicorn 19.0.0
16:28:16 web.1 | 2015-08-22 16:28:16 [14174] [INFO] Listening at: http://0.0.0.0:5000 (14174)
16:28:16 web.1 | 2015-08-22 16:28:16 [14174] [INFO] Using worker: sync
16:28:16 web.1 | 2015-08-22 16:28:16 [14177] [INFO] Booting worker with pid: 14177
This is python 2.7.10
(awe02)MoriartyMacBookAir13:awesomeapp macuser$ python manage.py shell
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import urllib2
>>>
Similarly python manage.py runserver works as well :)
Other thoughts
I think I have updated OS X to version 10.10.5 since the last time foreman start was working. Is it possible the system updated some core files?
Thanks
I am still teaching myself python/django so there are some gaps in my knowledge... Any trouble-shooting approaches or tips would be massively appreciated.
