2

I wrote a simple python program:

# /tmp/src/Code.py
import sys
print sys.path

# /tmp/src/Main.py
import Code

When I run it with python src/Main.py, it works as expected:

max% cd /tmp
max% setenv PYTHONPATH src
max% python src/Main.py
['/tmp/src',
 '/tmp/src',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

And just to make sure sys.path is working right, I create a file in the working directory:

# /tmp/Code.py
print "I never said to search CWD!!! Your Python is broken."
import sys
print sys.path

And the result is the same as above, as expected.

However, when I run in gunicorn I get:

max% gunicorn Main:app
2017-08-08 10:30:53 [26913] [INFO] Starting gunicorn 17.5
2017-08-08 10:30:53 [26913] [INFO] Listening at: http://127.0.0.1:8000 (26913)
2017-08-08 10:30:53 [26913] [INFO] Using worker: sync
2017-08-08 10:30:53 [26918] [INFO] Booting worker with pid: 26918
I never said to search CWD!!! Your Python is broken.
['/tmp',
 '/usr/bin',
 '/tmp/src',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

It appears that gunicorn randomly decided to add PWD to the sys.path. There is nothing in the gunicorn manpage about this.

python configuration:

Flask==0.10.1
Jinja2==2.7.2
MarkupSafe==0.18
PAM==0.4.2
Pillow==2.3.0
Twisted-Core==13.2.0
Twisted-Web==13.2.0
Werkzeug==0.9.4
adium-theme-ubuntu==0.3.4
apt-xapian-index==0.45
argparse==1.2.1
blinker==1.3
chardet==2.0.1
colorama==0.2.5
command-not-found==0.3
debtagshw==0.1
defer==1.0.6
dirspec==13.10
duplicity==0.6.23
gevent==1.0
greenlet==0.4.2
gunicorn==17.5
html5lib==0.999
httplib2==0.8
itsdangerous==0.22
lockfile==0.8
lxml==3.3.3
oauthlib==0.6.1
oneconf==0.3.7.14.04.1
pexpect==3.1
piston-mini-client==0.7.5
pyOpenSSL==0.13
pycrypto==2.6.1
pycups==1.9.66
pygobject==3.12.0
pyinotify==0.9.4
pyserial==2.6
pysmbc==1.0.14.1
python-apt==0.9.3.5ubuntu2
python-debian==0.1.21-nmu2ubuntu2
pyxdg==0.25
reportlab==3.0
requests==2.2.1
sessioninstaller==0.0.0
simplejson==3.3.1
six==1.5.2
software-center-aptd-plugins==0.0.0
ssh-import-id==3.21
system-service==0.1.6
unity-lens-photos==1.0
urllib3==1.7.1
wheel==0.24.0
wsgiref==0.1.2
xdiagnose==3.6.3build2
zope.interface==4.0.5

I know I could remove PWD from the sys.path by searching realpath(p) == realpath('.') but sometimes we want PWD in the PYTHONPATH. So a more careful solution is needed. Ideally we would try to figure out which piece of software is introducing the bug. It might simply be that the gunicorn manpage is incomplete?

personal_cloud
  • 3,943
  • 3
  • 28
  • 38
  • 1
    I found `gunicorn --pythonpath src` works better than `setenv PYTHONPATH src`. I wish gunicorn had actual documentation. – personal_cloud Aug 08 '17 at 17:50
  • It appears that gunicorn is prepending `--pythonpath` in front of PWD and then `PYTHONPATH`. So always running gunicorn with `--pythonpath $PYTHONPATH` should be a general solution. This really should be added to the manpage. – personal_cloud Aug 08 '17 at 18:00
  • describe how you solved your problem as an answer to the question, then approve it. – marcusshep Aug 10 '17 at 18:45
  • No, no. It turns out the `--pythonpath` is not reliable. On gunicorn 19.6.0, the PWD still gets inserted in front of the path. – personal_cloud Aug 10 '17 at 21:55
  • It appears that if you hack `wsgiapp.py` as shown in my answer below, then `--pythonpath` works as expected. – personal_cloud Sep 11 '17 at 16:51

2 Answers2

1

Edit /usr/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py and change the following line:

sys.path.insert(0, cwd)

to:

sys.path.append(cwd)

Then --pythonpath works as expected.

personal_cloud
  • 3,943
  • 3
  • 28
  • 38
  • How do we make a standard python package that executes the edit shown above? Let's call our package `gunicorn_pythonpath_enabler` or somesuch. – personal_cloud Sep 11 '17 at 16:53
  • Making a pull-request to Gunicorn would be more efficient than packaging a patch in some lib. – bfontaine Sep 11 '17 at 17:01
-1

I switched to BaseHTTPServer because it's 2X faster (my application has to scale efficiently).

And it's pre-installed in Python 2.7.

And it can stream: Python 2.7: streaming HTTP server supporting multiple connections on one port

And doesn't mess with standard logging settings: time.time() stops the server process silently when using Flask

And doesn't require me to hack WSGI.

And doesn't require 4 packages (gunicorn, gevent, flask, wsgi, werkzeug).

And the code base is much more stable.

And it's easier to understand what it's doing.

And users can just run ./app -arg instead of /usr/bin/gunicorn 'app:build_app("-arg")'

How many reasons do I need???

personal_cloud
  • 3,943
  • 3
  • 28
  • 38
  • Sure, `BaseHTTPServer` is small, but it has practically no features. What are you doing? If you just want to statically serve files, a server like Nginx may be better. – Nick T Sep 18 '17 at 16:53
  • @Nick I am Implementing a multithreaded service with a RESTful API and a database embedded in the server that various types of clients connect to, for submitting tasks, retrieving work handles, performing administration, etc. With scaling efficiency requirements. Note: SQL was way too slow for this application. – personal_cloud Sep 18 '17 at 16:55
  • @Nick please be more specific on which gunicorn features are actually useful to have in a package. When I got rid of it, I saved 30 lines of workaround code, which was greater than the 15 lines I had to add to get back Flask-like URL decorators and response code sugar. – personal_cloud Sep 18 '17 at 16:59
  • Whichever application server one prefers, this *doesn't actually answer the question about GUnicorn*. – Charles Duffy Sep 18 '17 at 20:59
  • @Charles Um, yes it does. The question was what to do about this problem with gunicorn. This was the best answer for me. Besides, my other answer got 0 votes. – personal_cloud Sep 18 '17 at 21:01
  • Not any more than "use C instead" is an answer to a problem with Python. It might be something a person could do, but it isn't a topical answer to the question *about Python* (or, in this case, about GUnicorn). – Charles Duffy Sep 18 '17 at 21:02
  • @Charles I gave a "topical answer" already. See above. Go hack your WSGI. That might be the best answer for you. But it wasn't the best answer for *me*. StackOverflow allows multiple answers; many will work for some people but not others. It is not reasonable to expect one answer to work for *all* people. – personal_cloud Sep 18 '17 at 21:04
  • Hey somebody just voted for my other answer! Cool! 20% more reputation!! Ooooohh yeah. – personal_cloud Sep 18 '17 at 21:06
  • That was me. It's topical, after all. :) – Charles Duffy Sep 18 '17 at 21:06
  • Thank you! I appreciate it. Now this is an interesting situation... If I accepted one answer but the other answer gets infinitely more votes, what does StackOverflow do? – personal_cloud Sep 18 '17 at 21:07
  • IIRC there's a point when a sufficiently-upvoted answer is ordered above the selected one, but I don't remember where that is. – Charles Duffy Sep 18 '17 at 21:09
  • @Nick Also keep in mind that many of the "features" provided in complex packages are there only to work around basic things (like streaming) that said complex package has obfuscated. In other words, you wouldn't need so many features if you just started with a simpler framework. – personal_cloud Sep 21 '17 at 18:55