10

I have a Django project running behind Nginx, and Gunicorn. One of the applications interacts with network devices using Exscript, which is in turn using Paramiko. Some devices do not work properly when they are behind Gunicorn.

The same exact code will work fine from within the django-admin shell. It will also work if I launch the built in django server, but I still get the error if I bypass nginx, and connect directly to Gunicorn.

I tried moving the functionality to a celery task, it had the same exact problem, but only behind Gunicorn.

I wrote a script using django-extensions that works from the command line, but will fail if called via subprocess. But only behind Gunicorn.

The devices that are failing all seem to be F5 LTMs, and it looks like the buffer on the exscript object is being modified somehow. If I had to guess I would say that Gunicorn, and Exscript/Paramiko are somehow stepping on each others memory, or perhaps Gunicorn is setting some environment variable that Exscript is picking up on.

In any case I am thoroughly stumped, and would appreciate any guidance on how to troubleshooot this.

James Robinson
  • 822
  • 6
  • 13
  • How have you installed paramiko? With an OS package? With pip? Do you use virtualenv? Are you certain gunicorn uses the same paramiko as your command line? – Antonis Christofides Dec 03 '14 at 20:32
  • Parimiko was installed with pip, and only in the virtualenv that gunicorn is using. Good idea though. – James Robinson Dec 08 '14 at 15:34
  • 2
    If you still have the issue, care to comment which [runner type](http://gunicorn-docs.readthedocs.org/en/latest/run.html#commonly-used-arguments) are you using in gunicorn? When you say it fails, can you elaborate what exactly happens, (an exception, process hangs, etc)? – tutuDajuju Mar 07 '15 at 08:47
  • 2
    My guess is that the thing you're interacting with via paramiko is expecting a TTY -- an interactive terminal. – dcrosta Apr 07 '15 at 02:29

1 Answers1

1

Total guess, but perhaps this will be helpful in debugging. Basically, ensure you've removed all output buffering, which can often be hiding what is really happening when layering multiple big frameworks (like you are doing here).

Ensure that you disable all output buffering in Python, both for your foreground webserver process and for any worker processes (setting PYTHONUNBUFFERED is an easy way to ensure that none of your python scripts have buffering, at least on the standard library functions).

The terminal can also introduce buffers that make debugging exceptionally hard. Consider switching your command to stdbuf -o0 -e0 your command to disable buffers on stdout and stderr (your command could still re-enable them, but most programs do not).

Hamy
  • 20,662
  • 15
  • 74
  • 102