I have an issue testing my flask API app where it won't consistently start up. Because of requirements, I have to make sure it runs on CentOS6 (python 2.6 by default), but I use a custom python 2.7 virtualenv. My startup command is as follows:
/opt/mist/mist_base/env/bin/uwsgi --py-autoreload 1 --master --socket=/tmp/mist_app.sock --pidfile=/tmp/mist_app.pid --module=wsgi --honour-stdin --chdir=/opt/mist/mist_base/app --threads=1 --virtualenv=/opt/mist/mist_base/env --die-on-term --uid mist --gid mist
where the custom virtualenv is /opt/mist/mist_base/env.
I had someone help me package this software to be installable using RPM/Yum (he had done it before and packaging it this way was a requirement); during this process he insisted that he build another custom python virtualenv (/opt/mist/mist_base/.localpython). Unfortunately he has since left the company, and we don't have details as to what was done. I looked for RPM build/init commands that would call binaries/libraries in that other virtualenv but I didn't find any.
Using the instructions provided to me, I built the resulting RPM and installed it on my test server and everything runs just fine. I gave the same RPM to my customer to install in his test server and it does not start.
My working boot sequence is as follows:
*** Starting uWSGI 2.0.14 (64bit) on [Thu May 18 13:49:09 2017] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-17) on 20 March 2017 07:18:55
os: Linux-2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Apr 11 17:13:24 UTC 2017
nodename: mistdb
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /opt/mist/mist_base/app
writing pidfile to /tmp/mist_app.pid
detected binary path: /opt/mist/mist_base/env/bin/uwsgi
setgid() to 487
setuid() to 493
chdir() to /opt/mist/mist_base/app
your processes number limit is 7394
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/mist_app.sock fd 3
Python version: 2.7.5 (default, Mar 20 2017, 06:43:30) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
Set PythonHome to /opt/mist/mist_base/env
Python main interpreter initialized at 0x21c4aa0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145536 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x21c4aa0 pid: 24902 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 24902)
spawned uWSGI worker 1 (pid: 24907, cores: 1)
Python auto-reloader enabled
Meanwhile this is what my customer sees:
*** Starting uWSGI 2.0.14 (64bit) on [Tue May 16 13:12:26 2017] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-17) on 20 March 2017 07:18:55
os: Linux-2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Mar 21 12:19:18 EDT 2017
nodename: mist-testbed
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /
writing pidfile to /tmp/mist_app.pid
detected binary path: /opt/mist/mist_base/env/bin/uwsgi
setgid() to 497
setuid() to 497
chdir() to /opt/mist/mist_base/app
your processes number limit is 15208
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/mist_app.sock fd 3
Python version: 2.7.5 (default, Mar 20 2017, 06:43:30) [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)]
Set PythonHome to /opt/mist/mist_base/env
Python main interpreter initialized at 0x280c890
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145536 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
File "/opt/mist/mist_base/.localpython/lib/python2.7/hashlib.py", line 139, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/opt/mist/mist_base/.localpython/lib/python2.7/hashlib.py", line 103, in __get_openssl_constructor
return __get_builtin_constructor(name)
File "/opt/mist/mist_base/.localpython/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
Traceback (most recent call last):
File "./wsgi.py", line 1, in <module>
from mist_main import app as application
File "./mist_main.py", line 1, in <module>
from _app import return_app
File "./_app.py", line 1, in <module>
from flask import Flask, Blueprint, render_template, session, redirect, url_for, escape, request, abort
File "/opt/mist/mist_base/env/lib/python2.7/site-packages/flask/__init__.py", line 17, in <module>
from werkzeug.exceptions import abort
File "/opt/mist/mist_base/env/lib/python2.7/site-packages/werkzeug/__init__.py", line 151, in <module>
__import__('werkzeug.exceptions')
File "/opt/mist/mist_base/env/lib/python2.7/site-packages/werkzeug/exceptions.py", line 71, in <module>
from werkzeug.wrappers import Response
File "/opt/mist/mist_base/env/lib/python2.7/site-packages/werkzeug/wrappers.py", line 26, in <module>
from werkzeug.http import HTTP_STATUS_CODES, \
File "/opt/mist/mist_base/env/lib/python2.7/site-packages/werkzeug/http.py", line 32, in <module>
from hashlib import md5
ImportError: cannot import name md5
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 25071)
spawned uWSGI worker 1 (pid: 25072, cores: 1)
Python auto-reloader enabled
During startup, the app first tries to use the .localpython virtualenv, then it tries using the env virtualenv. Since I can't find any RPM/init code that uses .localpython, I can't explain why 1. my app instance doesn't use it 2. my customer's instance does.
It's entirely possible it's due to a difference in environments, and we're going to try and build a VM based on their specs and see if that resolves this issue. Could this also simply be a case of having a broken virtualenv and/or not having all the needed external libraries not included?
