LookupError: unknown encoding: hex_codec(run code in not root user)

Question

create a Flask Test Project demo

1. app.py

from flask import Flask
import codecs

app = Flask(__name__)


@app.route('/')
def hello_world():
    c = codecs.encode(b"sdf", "hex_codec")

    return 'Hello World!({})'.format(c)


if __name__ == '__main__':
    app.run()

2. gunicorn_config.py

# -*- coding: utf-8 -*-
# from gunicorn.workers.ggevent import GeventWorker

import os
from gevent import monkey
import multiprocessing

HOST = "0.0.0.0"
PORT = "5000"
USER = "www"
GROUP = "www"

APP_LOGS = "logs"

def get_logs_path(filename):
    return os.path.join(APP_LOGS, filename)

monkey.patch_all()
# check_config = True
debug = True
print("debug: {}".format(debug))


bind = "{}:{}".format(
    HOST,
    PORT
)

print("bind: {}".format(bind))

# 设置进程文件 '/var/run/*.pid'
pidfile = get_logs_path('gunicorn.pid')
print("pidfile: {}".format(pidfile))
logfile = get_logs_path('gunicorn.log')
print("logfile: {}".format(logfile))
# 设置日志级别
loglevel = 'debug'
print("loglevel: {}".format(loglevel))
# 设置访问日志和错误信息日志路径
accesslog = get_logs_path('gunicorn_access.log')
errorlog = get_logs_path('gunicorn_error.log')
print("accesslog: {}".format(accesslog))
print("errorlog: {}".format(errorlog))
# 启动的进程数 cpu核心*2
workers = multiprocessing.cpu_count() * 2
print("workers: {}".format(workers))

# 工作模式协程 默认是sync 可以改成gevent
worker_class = 'gunicorn.workers.ggevent.GeventWorker'
print("worker_class: {}".format(worker_class))

# 设置最大并发量, 默认是1000
worker_connections = 2000
print("worker_connections: {}".format(worker_connections))


# 指定每个工作者的线程数，默认是1
threads = 2
print("threads: {}".format(threads))

# 设置守护进程，将进程交给supervisord管理
daemon = False
print("daemon: {}".format(daemon))

# 指定用户/用户组
user = USER
group = GROUP
print("user: {}".format(user))
print("group: {}".format(group))

x_forwarded_for_header = 'X-FORWARDED-FOR'
print("x_forwarded_for_header: {}".format(x_forwarded_for_header))


def main():
    pass


if __name__ == '__main__':
    main()

3. demo_supervisord.ini

[program: demo]
command=/www/wwwroot/demo/venv/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
directory=/www/wwwroot/demo
stopsignal=QUIT
stopasgroup=true
killasgroup=true
redirect_stderr=false
stdout_logfile=/www/wwwroot/demo/logs/supervisord_stdout.log
stderr_logfile=/www/wwwroot/demo/logs/supervisord_stderr.log
environment=LC_CTYPE="en_US.UTF-8",LANG="en_US.UTF-8",LC_ALL="en_US.UTF-8"

4. requirement.txt

Click==7.0
Flask==1.1.1
gevent==1.4.0
greenlet==0.4.15
gunicorn==20.0.0
itsdangerous==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
Werkzeug==0.16.0

5. Deploy in CentOS 7.4 64Bit

CentOS 7.4

Python 3.6.4

Supervisrod 4.2.0

put code file in /www/wwwroot/demo/*
copy demo_supervisor.ini to supervisord’s config path. eg./etc/supervisord.d/demo_supervosr.ini
run some commands

# 1. create logs directory to save logs file
~# cd /www/wwwroot/demo
~# mkdir logs
~# chmod 766 logs
# create supervisord’s logs file
~# touch /www/wwwroot/demo/logs/supervisord_stdout.log
~# touch /www/wwwroot/demo/logs/supervisord_stderr.log

# 2. create venv on `/www/wwwroot/demo/venv`
~$ python3 -m venv venv

# 3. run `demo` and listen 5000 port
~# systemctl start supervisord

# 4. see process
~# ps aux | grep demo

# 5. test api
~# curl 127.0.0.1:5000/

# then you can find some logs file in `/www/wwwroot/demo/logs`

6. Try To

run in `root` user

in gunicorn_config.py set USER="root" and GROUP="root”

in demo_supervisor.ini set command=/www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app or command=/www/wwwroot/demo/venv/bin/gunicorn -c /www/wwwroot/demo/gunicorn_config.py app:app

works ok!

curl 127.0.0.1:5000/ will response Hello World!(b'736466’)

run in `www` user

in gunicorn_config.py set USER="www" and GROUP="www”

branch 1:

in demo_supervisor.ini set command=/www/wwwroot/demo/venv/bin/gunicorn -c /www/wwwroot/demo/gunicorn_config.py app:app

~# systemctl restart supervisord
~# supervisrctl
demo                           RUNNING   pid 9224, uptime 0:03:20
supervisor>

~# ps aux | grep demo
root      9224  0.2  0.1 255744 23516 ?        S    15:22   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn -c /www/wwwroot/demo/gunicorn_config.py app:app
root      9555  0.0  0.0 112808   968 pts/0    R+   15:26   0:00 grep --color=auto demo

# only one process, workers don’t work!


~# curl 127.0.0.1:5000/
# will no response, to see logs/gunicorn_error.log

[2020-06-12 15:28:07 +0800] [9668] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process
    super().init_process()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 133, in init_process
    self.load_wsgi()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/gunicorn/util.py", line 331, in import_app
    mod = importlib.import_module(module)
  File "/root/.pyenv/versions/3.6.4/lib/python3.6/importlib/__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/www/wwwroot/demo/app.py", line 1, in <module>
    from flask import Flask
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/jinja2/__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/jinja2/environment.py", line 15, in <module>
    from jinja2 import nodes
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/jinja2/nodes.py", line 19, in <module>
    from jinja2.utils import Markup
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/jinja2/utils.py", line 12, in <module>
    import json
ModuleNotFoundError: No module named 'json'

branch 2:

in demo_supervisor.ini set command=/www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app

~# systemctl restart supervisord
~# suervisorctl
demo                           RUNNING   pid 10095, uptime 0:00:22
supervisor>

~$ ps aux | grep demo
root     10095  0.4  0.1 277956 30452 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10100  0.0  0.1 277956 26228 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10101  0.0  0.1 277956 26036 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10102  0.0  0.1 277956 26036 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10103  0.0  0.1 277956 26036 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10104  0.0  0.1 277956 26036 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10105  0.0  0.1 277956 26248 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10106  0.0  0.1 277956 26248 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10107  0.0  0.1 277956 26152 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10108  0.0  0.1 277956 26136 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10110  0.0  0.1 277956 26152 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10111  0.0  0.1 277956 26128 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10112  0.0  0.1 277956 26136 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10113  0.0  0.1 277956 26132 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10114  0.0  0.1 277956 26136 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10115  0.0  0.1 277956 26132 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
www      10116  0.0  0.1 277956 26132 ?        S    15:33   0:00 /www/wwwroot/demo/venv/bin/python /www/wwwroot/demo/venv/bin/gunicorn --preload -c /www/wwwroot/demo/gunicorn_config.py app:app
root     10202  0.0  0.0 112808   968 pts/0    S+   15:35   0:00 grep --color=auto demo

has some workers, it’s ok!

~# curl 127.0.0.1:5000/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>


# see logs/gunicorn_error.log

[2020-06-12 15:33:24 +0800] [10095] [DEBUG] Current configuration:
  config: /www/wwwroot/demo/gunicorn_config.py
  bind: ['0.0.0.0:5000']
  backlog: 2048
  workers: 16
  worker_class: gunicorn.workers.ggevent.GeventWorker
  threads: 2
  worker_connections: 2000
  max_requests: 0
  max_requests_jitter: 0
  timeout: 30
  graceful_timeout: 30
  keepalive: 2
  limit_request_line: 4094
  limit_request_fields: 100
  limit_request_field_size: 8190
  reload: False
  reload_engine: auto
  reload_extra_files: []
  spew: False
  check_config: False
  preload_app: True
  sendfile: None
  reuse_port: False
  chdir: /www/wwwroot/demo
  daemon: False
  raw_env: []
  pidfile: logs/gunicorn.pid
  worker_tmp_dir: None
  user: 1000
  group: 1000
  umask: 0
  initgroups: False
  tmp_upload_dir: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  forwarded_allow_ips: ['127.0.0.1']
  accesslog: logs/gunicorn_access.log
  disable_redirect_access_to_syslog: False
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  errorlog: logs/gunicorn_error.log
  loglevel: debug
  capture_output: False
  logger_class: gunicorn.glogging.Logger
  logconfig: None
  logconfig_dict: {}
  syslog_addr: udp://localhost:514
  syslog: False
  syslog_prefix: None
  syslog_facility: user
  enable_stdio_inheritance: False
  statsd_host: None
  dogstatsd_tags:
  statsd_prefix:
  proc_name: None
  default_proc_name: app:app
  pythonpath: None
  paste: None
  on_starting: <function OnStarting.on_starting at 0x7f3e7d832d08>
  on_reload: <function OnReload.on_reload at 0x7f3e7d832e18>
  when_ready: <function WhenReady.when_ready at 0x7f3e7d832f28>
  pre_fork: <function Prefork.pre_fork at 0x7f3e7d84c0d0>
  post_fork: <function Postfork.post_fork at 0x7f3e7d84c1e0>
  post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f3e7d84c2f0>
  worker_int: <function WorkerInt.worker_int at 0x7f3e7d84c400>
  worker_abort: <function WorkerAbort.worker_abort at 0x7f3e7d84c510>
  pre_exec: <function PreExec.pre_exec at 0x7f3e7d84c620>
  pre_request: <function PreRequest.pre_request at 0x7f3e7d84c730>
  post_request: <function PostRequest.post_request at 0x7f3e7d84c7b8>
  child_exit: <function ChildExit.child_exit at 0x7f3e7d84c8c8>
  worker_exit: <function WorkerExit.worker_exit at 0x7f3e7d84c9d8>
  nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f3e7d84cae8>
  on_exit: <function OnExit.on_exit at 0x7f3e7d84cbf8>
  proxy_protocol: False
  proxy_allow_ips: ['127.0.0.1']
  keyfile: None
  certfile: None
  ssl_version: 2
  cert_reqs: 0
  ca_certs: None
  suppress_ragged_eofs: True
  do_handshake_on_connect: False
  ciphers: None
  raw_paste_global_conf: []
[2020-06-12 15:33:24 +0800] [10095] [INFO] Starting gunicorn 20.0.0
[2020-06-12 15:33:24 +0800] [10095] [DEBUG] Arbiter booted
[2020-06-12 15:33:24 +0800] [10095] [INFO] Listening at: http://0.0.0.0:5000 (10095)
[2020-06-12 15:33:24 +0800] [10095] [INFO] Using worker: gunicorn.workers.ggevent.GeventWorker
[2020-06-12 15:33:24 +0800] [10100] [INFO] Booting worker with pid: 10100
[2020-06-12 15:33:24 +0800] [10101] [INFO] Booting worker with pid: 10101
[2020-06-12 15:33:24 +0800] [10102] [INFO] Booting worker with pid: 10102
[2020-06-12 15:33:24 +0800] [10103] [INFO] Booting worker with pid: 10103
[2020-06-12 15:33:24 +0800] [10104] [INFO] Booting worker with pid: 10104
[2020-06-12 15:33:25 +0800] [10105] [INFO] Booting worker with pid: 10105
[2020-06-12 15:33:25 +0800] [10106] [INFO] Booting worker with pid: 10106
[2020-06-12 15:33:25 +0800] [10107] [INFO] Booting worker with pid: 10107
[2020-06-12 15:33:25 +0800] [10108] [INFO] Booting worker with pid: 10108
[2020-06-12 15:33:25 +0800] [10110] [INFO] Booting worker with pid: 10110
[2020-06-12 15:33:25 +0800] [10111] [INFO] Booting worker with pid: 10111
[2020-06-12 15:33:25 +0800] [10112] [INFO] Booting worker with pid: 10112
[2020-06-12 15:33:25 +0800] [10113] [INFO] Booting worker with pid: 10113
[2020-06-12 15:33:25 +0800] [10114] [INFO] Booting worker with pid: 10114
[2020-06-12 15:33:25 +0800] [10115] [INFO] Booting worker with pid: 10115
[2020-06-12 15:33:25 +0800] [10116] [INFO] Booting worker with pid: 10116
[2020-06-12 15:33:25 +0800] [10095] [DEBUG] 16 workers
[2020-06-12 15:36:32 +0800] [10100] [DEBUG] GET /
[2020-06-12 15:36:32,342] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/www/wwwroot/demo/venv/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/www/wwwroot/demo/app.py", line 9, in hello_world
    c = codecs.encode(b"sdf", "hex_codec")
LookupError: unknown encoding: hex_codec
[2020-06-12 15:36:32 +0800] [10100] [DEBUG] Closing connection.

7. Why?

Probably you end up running different Python versions than you say, but the question doesn't contain enough information to let us tell. — tripleee, Jun 12 '20 at 04:44
Naming your module `codecs` makes it shadow the built-in library with the same name. This is a common FAQ. — tripleee, Jun 12 '20 at 07:59
I only create a project named “codecs”, use other name, the situation is the same. — ddbb, Jun 12 '20 at 08:04
I’m sure my question is not the same question as the related post, because it works ok as root, don’t work by other user! The question is not at the code level, It may have something to do with some system feature that I don't know about. — ddbb, Jun 12 '20 at 08:47
Reopened; but please [edit] to clarify this (probably just rename the project to something which doesn't clash with any builtins or standard libraries). — tripleee, Jun 12 '20 at 08:50

score -1 · Answer 1 · answered Jun 13 '20 at 09:40

There are two ways to solve this problem

One:

from encodings import hex_codec

Two:

Manage multiple versions of Python by self, don’t use 3rd python manager program, like pyenv

I have this problem because I use pyenv , but why? I’d like to know. Maybe someone can answer my question.