Python virtual environment apparently not working as intended in production

Question

I have a python script that is running without a hitch in my development environment (PyCharm with venv) but as soon as I move it to a server, where it is meant to run as a service, it fails.

Excerpt from the code:

import configparser


config_file = "config.ini"
config = configparser.ConfigParser()
config.read(config_file)
mqtt_broker = config["mqtt_broker"]["address"]

The config.ini looks like this:

[mqtt_broker]
address = 10.0.0.10:1883
# maximum wait in milliseconds before raising an error
maximum_wait_ms = 3000

When I run it on the server (with user mqtt), I get:

$ sudo runuser -l mqtt -c "/etc/mqtt_kafka_bridge/venv/bin/python3.9 /etc/mqtt_kafka_bridge/venv/main.py --serve-in-foreground -log.level=debug"
Traceback (most recent call last):
  File "/etc/mqtt_kafka_bridge/venv/main.py", line 19, in <module>
    mqtt_broker = config["mqtt_broker"]["address"]
  File "/usr/lib/python3.9/configparser.py", line 963, in __getitem__
    raise KeyError(key)
KeyError: 'mqtt_broker'

Note: The whole virtual environment has been copied from the development environment onto the server. Moreover:

eric@ubuntu-server:~$ /etc/mqtt_kafka_bridge/venv/bin/python3 -m site
sys.path = [
    '/home/eric',
    '/usr/lib/python39.zip',
    '/usr/lib/python3.9',
    '/usr/lib/python3.9/lib-dynload',
    '/etc/mqtt_kafka_bridge/venv/lib/python3.9/site-packages',
]
USER_BASE: '/home/eric/.local' (exists)
USER_SITE: '/home/eric/.local/lib/python3.9/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

eric@ubuntu-server:~$ python3 -m site
sys.path = [
    '/home/eric',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/home/eric/.local/lib/python3.8/site-packages',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/home/eric/.local' (exists)
USER_SITE: '/home/eric/.local/lib/python3.8/site-packages' (exists)
ENABLE_USER_SITE: True

The server is running Ubuntu 20.04 LTS, which does not include Python3.9. This was installed later.

What am I not seeing? Thanks.

What the connection between the config file and the virtual environment? The code you've given is just reading a file from the CWD. Are you expecting the config file to be provided by package resources? The KeyError you're getting is because the `config.ini` file on the server doesn't have a `mqtt_broker` section. — Brian61354270, Aug 21 '23 at 18:17
_"Note: The whole virtual environment has been copied from the development environment onto the server."_ is generally a _really bad idea_. Virtual environments are not even intended to be moved around _on the same system_, much less coped to a different machine. — Brian61354270, Aug 21 '23 at 18:20
Did you try to check more directly whether the `config.ini` file can actually be found in production? In production, **exactly where** do you expect the code to look for this file, and **why**? Why do you expect it to exist in that location? What do you think should be the current working directory for the process, and why? — Karl Knechtel, Aug 21 '23 at 18:25
Virtual environments are not meant to be copied. Instead, you typically copy the *instructions* on how to re-create the virtual environment. One common way to do this is to create a `requirements.txt` and run `python -m pip install -r requirements.txt` in the server. — Niko Föhr, Aug 21 '23 at 18:37
Thanks for your comment, @karl-knechtel. Without adding the path to the script and related files in the script, I end up with the server user's home directory as my current working directory instead of the script's directory. Now, I have added the following to the script; I figure out where the script is by using os.path.dirname(os.path.abspath(__file__)), and then join the path with any filename I am using in the script. For instance, the configuration file would be config = os.path.join(path, "config.ini")- — ElToro1966, Aug 22 '23 at 15:45
Thank you, @niko-f%c3%b6hr and @brian61354270. Yes, you are right. Using freeze and requirements.txt is the right way. In this case, the “wrong way” worked (only needed to fix the path as set out above), but Ansible in combination with requirements.txt will be used going forward (https://docs.ansible.com/ansible/latest/collections/ansible/builtin/pip_module.html). — ElToro1966, Aug 22 '23 at 15:53
One thing I am still wondering about is why the script didn't fail on config_file = "config.ini". I mean, it didn't find the file... — ElToro1966, Aug 22 '23 at 15:57
As you have given the path of your config file as a *relative* path ("config.ini"), whether the script finds your config depends on your current working directory. You can either change your working directory with `os.chdir` or define your config with absolute path using something like `Path(__file__).resolve().parent / 'config.ini'` (or whatever is the right path to the ini from the .py file) — Niko Föhr, Aug 22 '23 at 16:09
"why the script didn't fail on config_file = "config.ini"." - that's a separate question, but the short version is that `configparser`, by design, silently skips files it can't find. The documentation explains the reasoning. — Karl Knechtel, Aug 22 '23 at 16:55

Python virtual environment apparently not working as intended in production

0 Answers0