Where is the connection between the two Python venv environments

Question

I have two python environments and somehow there is some kind of link between them.

/home/testapi/API25/env is the original venv /home/preprodapi/API25/env was created by means of a cpio copy from the first. This worked find for many months. But now a problem has cropped up.

The symptom is that in preprodapi the pytz package isnt able to find the timezone Africa/Johannesburg (and presumably others), as evidenced by the stack trace:

 Traceback (most recent call last):
   File "/home/preprodapi/API25.8512/validator/echo.py", line 244, in jsonified_wrapper
    response_obj = request_handler(*args, **kwargs)
   File "/home/preprodapi/API25.8512/validator/echo.py", line 478, in bearer_token_wrapper
    return request_handler(*args, **kwargs)
   File "/home/preprodapi/API25.8512/validator/echo.py", line 1068, in globaldb_connection_wrapper
    return request_handler(*args, **kwargs)
   File "/home/preprodapi/API25.8512/validator/echo.py", line 569, in get_school_wrapper
    return request_handler(*args, **kwargs)
   File "/home/preprodapi/API25.8512/validator/echo.py", line 697, in school_admin_wrapper
    return request_handler(*args, **kwargs)
   File "/home/preprodapi/API25.8512/routehandlers.py", line 4035, in email_report
    do_email_report(kwargs.get('reportid'), json_attrs, getctx_school().get('schoolname'))
   File "/home/preprodapi/API25.8512/validator/echo.py", line 1155, in do_email_report
    tz = pytz.timezone(sender.school.get("local_timezone"))
   File "/home/testapi/API25/env/lib64/python3.5/site-packages/pytz/__init__.py", line 181, in timezone
 pytz.exceptions.UnknownTimeZoneError: 'Africa/Johannesburg'

Notice how it switches from /home/preprodapi/.... to /home/testapi/... in the final item.

But WHY is this happening?

(env) [root@ip-172-31-8-200 API25]# deactivate
[root@ip-172-31-8-200 API25]# pwd
/home/preprodapi/API25
[root@ip-172-31-8-200 API25]# . env/bin/activate
(env) [root@ip-172-31-8-200 API25]# pip uninstall pytz
Uninstalling pytz-2017.2:
  Would remove:
    /home/testapi/API25/env/lib/python3.5/site-packages/pytz-2017.2.dist-info/*
    /home/testapi/API25/env/lib/python3.5/site-packages/pytz/*
Proceed (y/n)? n
(env) [root@ip-172-31-8-200 API25]# python
Python 3.5.5 (default, Feb  6 2018, 10:57:32) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytz
>>> print(pytz.timezone('Africa/Johannesburg'))
Africa/Johannesburg

For the record I can't find any soft links between the venv's

(env) [root@ip-172-31-8-200 API25]# pwd
/home/preprodapi/API25
(env) [root@ip-172-31-8-200 API25]# find env -type l -ls
40549590    0 lrwxrwxrwx   1 root     root            3 Feb  4 12:54 env/lib64 -> lib
40549592    0 lrwxrwxrwx   1 root     root           15 Feb  4 12:54 env/bin/python3.5m -> /bin/python3.5m
40549593    0 lrwxrwxrwx   1 root     root           10 Feb  4 12:54 env/bin/python -> python3.5m
40549594    0 lrwxrwxrwx   1 root     root           10 Feb  4 12:54 env/bin/python3 -> python3.5m

Help please!

P.S note /home/preprodapi/API25.8512 is a cpio copy of /home/preprodapi/API25. I get the exact same results when I test in the API25.8512 sub-directory

Note #2: The same does not happen with another venv on this host

[root@ip-172-31-8-200 API25.8512]# cd /home/apiuser
[root@ip-172-31-8-200 apiuser]# cd API25
[root@ip-172-31-8-200 API25]# . env/bin/activate
(env) [root@ip-172-31-8-200 API25]# pip uninstall pytz
Uninstalling pytz-2018.9:
  Would remove:
    /home/apiuser/API25/env/lib/python3.5/site-packages/pytz-2018.9.dist-info/*
    /home/apiuser/API25/env/lib/python3.5/site-packages/pytz/*
Proceed (y/n)? n

What's a "cpio copy"? `cpio` is an archiver, not a copy utility. — ivan_pozdeev, Feb 04 '19 at 20:45
Could you upgrade your examples to a [mcve]? Without the means to reproduce the problem, it's impossible to say with confidence what the issue is. E.g. will the same happen if you create a new `venv`, install some packages into it, then copy it? — ivan_pozdeev, Feb 04 '19 at 20:48
What's the value of `pytz.__file__` after you "uninstalled" `pytz` from the current virtual env? — chepner, Feb 04 '19 at 20:53
What's `sys.path` and `PYTHONPATH`? Are you sure you're inside the right venv when the first error happens? — ivan_pozdeev, Feb 05 '19 at 00:17

ivan_pozdeev · Accepted Answer · 2019-02-05T00:35:37.003

You need to check your sys.path and find the source of anomalies, if any. See Debugging modifications of sys.path for a way to track changes to sys.path and Can I zip all the python standard libs and the python still able to import it? for how it's constructed.

venv is implemented via Py3's stock site.py:

If a file named "pyvenv.cfg" exists one directory above sys.executable,
sys.prefix and sys.exec_prefix are set to that directory and
it is also checked for site-packages (sys.base_prefix and
sys.base_exec_prefix will always be the "real" prefixes of the Python
installation). If "pyvenv.cfg" (a bootstrap configuration file) contains
the key "include-system-site-packages" set to anything other than "false"
(case-insensitive), the system-level prefixes will still also be
searched for site-packages; otherwise they won't.

When creating a venv, python and a number of other files are copied into <venv>/bin (<venv\Scripts in Windows), and a pyvenv.cfg is placed into <venv> for site.py to find upon Python startup. activate prepends <venv>/bin to PATH so that the local executable is started instead of the system one when you type "python".

Ultimately, this results in a sys.path that combines system-wide standard library with venv-specific 3rd-party modules. It would look something like this:

>>> sys.path
['', '<venv>/bin/python36.zip', <system Python platlib>, <system Python purelib>, '<venv>', '<venv>/lib/site-packages']

So, normally, there shouldn't be folders from another venv in sys.path resulting directly from venv logic. They may result from PYTHONPATH, or from some .pth files, or even from your own code. The above diagnostics should show where they come from.

It appears that the solution is when one clones a Python application, exclude the venv, re-create it, and re-install those packages from a pip freeze > requirements file from the source. I don't have time to test this properly right now but it makes complete sense based on my findings so I'm awarding the solution. — The Tahaan, Feb 05 '19 at 04:32

Where is the connection between the two Python venv environments

1 Answers1