1

I have two python environments with different versions running in parallel. When I execute a python script (test2.py) from one python environment in the other python environment, I get the following error:

Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
  File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
    print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
                                                      ^
SyntaxError: invalid syntax

So my setup is this:

Python 3.7
 (test.py)
    │
    │                           Python 3.5.6
    ├───────────────────────────────┐
    ┆                               │
    ┆                         execute test2.py
    ┆                               │
    ┆                            Error

How can I fix this?

For -people: How can I execute a module with a different python version in Digital Micrograph?

Details

I have two python files.

File 1 (test.py):

# execute in Digital Micrograph
import os
import subprocess

command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe', 
           os.path.join(os.getcwd(), 'test2.py')]
print(*command)

result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("Subprocess result: '{}', '{}'".format(result.stdout.decode("utf-8"), result.stderr.decode("utf-8")))

and File 2 (test2.py)

# only executable in python 3.5.6
print("Hi")

in the same directory. test.py is executing test2.py with a different python version (python 3.5.6, legacy environment).

My python script (test.py) is running in the python interpreter in a third party program (Digital Micrograph). This program installs a miniconda python enviromnent called GMS_VENV_PYTHON (python version 3.7.x) which can be seen in the above traceback. The legacy miniconda environment is used only for running test2.py (from test.py) in python version 3.5.6.

When I run test.py from the command line (also in the conda GMS_VENV_PYTHON environment), I get the expected output from test2.py in test.py. When I run the exact same file in Digital Micrograph, I get the response

Subprocess result: '', 'Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\io.py", line 52, in <module>
  File "C:\ProgramData\Miniconda3\envs\GMS_VENV_PYTHON\lib\abc.py", line 147
    print(f"Class: {cls.__module__}.{cls.__qualname__}", file=file)
                                                      ^
SyntaxError: invalid syntax
'

This tells me the following (I guess):

  • The test2.py is called since this is the error output of the subprocess call. So the subprocess.run() function seems to work fine
  • The paths are in the GMS_VENV_PYTHON environment which is wrong in this case. Since this is test2.py, they should be in the legacy paths
  • There is a SyntaxError because a f-string (Literal String Interpolation) is used which is introduced with python 3.6. So the executing python version is before 3.6. So the legacy python environment is used.
  • test2.py uses either use io nor abc (I don't know what to conclude here; are those modules loaded by default when executing python?)

So I guess this means, that the standard modules are loaded (I don't know why, probably because they are always loaded) from the wrong destination.

How can I fix this? (See What I've tried > PATH for more details)

What I've tried so far

Encoding

I came across this post "Fatal Python error: Py_Initialize: can't initialize sys standard streams LookupError: unknown encoding: 65001" telling me, that there might be problems with the encoding. I know that Digital Micrograph internally uses ISO 8859-1. I tried to use python -X utf8 and python -X utf8 (test2.py doesn't care about UTF-8, it is ASCII only) as shown below. But neither of them worked

command = ['C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe', 
           "-X", "utf8=0", 
           os.path.join(os.getcwd(), 'test2.py')]

PATH

As far as I can tell, I think this is the problem. The answer "https://stackoverflow.com/a/31877629/5934316" of the post "PyCharm: Py_Initialize: can't initialize sys standard streams" suggests to change the PYTHONPATH.

So to specify my question:

  1. Is this the way to go?
  2. How can I set the PYTHONPATH for only the subprocess (while executing python with other libraries in the main thread)?
  3. Is there a better way to have two different python versions at the same time?

Thank you for your help.

Background

I am currently writing a program for handling an electron microscope. I need the "environment" (the graphical interface, the help tools but also hardware access) from Digital Micrograph. So there is no way around using it. And DigitalMicrograph does only support python 3.7.

On the other hand I need an external module which is only available for python 3.5.6. Also there is no way around using this module since it controlls other hardware.

Both rely on python C modules. Since they are compiled already, there is no possibility to check if they work with other versions. Also they are controlling highly sensitive aperatures where one does not want to change code. So in short words: I need two python versions parallel.

miile7
  • 2,547
  • 3
  • 23
  • 38

2 Answers2

2

I was actually quite close. The problem is, that python imports invalid modules from a wrong location. In my case modules were imported from another python installation due to a wrong path. Modifying the PYTHONPATH according to "https://stackoverflow.com/a/4453495/5934316" works for my example.

import os

my_env = os.environ.copy()
my_env["PYTHONHOME"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy"
my_env["PYTHONPATH"] = "C:\\ProgramData\\Miniconda3\\envs\\legacy;"
my_env["PATH"] = my_env["PATH"].replace("C:\\ProgramData\\Miniconda3\\envs\\GMS_VENV_PYTHON", 
                                        "C:\\ProgramData\\Miniconda3\\envs\\legacy")

command = ["C:\\ProgramData\\Miniconda3\\envs\\legacy\\python.exe", 
           os.path.join(path, "test2.py")]

result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)

For Digital Micrograph users: The python environment is saved in the global tags in "Private:Python:Python Path". So replace:

import DigitalMicrograph as DM

# ...

success, gms_venv = DM.GetPersistentTagGroup().GetTagAsString("Private:Python:Python Path")

if not success:
    raise KeyError("Python path is not set.")

my_env["PATH"] = my_env["PATH"].replace(gms_venv, "C:\\ProgramData\\Miniconda3\\envs\\legacy")
miile7
  • 2,547
  • 3
  • 23
  • 38
0

I had set "PYTHONPATH" as "D:\ProgramData\Anaconda3" for my python (base) python environment before, but i found when I had changed to another env my python still import basic python package from "D:\ProgramData\Anaconda3",which means it get the wrong basic package with the wrong "System environment variables" config.

so I delete "PYTHONPATH" from my windows "System environment variables", and that will be fixed.

tridu33
  • 11
  • 3