Why are PATH variables not effecting subprocess.Popen in Windows

Question

I am trying to execute a .jar file from a Python program using subprocess.Popen and noticed that I'm not able to link to the desired version of Java without specifying the full path. For this particular application, I feel Popen is the right choice, since I would like to keep track of the process and ensure it goes away when I'm done with it, but I am open to other options.

In the code below JAVA_HOME, sys.path and os.environ['PATH'] are all set in order to make sure the proper version of Java executable is found. If the full path is not specified, Java complains "Error: could not find java.dll" and other messages seen below in the example output. Furthermore, java.dll is in the C:\Program Files\Java\jdk1.8.0_221\bin folder. Clearly, when the full path is specified it finds these variables (although the message goes to STDERR?).

My question is: why aren't my PATH variables influencing Popen?

import subprocess
import sys
import os

# check for any references of java in os.environ
found = False
for k,v in os.environ.items():
    if 'java' in v.lower():
        print(f'java reference found in: {k}={v}')
        found = True
      
if not found:
    print('no java referenes found in os.environ')
print()


# Correct java version inserted into system path as first in the path
sys.path.insert(0, 'C:\\Program Files\\Java\\jdk1.8.0_221\\bin')
my_env = os.environ.copy()
# environ PATH set to find JDK 1.8 first
my_env["PATH"] = 'C:\\Program Files\\Java\\jdk1.8.0_221\\bin;' + my_env["PATH"]
# JAVA_HOME set and bin folder here has java.dll
my_env["JAVA_HOME"] = '"C:\\Program Files\\Java\\jdk1.8.0_221\\bin"'

# looking at where java.exe is found from Popen on Windows
p = subprocess.Popen(['where', 'java.exe'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
stdout,stderr = p.communicate()
print('where java.exe')
print('STDOUT: ' + stdout.decode('utf-8'), end='')
print('STDERR: ' + stderr.decode('utf-8'))
print()

# java.exe -version command does not work
p = subprocess.Popen(['java.exe', '-version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
stdout,stderr = p.communicate()
print('java.exe -version')
print('STDOUT: ' + stdout.decode('utf-8'))
print('STDERR: ' + stderr.decode('utf-8'))

# however, when full path is specified it works
p = subprocess.Popen(['C:\\Program Files\\Java\\jdk1.8.0_221\\bin\\java.exe', '-version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=my_env)
stdout,stderr = p.communicate()
print('C:\\Program Files\\Java\\jdk1.8.0_221\\bin\\java.exe -version')
print('STDOUT: ' + stdout.decode('utf-8'))
print('STDERR: ' + stderr.decode('utf-8'))

# I can see the output here is the correct version
os.system('java.exe -version')

produces the output:

no java referenes found in os.environ

where java.exe
STDOUT: C:\Program Files\Java\jdk1.8.0_221\bin\java.exe
C:\Windows\System32\java.exe
STDERR: 

java.exe -version
STDOUT: 
STDERR: Error: Registry key 'Software\JavaSoft\Java Runtime Environment'\CurrentVersion'
has value '1.8', but '1.7' is required.
Error: could not find java.dll
Error: Could not find Java SE Runtime Environment.

C:\Program Files\Java\jdk1.8.0_221\bin\java.exe -version
STDOUT: 
STDERR: java version "1.8.0_221"
Java(TM) SE Runtime Environment (build 1.8.0_221-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

I'm using Python 3.8.8 on Windows 10. Trying to link to Java 1.8.0_221. I'm sure I am doing something horribly wrong here but not sure what.

UPDATE: I have tried the following additional solutions from the comment section since posting.

1. Used Popen env argument with a copy of os.environ

2. Added quotes around java path ex:

   my_env["PATH"] = '"C:\\Program Files\\Java\\jdk1.8.0_221\\bin";' + my_env["PATH"] - based on some testing I don't think this needs to be done at least for os.envirion['PATH'].

3. Removed all references of Java from the Windows system PATH variables in the Windows GUI. There is even a check at the beginning of the code to verify Java references are not in os.environ.

Still no luck.

Answer 1

Thanks to the comment by VGR, I took a more studied approach to the big red warning box on in the subprocess documentation here which states.

For Windows, see the documentation of the lpApplicationName and lpCommandLine parameters of WinAPI CreateProcess, and note that when resolving or searching for the executable path with shell=False, cwd does not override the current working directory and env cannot override the PATH environment variable. Using a full path avoids all of these variations.

I guess I did not fully appreciate the last sentence, which says in Windows the PATH does not get overridden and the full path to the executable should be used to avoid confusion between executable names. This behavior explains what I am seeing.