Python - [Subprocess Popen Communicate] hangs on execution

Question

I am trying to run a simple command that initiates a port forward before the execution of my automated tests but it hangs every time.

The end goal was to setup the port forward, get the PID and terminate the port forward, at the end of the session.

I am on macOS and using Python 3.9.7 and trying to execute this inside of PyCharm IDE.

Here is the code snippet:

def setup_port_forward():

     # command
     command = 'kubectl port-forward api_service 8080:80 -n service_name'

     # shell
     shell_script = subprocess.Popen(command,
                                shell=True,
                                stdin=subprocess.PIPE,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE,
                                start_new_session=True)

     # extract
     try:
         stdout, stderr = shell_script.communicate(timeout=15)
         pid = int(stdout.decode().strip().split(' ')[-1])
     except subprocess.TimeoutExpired:
         shell_script.kill()

     yield

     # kill session
     os.kill(pid, signal.SIGTERM)

I don't pretend to know what this does or how it works, because I am still learning python.

Here's a few threads I have looked at:

Python Script execute commands in Terminal

python subprocess.Popen hanging

Python Script execute commands in Terminal

Python hangs when executing a shell script that runs a process as a daemon

https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate

Many threads say that using subprocess.PIPE in the shell script could cause issues, but again, on a different thread on how to get the PID, this was the method used.

I have tried using different ways as suggested in the different threads:

command = 'kubectl port-forward api_service 8080:80 -n service_name'

# 1
os.system(command)

# 2
subprocess.Popen(command).communicate

# 3
subprocess.run(command)

# 4
subprocess.call(command)

# 5
commands.getstatusoutput(command)

With all of them, they hang. Running this is terminal, it works fine.

Answer 1

The main problem here is with the communicate. You just want to Popen the process, then leave it running until you kill it.

You will definitely want to avoid shell=True when you can; see also Actual meaning of shell=True in subprocess

I don't see that the stdout and stderr redirections are useful either. If you just want to run it quietly, probably just redirect to and from subprocess.DEVNULL instead.

Creating a separate session seems dubious here; I would perhaps drop that too.

Running Bash commands in Python has some guidance for which subprocess method to prefer under what circumstances. TLDR is subprocess.run for situations where you want to wait for the process to finish, subprocess.Popen when you don't (but then understand your responsibilities around managing the object).

def setup_port_forward():
    proc = subprocess.Popen(
        ['kubectl', 'port-forward', 'api_service', '8080:80', '-n',  'service_name'],
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        start_new_session=True)
    yield
    # When done
    proc.kill()

The design with this function as a generator is also slightly weird; I would perhaps suggest you make this into a context manager instead.