0

I'm trying to use subprocess.Popen to run a check on Kafka consumer groups and log their state, but it doesn't appear to be waiting for all the commands to run. It isn't giving me any stdout, but its also returning an exit code of 0.

prompt = ["cd", "~/path/to/kafka_2.11-2.1.0;", "pwd;", "./bin/kafka-consumer-groups.sh", 
          "--bootstrap-server", "localhost:9092", "--describe", "--group", "groupname"]

response = subprocess.run(prompt, stdout=subprocess.PIPE, stderr=subprocess.PIPE, 
                          shell=True, check=True)
print(response)

Prints:

CompletedProcess(args=['cd', '~/path/to/kafka_2.11-2.1.0;', 'pwd;', './bin/kafka-consumer-groups.sh', '--bootstrap-server', 'localhost:9092', '--describe', '--group', 'groupname'], returncode=0, stdout=b'', stderr=b'')

The pwd command was to primarily test if it would return any kind of stout, it won't be kept.

I've looked through the docs for subprocess, and I haven't see anything that suggests that it is unable to capture multiple stdouts. Also, according to the logs, the CompletedProcess is returned in less than 10ms, while running cd ~path/to/kafka_2.11-2.1.0; pwd; ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group groupname takes about 10-15s on my machine.

Please note that I'm using python3.5.2

Victoria
  • 3
  • 2

2 Answers2

1

Your error is more fundamental than you think. Your code runs

sh -c 'cd'

with $0 set to the directory, $1 set to pwd;, etc; so very far from what you want. (Just cd simply switches to your home directory; then the shell exits, without doing anything with all those arguments you passed in, and Python continues back in whichever directory was current before you launched the subprocess.)

Generally, pass a single string as the first argument with shell=True, and a list of strings when you don't have a shell.

subprocess.run(r"cd foo; pwd; use shell commands to your heart\'s content; run as many processes as you like as subprocesses of your shell subprocess", shell=True)
subprocess.run(['/bin/echo', 'one', 'single', 'process', 'with', 'arguments])
tripleee
  • 175,061
  • 34
  • 275
  • 318
  • Cool, thanks. I was able to fix it with the method in my answer because I no longer needed shell=True, and since it comes with security risks, I just set it to False again. But thank you for the additional information, it helped my understanding a lot. – Victoria Oct 20 '20 at 19:27
0

I found my error. cd doesn't work with subprocess, but subprocess offers the cwd named argument that accepts the path you need to run arguments. The reason it was returning a CompletedProcess so quickly is that it was successfully changing directories and then exiting the subprocess.

Sorry, I wasn't thinking about cd being the culprit until I came across this question which answered my problem too.

Victoria
  • 3
  • 2
  • It "works" in the sense that the subprocess briefly switches to a subdirectory; but the way you call it, that's all it does before dying. See my answer for a somewhat fuller explanation. – tripleee Oct 20 '20 at 19:20