Python piping output between two subprocesses

Question

I'm working on some code that will DD a block device over SSH, and I'm wanting to do this with subprocess so that I can monitor the status of DD during the transfer (killing the dd process with SIGUSR1 to get its current state, and reading that using selects).

The command that I'm trying to implement would be something like this:

dd if=/dev/sda | ssh root@example.com 'dd of=/dev/sda'

The current method I tried was:

dd_process = subprocess.Popen(['dd','if=/dev/sda'],0,None,None,subprocess.PIPE, subprocess.PIPE)  
ssh_process = subprocess.Popen(['ssh','root@example.com','dd of=/dev/sda'],0,None,dd_process.stdout)

However when I run this, the SSH process becomes defunct after 10-40 seconds.
Am I being completely obtuse here, or is there no way to pipe between subprocesses like this?

Edit: Turns out my real code didn't have the hostname in it. This is the correct way to do things.

I hope that you're using key based authentication or other non-interactive mechanism. — Cristian Ciupitu, Jan 31 '11 at 01:40
Obviously ;) I just took that ssh argument out to shorten the example — Philip J, Jan 31 '11 at 01:45
related: [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/q/295459/4279) — jfs, Jul 14 '15 at 23:28

score 34 · Accepted Answer · edited Jul 14 '15 at 23:31

34

from subprocess import Popen, PIPE
dd_process = Popen(['dd', 'if=/dev/sda'], stdout=PIPE)
ssh_process = Popen(['ssh', 'root@example.com', 'dd','of=/dev/sda'],stdin=dd_process.stdout, stdout=PIPE)
dd_process.stdout.close() # enable write error in dd if ssh dies
out, err = ssh_process.communicate()

This is way to PIPE the first process output to the second. (notice stdin in the ssh_process)

edited Jul 14 '15 at 23:31

jfs

399,953
195
994
1,670

answered Jan 31 '11 at 01:35

Senthil Kumaran

54,681
14
94
131

1

Well, that's what I already had, and it's still not working. The error specifically when ssh goes defunct is: ssh: dd of=/dev/sda: Temporary failure in name resolution – Philip J Jan 31 '11 at 01:43
1

@Cristian, sorry - It should be ssh_process. @Philip - The error has to do with your domain and internet connectivity (which you have given as example.com). Can you accomplish directly? That is without using python? – Senthil Kumaran Jan 31 '11 at 02:00
I only used example.com because I'm not going to post the public IP. I am able to complete the DD when not using python. – Philip J Jan 31 '11 at 02:08
@Philip. I would suggest, trying out just a simple remote command execution like 'ls' using subprocess, where you have stdout=PIPE and stderr=PIPE and see that it is working. Temp failure in name resolution means that when python is trying to do ssh, the domain is not getting resolved. – Senthil Kumaran Jan 31 '11 at 02:17

score 5 · Answer 2 · edited Dec 08 '21 at 19:49

5

The sh Python library makes it easy to call OS commands and pipe them.

From the documentation:

for line in tr(tail("-f", "test.log", _piped=True), "[:upper:]", "[:lower:]", _iter=True):
    print(line)

edited Dec 08 '21 at 19:49

temparus

119
3
10

answered Mar 13 '13 at 02:20

Apalala

9,017
3
30
48

1

That `sh` module is really nice. Thanks for sharing. – mpontillo Jul 08 '16 at 00:04
2

But it's dangerous to use with unchecked input – LtWorf May 08 '17 at 13:29

Python piping output between two subprocesses

2 Answers2

Linked

Related