Read remote file using python subprocess and ssh?

Question

How can I read data from a big remote file using subprocess and ssh?

Do you want to run programs on the remote server with SSH that need to access the file? Do you want to copy the file to your computer with SCP? Do you want Python to be able to read or write directly to the file? You need to give more details about what you're trying to do for anyone to be able to answer this question. — agf, Aug 21 '11 at 01:23
In my program I would like to list the files under a specific directory on a remote server. The user then selects to operate on one of these files. The files are quite big so I want to be able to read on the fly (to avoid memory issues) ie line by line. If the line meets a condition; do something with the line until end of file is reached. — sdfasdfasdf, Aug 21 '11 at 10:14

Ross Patterson · Answer 1 · 2014-11-29T01:52:40.460

11

import subprocess
ssh = subprocess.Popen(['ssh', 'user@host', 'cat', 'path/to/file'],
                       stdout=subprocess.PIPE)
for line in ssh.stdout:
    line  # do stuff

edited Nov 29 '14 at 01:52

answered Aug 21 '11 at 01:35

Ross Patterson

5,702
20
38

Ross, I have the exact same command that you suggest but that only reads the first line and does not read the other lines.. How do I make it read all the lines in the file? – user2921139 Nov 12 '14 at 00:29
I should say that by using the above you have, the file contents are read character by character.. how can i read a whole line completely? "split('\n')" does not seem to work.. – user2921139 Nov 12 '14 at 00:39
1

Remove `.readline()` after `ssh.stdout`, you want to loop on the output not the first line from it. – Elle Nov 21 '14 at 18:35
@JordanTrudgett Hmm, I submitted the edit when I said, but I can't find anything about it being rejected. Submitted it again. – Ross Patterson Nov 29 '14 at 01:53
@RossPatterson will this affect the servers bandwidth? – Haifeng Zhang Aug 12 '15 at 19:46
@haifzhan No more or less than any other way to transfer the file. – Ross Patterson Aug 13 '15 at 21:07
I am not getting any feedback - dont know if it can connect but not find the file or.. – CutePoison Nov 07 '19 at 10:35

score 1 · Answer 2 · answered Aug 21 '11 at 01:46

The answer above will work, but you'll have to setup your ssh login to use no password between your boxes. There are other ways to transfer files between computers using Python. A simple way, without authentication is to setup an apache server and use an http request.

score 0 · Answer 3 · edited May 23 '17 at 12:24

For performance improvement, which is important when the file is big, there is rsync. For more information about the exact improvement see following post and the answer from Rafa: How does `scp` differ from `rsync`?

The algorithm would then be the following using rsync

import subprocess

subprocess.Popen(["rsync", host-ip+'/path/to/file'],stdout=subprocess.PIPE)
for line in ssh.stdout:
    line  # do stuff

score -1 · Answer 4 · edited Nov 12 '14 at 01:54

-1

Use iter with readline to read each full line:

for i in iter(f.stdout.readline,"")

edited Nov 12 '14 at 01:54

Pokechu22

4,984
9
37
62

answered Nov 12 '14 at 01:36

user2921139

1,669
4
17
25

Read remote file using python subprocess and ssh?

4 Answers4

Linked