1

I have a program, which collects various log-files from multiple computers into a single ZIP-file on the client machine. It is currently using Fabric and I'd like to rewrite it with AsyncSSH to make all the servers send their files in parallel.

What I'm trying to figure out is getting the files over -- as streams. My current Fabric-based code is:

    result = io.BytesIO()
    connection.get(remote = path, local = result)
    result.seek(0)
    return result

What's the equivalent for when the connection is a result of asyncssh.connect()?

Mikhail T.
  • 3,043
  • 3
  • 29
  • 46
  • From what I see, [asyncssh.connect](https://asyncssh.readthedocs.io/en/latest/) returns an object which allows you to execute a command, e.g. `echo` in the example. So just execute a `cat` of the log file. You can fire multiple requests in parallel using asyncio.gathet or .wait (see [here](https://stackoverflow.com/questions/42231161/asyncio-gather-vs-asyncio-wait)) – Pynchia Dec 27 '20 at 06:50
  • 1
    I mean, for each source host you can do `result = io.StringIO(await conn.run('cat logfile")`, but in parallel using `gather` (assuming your logfiles are text and aren't huge) – Pynchia Dec 27 '20 at 06:58
  • I was hoping for some channel/stream kind of semantics -- so that, if a particular file is _large_, it does not need to be read fully into client's RAM, but can be read -- and compressed by `zipfile` -- _in chunks_... – Mikhail T. Dec 27 '20 at 19:58

1 Answers1

2

I think you're looking for create_process() instead of run(). That will return you an SSHClientProcess object that has stdin/stdout/stderr classes which are SSHReader and SSHWriter objects that behave similar to asyncio's StreamReader and StreamWriter.

Alternately, if "connection.get" there is actually doing an SFTP file transfer, I think you want to look into use open() on the SFTPClient object, rather than get(). That would give you a file object you could call read() on. With SFTP you normally want to schedule several reads from the same file in parallel to improve speed, though, and so the read() calls take an offset and a size rather than treating it like a stream that you sequentially access.

Ron Frederick
  • 21
  • 1
  • Thanks. Could you give some sample code for both options? – Mikhail T. Dec 27 '20 at 21:15
  • You can find an example of interactive input using create_process() at https://asyncssh.readthedocs.io/en/latest/#interactive-input. I'm still a bit unclear on exactly what you are trying to do, though, so it's hard to be more specific. I'm not really familiar with Fabric -- what exactly is the "connection.get" call doing? – Ron Frederick Dec 28 '20 at 23:00
  • If what you're doing is more file transfer related, https://asyncssh.readthedocs.io/en/latest/#sftp-client provides an example of how to retrieve a remove file into a local file in a single call. If you want to read data from that file interactively rather than writing it to disk on the local machine, you'd replace the "sftp.get" call with an "sftp.open" call and then call read(), seek(), or other such calls on the file object which open() returns. – Ron Frederick Dec 28 '20 at 23:06