0

Apologies if this is a really simple question, but I cannot find any information on Google to help.

I have an Amazon EC2 server set-up, which collects data which is streamed to the server by some scientific instruments and saves this as .csv files.

I want to access these .csv files from a Python script running on my (remote) laptop. Is this possible?

My understanding is that I would need my Python code to 'login' to the server and then download the file to a local directory.

Any help/pointers would be gratefully recieved.

Update 1

I installed paramiko and tried the code below

import paramiko
paramiko.util.log_to_file("D:/Temp/aws/paramiko.log")

# Open a transport
host,port = "##.##.###.##",##
transport = paramiko.Transport((host,port))

# Auth    
username,password = "username","password"
transport.connect(None,username,password)

But, i get the following error:

SSHException: Error reading SSH protocol banner

The log file looks like this:

DEB [20210202-17:54:47.282] thr=1   paramiko.transport: starting thread (client mode): 0x5202388
DEB [20210202-17:54:47.283] thr=1   paramiko.transport: Local version/idstring: SSH-2.0-paramiko_2.7.2
DEB [20210202-17:54:47.384] thr=1   paramiko.transport: Banner: 220-FileZilla Server 0.9.60 beta
DEB [20210202-17:54:47.384] thr=1   paramiko.transport: Banner: 220-written by Tim Kosse (tim.kosse@filezilla-project.org)
DEB [20210202-17:54:47.384] thr=1   paramiko.transport: Banner: 220 Please visit https://filezilla-project.org/
DEB [20210202-17:54:48.003] thr=1   paramiko.transport: Banner: 500 Syntax error, command unrecognized.
ERR [20210202-17:54:50.019] thr=1   paramiko.transport: Exception: Error reading SSH protocol banner
ERR [20210202-17:54:50.026] thr=1   paramiko.transport: Traceback (most recent call last):
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\transport.py", line 2211, in _check_banner
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:     buf = self.packetizer.readline(timeout)
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\packet.py", line 380, in readline
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:     buf += self._read_timeout(timeout)
ERR [20210202-17:54:50.027] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\packet.py", line 622, in _read_timeout
ERR [20210202-17:54:50.028] thr=1   paramiko.transport:     raise socket.timeout()
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: socket.timeout
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: 
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: During handling of the above exception, another exception occurred:
ERR [20210202-17:54:50.028] thr=1   paramiko.transport: 
ERR [20210202-17:54:50.029] thr=1   paramiko.transport: Traceback (most recent call last):
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\transport.py", line 2039, in run
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:     self._check_banner()
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:   File "C:\Users\caira\Anaconda3\lib\site-packages\paramiko\transport.py", line 2216, in _check_banner
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:     "Error reading SSH protocol banner" + str(e)
ERR [20210202-17:54:50.029] thr=1   paramiko.transport: paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
ERR [20210202-17:54:50.029] thr=1   paramiko.transport:

Am i doing something obvious that is wrong?

Should i be using the 'Public IPv4 DNS' instead? => ec2-##-##-###-##.us-east-2.compute.amazonaws.com

I am not sure if my username for above should be my full email address or my shorter username.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Cairan Van Rooyen
  • 283
  • 1
  • 9
  • **Suggestion:** It might be simpler to have a script on the EC2 instance that regularly copies the data to an Amazon S3 bucket (eg with the `aws s3 sync` command). Then, you can easily retrieve that data from another computer. – John Rotenstein Feb 02 '21 at 21:07
  • i have created an s3 bucket and got my ec2 to upload files to the bucket using the code: aws s3 cp C:\EltekGateway\details.txt s3://######-s3-bucket - but how do i make this code run every 24hrs for example? – Cairan Van Rooyen Feb 03 '21 at 21:10
  • On Linux, you can use `cron` to run the script on a regular basis. On Windows, you can use a Scheduled Task. – John Rotenstein Feb 03 '21 at 21:56

2 Answers2

1

It is possible you can use SCP or SFTP to connect to the server and copy the data to your local machine. There are many ways to do that but for your use case you can use a library like Paramiko to handle the connection and the file transfer, if you like a sample code look at the answer https://stackoverflow.com/a/3635163/2156106

Update: For EC2 instance, you need to use the private key you downloaded or provided while you created the instance as follows

import paramiko

key = paramiko.RSAKey.from_private_key_file("path_to_key.pem")

transport = paramiko.Transport((host, port))
transport.connect(username="username", pkey=key)

sftp = paramiko.SFTPClient.from_transport(transport)
sftp.get(remote_path , local_path)

The username should be the username on the server. if it is an ubuntu server it is ubuntu by default in EC2.

aymenim
  • 136
  • 3
  • Thanks. I have had an error and updated my original post with the error. SSHException: Error reading SSH protocol banner. Do you know why this might be? – Cairan Van Rooyen Feb 02 '21 at 18:57
0

you can use scp or sftp to do that (https://www.ssh.com/ssh/sftp/), but I'd strongly recommend sending those files to aws s3 from the server (or even stream it).

you could then use awscli (https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html) or python's boto3 (https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html) to copy the files from s3 to your local directory

Rne
  • 53
  • 6