0

I'm computing smaller things on a AWS EC2 gpu-instance. Currently I have to find the right parameters, so there are shorter computations I have to analyze and then adjust some parameter to compute again.

For the analysis, I depend on things, which aren't available on my EC2 instance. So I'm looking for an convenient way to transfer small amounts of data (not bigger than 2MB) from an EC2 instance directly onto my local computer. I'm looking for a way, which interferes with my workflow as little as possible, so that I can concentrate on analyzing the data and updating the parameters. Currently I use the terminal and do it manually via SCP but there has to be a better way-

I_like_foxes
  • 1,179
  • 1
  • 7
  • 12

2 Answers2

1

If you have scp access to your systems then the paramiko library can perform the transfers for you, as detailed in this answer. There's an scp library that works well with it.

For fuller automation over ssh links consider fabric, built to handle single- and multi-system automation tasks.

Community
  • 1
  • 1
holdenweb
  • 33,305
  • 7
  • 57
  • 77
0

You can use tar for your files.

tar cvzf - -T list_of_filenames | ssh -i ec2key.pem ec2-user@hostname tar xzf -

You can also use recursive scp.

FallAndLearn
  • 4,035
  • 1
  • 18
  • 24