1

In general, I can scp files from the remote server to my localhost one by one; however, I want to extract 200 files from a large remote folder, and it is not efficient to scp one by one. I wonder if there is a more elegant way to achieve this. This is what I did normally one by one: scp remote/Users/folder/file.txt local/Desktop

I have created a .txt file including filenames I wanted to scp.

Any comment would be appreciated!

Renaud Pacalet
  • 25,260
  • 3
  • 34
  • 51
M Shen
  • 51
  • 5
  • `cat file_name.txt | awk '{ printf "scp user@host_ip:/path/to/folder/%s .", $1}' | bash` you could use something like this one – Abu Hanifa Dec 14 '21 at 05:12
  • 2
    `rsync` using `--files-from=FILE` will transfer the list of files provided in `FILE`. Your list of files to transfer. Seen `man 1 rsync`. Pay attention to providing a relative or absolute file location on the remote host. – David C. Rankin Dec 14 '21 at 05:36
  • 1
    @AbuHanifa : This would be even more inefficient: It not only causes one `scp` invocation per file as well, but also an additional bash process. – user1934428 Dec 14 '21 at 08:47
  • @DavidC.Rankin : If you add `-z` to your _rsync_, I think your solution would be the most efficient of those discussed here. Perhaps it would be worth providing it as an answer. – user1934428 Dec 14 '21 at 09:11

3 Answers3

1

Do not use scp. Instead use ssh to run tar on the remote system to create an archive. Run another tar on the local system to extract the data.

ssh user@remote 'tar cf - folder' | tar xf -

Or with a list of files:

ssh user@remote 'tar cf - -T mylist.txt' | tar xf -

Add compression, if you like.

ssh user@remote 'tar cJf - -T mylist.txt' | tar xJf -
ceving
  • 21,900
  • 13
  • 104
  • 178
  • This does not take the file list into account. It just transfers the content of the folder. – user1934428 Dec 14 '21 at 08:48
  • @user1934428 [Add what ever option is necessary to `tar`.](https://stackoverflow.com/questions/8033857/tar-archiving-that-takes-input-from-a-list-of-files) – ceving Dec 14 '21 at 08:56
  • Good point. I never noticed the `-T` option of `tar`. However from my understanding (the OP is a bit unclear in this respect), the files in _mylist.txt_ are relative pathes, so we need some combination of folder and file list. With other words: The file list needs to be "preprocessed" somehow. See my answer (using a less elegant approach compared to yours), where I use `sed` to generate the correct pathes in the file list. – user1934428 Dec 14 '21 at 09:05
1

Provided that your txt file containing the files to transfer, does not contain filenames with spaces inside, this will transfer several files in a single invocation:

sed 's:^:remote/Users/folder/:' filelist.txt | xargs -i -r scp {} target_directory

where filelist.txt is your text file. If you do have files with spaces, there are two work-arounds, depending on how your file list is beeing produced:

  • If you create the filelist manually, prepend each space with a backslash, so that xargs takes the whole line as a single file.

  • If you use a program to create the list, ensure that each file name is terminated by a null byte (for instance find -print0 is doing this), and add the -0 option to xargs.

You can speed up things by asking xargs to use several processes in parallel. See the option -P.

user1934428
  • 19,864
  • 7
  • 42
  • 87
0

You could try grabbing them in parallel with GNU Parallel like this:

parallel -a FILELIST.TXT scp remote:{} $HOME/Desktop

Performance may be better or worse depending on the sizes of your files, your local and remote disk subsystems and your network.

In general, if the files are large in size and largely constant in content, you'll do better with rsync. If the files are very small, you'll probably do better with tar. If your network is "laggy", you may do better with parallel. If performance is important, it's best to measure and analyse.

Mark Setchell
  • 191,897
  • 31
  • 273
  • 432