1

When using scp or rsync I often fail to deal with 'Argument list too long' error. When having to mv or rm, I have no problem to use find and xargs but I fail to understand how to use find and -exec despite all the SE posts on the subject. Consider the following issue...

I tried

$scp /Path/to/B* Me@137.92.4.152:/Path/to/

-bash: /usr/bin/scp: Argument list too long

So I tried

$find . -name "/Path/to/B*" -exec scp "{}" Me@137.92.4.152:/Path/to/ '\;'

find: -exec: no terminating ";" or "+"

so I tried

$find . -name "/Path/to/B*" -exec scp "{}" Me@137.92.4.152:/Path/to/ ';'

find: ./.gnupg: Permission denied
find: ./.subversion/auth: Permission denied

So I tried

$sudo find . -name "/Path/to/B*" -exec scp "{}" Me@137.92.4.152:/Path/to/ ';'

and nothing happen onces I enter my password

I am on Mac OSX version 10.11.3, Terminal version 2.6.1

mklement0
  • 382,024
  • 64
  • 607
  • 775
Remi.b
  • 17,389
  • 28
  • 87
  • 168
  • why don't you just copy the directory, not all files inside it? "scp - r" would do the job – RaphaMex Jun 09 '17 at 16:04
  • Sorry, my example was bad as I selected all the files. I changed the example to make explicit that I don't want to copy all the files. What I typically do now is that I `mkdir`, `find ... |xargs mv ..` the files, then `scp -r` the dir, `mv` the files back and `rm -r` the directory which is rather silly. – Remi.b Jun 09 '17 at 16:06

3 Answers3

2

EDIT after your update:

find "/Path/to" -maxdepth 1 -name "B*" -exec scp {} Me@137.92.4.152:/Path/to/ \;
RaphaMex
  • 2,781
  • 1
  • 14
  • 30
  • Oh... I silly I am. Just had to not put the whole path in the name. I now have to enter my password at every single file. Do you happen to also have a solution for that? – Remi.b Jun 09 '17 at 16:13
  • You can accept this answer and open another one for your new issue ;) – RaphaMex Jun 09 '17 at 16:15
2

R. Saban's helpful answer solves your primary problem:

  • -name only accepts a filename pattern, not a path pattern.

  • Alternatively, you could simply use the -path primary instead of the -name primary.

As for using as few invocations of scp as possible - each of which requires specifying a password by default:

  • As an alternative, consider bypassing the use of scp altogether, as suggested in Eric Renouf's helpful answer.

  • While find's -exec primary allows using terminator + in lieu of ; (which must be passed as ';' or \; to prevent the shell from interpreting ; as a command terminator) for passing as many filenames as will fit on a single command line (a built-in xargs, in a manner of speaking), this is NOT an option here, because use of + requires that placeholder {} come last on the command line, immediately before +.

  • However, since you're on macOS, you can use BSD xarg's nonstandard -J option for placing the placeholder anywhere on the command line, while still passing as many arguments as possible at once (using BSD find's nonstandard -print0 option in combination with xargs's nonstandard -0 option ensures that all filenames are passed as-is, even if they have embedded spaces, for instance):

find . -path "/Path/to/B*" -print0 | xargs -0 -J {} scp {} Me@137.92.4.152:/Path/to/

Now you will at most be prompted a few times: one prompt for every batch of arguments, as required to accommodate all arguments while observing the max. command-line length with the fewest number of calls possible.

mklement0
  • 382,024
  • 64
  • 607
  • 775
0

A solution that wouldn't require multiple scp connections (and therefore password entries) would be to tar on one side and untar on the other like:

find /Path/to -maxdepth 1 -name 'B*' -print0 | tar -c --null -T - | ssh ME@137.92.4.152 tar -x -C /Path/to

assuming your version of find supports -print0 and the like. It works by printing out null terminated list of files from find and telling tar to read its list of files from stdin (-T -) treating the list as null terminated (--null) and create a new archive (-c). By default, tar will write to stdout.

So then we'll pipe that archive to an ssh command to the target host. That will read the output of the previous command on its stdin, so we'll use tar there to extract (-x) the archive into the given directory (-C /Path/to)

Eric Renouf
  • 13,950
  • 3
  • 45
  • 67