How can I use xargs to copy files that have spaces and quotes in their names?

Question

I'm trying to copy a bunch of files below a directory and a number of the files have spaces and single-quotes in their names. When I try to string together find and grep with xargs, I get the following error:

find .|grep "FooBar"|xargs -I{} cp "{}" ~/foo/bar
xargs: unterminated quote

Any suggestions for a more robust usage of xargs?

This is on Mac OS X 10.5.3 (Leopard) with BSD xargs.

The GNU xargs error message for this with a filename containing a single quote is rather more helpful: "xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option". — Steve Jessop, Sep 27 '08 at 11:52
GNU xargs also has `--delimiter` option (`-d`). Try it with `\n` as the delimiter, This prevents `xargs` from separating lines with spaces into several words/arguments. — MattBianco, Feb 27 '17 at 09:48

godbyk · Accepted Answer · 2018-03-29T20:19:09.057

213

You can combine all of that into a single find command:

find . -iname "*foobar*" -exec cp -- "{}" ~/foo/bar \;

This will handle filenames and directories with spaces in them. You can use -name to get case-sensitive results.

Note: The -- flag passed to cp prevents it from processing files starting with - as options.

edited Mar 29 '18 at 20:19

answered Sep 27 '08 at 08:22

godbyk

8,359
1
31
26

-exec will work with any find, I've never understood why people use xargs (and just wait until you hit the xargs directory length limit!!) – Kendall Helmstetter Gelner Oct 02 '08 at 07:15
1

What is an 'xargs directory length limit'? Do you mean maximum command size? If yes, xargs is supposed to split its arguments in appropriate group sizes. – tzot Oct 14 '08 at 01:22
74

People use xargs because typically it's faster to call an executable 5 times with 200 arguments each time than to call it 1000 times with one argument every time. – tzot Oct 14 '08 at 01:23
12

The answer from Chris Jester-Young ought to be the "good answer" there... BTW this solution does not work if a filename begins with "-". At least, it needs "--" after cp. – Keltia Jan 23 '09 at 22:49
11

Speed example -- over 829 files, the "find -exec" method took 26 seconds while the "find -print0 | xargs --null" method tool 0.7 seconds. Significant difference. – Peter Porter Aug 27 '12 at 14:47
2

Using cp is OK, but not in such way - in this way, cp is run for every file, thus putting non-trivial overhead of fork+execve for every single file. It's better to use -t option of cp. Like: `find .. -exec cp -t ~/foo/bar -- {} +` – Jan 30 '13 at 11:25
7

@tzot A late comment but anyway, `xargs` is not required to address the issue you are describing, `find` already supports it with the `-exec` `+` punctuation. – jlliagre Jun 26 '13 at 16:57
Hi, if `foobar` is the output of some other command, how can I perform this ? Thank you very much – ying17zi May 21 '15 at 08:44
@ying17zi Replace `foobar` with either `$(your-command args)` or `\`your-command args\``. – godbyk May 24 '15 at 16:25
3

doesn't answer the question of how to deal with spaces – Ben Glasser Jan 11 '19 at 20:06
@BenGlasser Placing the `{}` within quotation marks protects the spaces within the filename/path. Is that what you're referring to? – godbyk Jan 12 '19 at 00:45
2

@godbyk no I mean that this answer still has not answered the question about how to handle spaces with xargs, which is the title of the question – Ben Glasser Jan 13 '19 at 23:30
2

@BenGlasser Ah, gotcha. See Chris' [answer](https://stackoverflow.com/a/143172/4214) for using `xargs` with spaces. – godbyk Jan 14 '19 at 03:22

score 122 · Answer 2 · edited Feb 02 '15 at 21:38

122

find . -print0 | grep --null 'FooBar' | xargs -0 ...

I don't know about whether grep supports --null, nor whether xargs supports -0, on Leopard, but on GNU it's all good.

edited Feb 02 '15 at 21:38

James McMahon

48,506
64
207
283

answered Sep 27 '08 at 07:15

C. K. Young

219,335
46
382
435

1

Leopard does support "-Z" (it is GNU grep) and of course find(1) and xargs(1) do support "-0". – Keltia Jan 23 '09 at 22:47
@Keltia Note that in OS X Mountain Lion (10.8) Apple replaced GNU grep with the BSD grep and the -{z|Z} switch doesn't work. – cosmix Nov 28 '12 at 18:14
1

On OS X 10.9 `grep -{z|Z}` means "behave as zgrep" (decompress) and not the intended "print a zero byte after each filename". Use `grep --null` to achieve the latter. – bassim Feb 07 '14 at 11:26
@ChrisJester-Young You're very welcome. Although now your comment regarding the "-z" option has lost its context ;) – bassim Feb 07 '14 at 11:47
4

What's wrong with `find . -name 'FooBar' -print0 | xargs -0 ...`? – Quentin Pradet May 17 '14 at 09:42
1

@QuentinPradet Obviously, for a fixed string like "FooBar", `-name` or `-path` work just fine. The OP has specified the use of `grep`, presumably because they want to filter the list using regular expressions. – C. K. Young May 17 '14 at 09:59
You may get a `Binary file (standard input) matches` from grep when you try this because of `-print0`; use `grep -a` to force it to interpret the pipe. – James Ko Jun 20 '15 at 21:43
Doesn't work with with `duperemove` — passing a 0-ended filename results in `No such file or directory`. – Hi-Angel Jul 30 '18 at 10:21
Ok, I figured, you're using xargs wrong. You should be using `xargs -d '\n'`, otherwise xargs would break files on spaces. Try this: `touch "foo bar"; echo -e "foo bar" | xargs -0 ls`, you'd get an error. But replacing the `-0` with `-d '\n'` works. [See also this answer](https://stackoverflow.com/a/33528111/2388257). – Hi-Angel Jul 30 '18 at 11:06
1

@Hi-Angel That's _exactly_ why I use `xargs -0` *in conjunction with* `find -print0`. The latter prints filenames with a NUL terminator and the former receives files that way. Why? Filenames in Unix can contain newline characters. But they cannot contain NUL characters. – C. K. Young Jul 30 '18 at 13:17
Use this if you need other arguments, e.g. target dir for cp: `... | xargs -0 -J % cp -v % /target/dir` – Phlogi Dec 31 '20 at 09:44

score 107 · Answer 3 · edited Nov 08 '16 at 09:03

107

The easiest way to do what the original poster wants is to change the delimiter from any whitespace to just the end-of-line character like this:

find whatever ... | xargs -d "\n" cp -t /var/tmp

edited Nov 08 '16 at 09:03

CupawnTae

14,192
3
29
60

answered Nov 04 '15 at 17:21

user87601

1,171
1
7
2

4

This anwser is simple, effective, and straight to the point : the default delimiter set for xargs is too broad and needs to be narrowed for what OP wants to do. I know this first-hand because I ran into this exact same issue today doing something similar, except in cygwin. Had I read the help for xargs command, I might have avoided a few headaches, but your solution fixed it for me. Thanks ! (Yeah, OP was on MacOS using BSD xargs, which I don't use, but I have hope that the xargs "-d" parameter exists in all versions). – Etienne Delavennat May 07 '16 at 19:18
8

Nice answer but not working on Mac. Instead we can pipe the *find* into `sed -e 's_$.*$_"\1"_g'` to force quotes around the file name – ishahak Aug 21 '16 at 11:08
10

This should be the accepted answer. The question was about using `xargs`. – Mohammad Alhashash Nov 06 '16 at 05:10
4

I get `xargs: illegal option -- d` – nehem Nov 11 '17 at 22:09
1

It's worth pointing out that filenames can contain a newline character on many *nix systems. You're unlikely to ever run into this in the wild, but if you're running shell commands on untrusted input this could be a concern. – Soren Bjornstad Jan 19 '19 at 03:30
1

(Example: Files `B` and `C` exist in a big directory. A user creates a new file named `B\nC`. Later, that user goes away and someone cleans all that user's files by querying the owners and piping a list of their files through `xargs -d "\n" rm`. Files `B` and `C` are promptly deleted.) – Soren Bjornstad Jan 19 '19 at 03:51
@nehem `-d` is a GNU option, so if you're on mac you'll need to install GNU tools – phuclv Aug 28 '22 at 15:20

score 78 · Answer 4 · answered Sep 29 '08 at 14:50

78

This is more efficient as it does not run "cp" multiple times:

find -name '*FooBar*' -print0 | xargs -0 cp -t ~/foo/bar

answered Sep 29 '08 at 14:50

Tometzky

22,573
5
59
73

1

This didn't work for me. It tried to cp ~/foo/bar into whatever you find, but not the opposite – Shervin Asgari Jun 21 '10 at 07:40
14

The -t flag to cp is a GNU extension, AFAIK, and isn't available on OS X. But if it were, it would work as shown in this answer. – metamatt May 18 '12 at 07:03
3

I'm using Linux. Thanks for the '-t' switch. That's what I was missing :-) – Vahid Pazirandeh Apr 21 '17 at 03:39
It works for me! +1 vote – Tung Aug 12 '22 at 10:32

oyouareatubeo · Answer 5 · 2012-08-22T06:08:25.153

64

I ran into the same problem. Here's how I solved it:

find . -name '*FoooBar*' | sed 's/.*/"&"/' | xargs cp ~/foo/bar

I used sed to substitute each line of input with the same line, but surrounded by double quotes. From the sed man page, "...An ampersand (``&'') appearing in the replacement is replaced by the string matching the RE..." -- in this case, .*, the entire line.

This solves the xargs: unterminated quote error.

edited Aug 22 '12 at 06:08

answered Aug 22 '12 at 05:57

oyouareatubeo

737
5
7

3

I'm on windows and using gnuwin32, so I had to use `sed s/.*/\"&\"/` to get it to work. – Pat Oct 16 '12 at 05:08
Yes but presumably this would not handle filenames with `"` in - unless sed also quotes quotes? – artfulrobot May 27 '15 at 08:56
Using `sed` is genius and for now the correct solution without rewriting the problem! – entonio Jun 03 '15 at 17:36

score 58 · Answer 6 · edited Jul 29 '18 at 20:51

58

This method works on Mac OS X v10.7.5 (Lion):

find . | grep FooBar | xargs -I{} cp {} ~/foo/bar

I also tested the exact syntax you posted. That also worked fine on 10.7.5.

edited Jul 29 '18 at 20:51

Peter Mortensen

30,738
21
105
131

answered Aug 28 '13 at 19:20

frediy

1,766
15
16

4

This works, but `-I` implies `-L 1` (so says the manual), which means the cp command is being run once per file = v slow. – artfulrobot May 27 '15 at 08:57
xargs -J % cp % Is possibly more efficient on OSX. – Walker D Jan 26 '16 at 05:43
3

Sorry, but this is WRONG. First it produces exactly the error the TO wanted to avoid. You must use `find ... -print0` and `xargs -0` to work arround xargs's "by default quotes are special". Second, generally use `'{}'` not `{}` in commands passed to xargs, to protect against spaces and special characters. – Andreas Spindler Mar 20 '16 at 12:13
3

Sorry Andreas Spindler, I am not that familiar with xargs and found this line after some experimentation. It seems to work for most people who have commented on it and upvoted it. Would you mind going into a bit more detail about what kind of error it produces? Also, would you mind posting the exact input you think would be more correct? Thank you. – frediy Mar 20 '16 at 23:06
This is the xargs part that worked on *MacOS* 10.15 for me: `xargs -0 -J % cp -v % /foo/bar` – Phlogi Dec 31 '20 at 09:37

jlliagre · Answer 7 · 2019-02-10T22:54:20.953

Just don't use xargs. It is a neat program but it doesn't go well with find when faced with non trivial cases.

Here is a portable (POSIX) solution, i.e. one that doesn't require find, xargs or cp GNU specific extensions:

find . -name "*FooBar*" -exec sh -c 'cp -- "$@" ~/foo/bar' sh {} +

Note the ending + instead of the more usual ;.

This solution:

correctly handles files and directories with embedded spaces, newlines or whatever exotic characters.
works on any Unix and Linux system, even those not providing the GNU toolkit.
doesn't use xargs which is a nice and useful program, but requires too much tweaking and non standard features to properly handle find output.
is also more efficient (read faster) than the accepted and most if not all of the other answers.

Note also that despite what is stated in some other replies or comments quoting {} is useless (unless you are using the exotic fishshell).

Why is it faster? [User tzot wrote: *"People use xargs because typically it's faster to call an executable 5 times with 200 arguments each time than to call it 1000 times with one argument every time."*](https://stackoverflow.com/questions/143171/how-can-i-use-xargs-to-copy-files-that-have-spaces-and-quotes-in-their-names#comment75541_143222) — Peter Mortensen, Jul 29 '18 at 20:48
@PeterMortensen You probably overlook the ending plus. `find` can do what `xargs` does without any overhead. — jlliagre, Jul 29 '18 at 21:24

score 9 · Answer 8 · answered Mar 20 '14 at 14:10

9

For those who relies on commands, other than find, eg ls:

find . | grep "FooBar" | tr \\n \\0 | xargs -0 -I{} cp "{}" ~/foo/bar

answered Mar 20 '14 at 14:10

Aleksandr Guidrevitch

418
6
7

1

Works but slow because `-I` implies `-L 1` – artfulrobot May 27 '15 at 09:00

score 8 · Answer 9 · answered Sep 27 '08 at 07:16

8

Look into using the --null commandline option for xargs with the -print0 option in find.

answered Sep 27 '08 at 07:16

Shannon Nelson

2,090
14
14

score 6 · Answer 10 · answered Feb 09 '12 at 22:23

6

find | perl -lne 'print quotemeta' | xargs ls -d

I believe that this will work reliably for any character except line-feed (and I suspect that if you've got line-feeds in your filenames, then you've got worse problems than this). It doesn't require GNU findutils, just Perl, so it should work pretty-much anywhere.

answered Feb 09 '12 at 22:23

mavit

619
6
13

Is it possible to have a line-feed in a filename? Never heard of it. – mtk May 16 '12 at 17:31
2

Indeed it is. Try, e.g., `mkdir test && cd test && perl -e 'open $fh, ">", "this-file-contains-a-\n-here"' && ls | od -tx1` – mavit May 17 '12 at 18:33
1

`|perl -lne 'print quotemeta'` is exactly what I have been looking for. Other posts here didn't help me because rather than `find` I needed to use `grep -rl` to vastly reduce the number of PHP files to only the malware-infected ones. – Marcos Jun 14 '13 at 14:58
perl and quotemeta are far more general than print0/-0 - thanks for the general solution to pipelining files with spaces – bmike Oct 16 '15 at 19:43

score 5 · Answer 11 · edited Jul 29 '18 at 20:33

I have found that the following syntax works well for me.

find /usr/pcapps/ -mount -type f -size +1000000c | perl -lpe ' s{ }{\\ }g ' | xargs ls -l | sort +4nr | head -200

In this example, I am looking for the largest 200 files over 1,000,000 bytes in the filesystem mounted at "/usr/pcapps".

The Perl line-liner between "find" and "xargs" escapes/quotes each blank so "xargs" passes any filename with embedded blanks to "ls" as a single argument.

score 4 · Answer 12 · answered Jun 28 '19 at 08:43

Frame challenge — you're asking how to use xargs. The answer is: you don't use xargs, because you don't need it.

The comment by user80168 describes a way to do this directly with cp, without calling cp for every file:

find . -name '*FooBar*' -exec cp -t /tmp -- {} +

This works because:

the cp -t flag allows to give the target directory near the beginning of cp, rather than near the end. From man cp:

   -t, --target-directory=DIRECTORY
         copy all SOURCE arguments into DIRECTORY

The -- flag tells cp to interpret everything after as a filename, not a flag, so files starting with - or -- do not confuse cp; you still need this because the -/-- characters are interpreted by cp, whereas any other special characters are interpreted by the shell.
The find -exec command {} + variant essentially does the same as xargs. From man find:

   -exec command {} +                                                     
         This  variant  of the -exec action runs the specified command on
         the selected files, but the command line is built  by  appending
         each  selected file name at the end; the total number of invoca‐
         matched  files.   The command line is built in much the same way
         that xargs builds its command lines.  Only one instance of  `{}'
         is  allowed  within the command, and (when find is being invoked
         from a shell) it should be quoted (for example, '{}') to protect
         it  from  interpretation  by shells.  The command is executed in
         the starting directory.  If any invocation  returns  a  non-zero
         value  as exit status, then find returns a non-zero exit status.
         If find encounters an error, this can sometimes cause an immedi‐
         ate  exit, so some pending commands may not be run at all.  This
         variant of -exec always returns true.

By using this in find directly, this avoids the need of a pipe or a shell invocation, such that you don't need to worry about any nasty characters in filenames.

Amazing find, I had no idea!!! " -exec utility [argument ...] {} + Same as -exec, except that ``{}'' is replaced with as many pathnames as possible for each invocation of utility. This behaviour is similar to that of xargs(1)." in the BSD implementation. — conny, Oct 30 '19 at 09:20

score 3 · Answer 13 · edited Jul 29 '18 at 20:53

3

With Bash (not POSIX) you can use process substitution to get the current line inside a variable. This enables you to use quotes to escape special characters:

while read line ; do cp "$line" ~/bar ; done < <(find . | grep foo)

edited Jul 29 '18 at 20:53

Peter Mortensen

30,738
21
105
131

answered Jan 01 '14 at 13:09

StackedCrooked

34,653
44
154
278

score 2 · Answer 14 · answered Oct 02 '08 at 07:00

2

Be aware that most of the options discussed in other answers are not standard on platforms that do not use the GNU utilities (Solaris, AIX, HP-UX, for instance). See the POSIX specification for 'standard' xargs behaviour.

I also find the behaviour of xargs whereby it runs the command at least once, even with no input, to be a nuisance.

I wrote my own private version of xargs (xargl) to deal with the problems of spaces in names (only newlines separate - though the 'find ... -print0' and 'xargs -0' combination is pretty neat given that file names cannot contain ASCII NUL '\0' characters. My xargl isn't as complete as it would need to be to be worth publishing - especially since GNU has facilities that are at least as good.

answered Oct 02 '08 at 07:00

Jonathan Leffler

730,956
141
904
1,278

2

GitHub or it didn't happen – Corey Goldberg Sep 26 '17 at 15:47
@CoreyGoldberg: I guess it didn't happen then. – Jonathan Leffler Sep 26 '17 at 15:48
POSIX `find` doesn't need `xargs` in the first place (and that was already true 11 years ago). – jlliagre Feb 10 '19 at 22:51

score 2 · Answer 15 · edited Jul 29 '18 at 20:59

2

For me, I was trying to do something a little different. I wanted to copy my .txt files into my tmp folder. The .txt filenames contain spaces and apostrophe characters. This worked on my Mac.

$ find . -type f -name '*.txt' | sed 's/'"'"'/\'"'"'/g' | sed 's/.*/"&"/'  | xargs -I{} cp -v {} ./tmp/

edited Jul 29 '18 at 20:59

Peter Mortensen

30,738
21
105
131

answered Oct 08 '15 at 01:58

Moises

21
2

score 1 · Answer 16 · edited Jul 29 '18 at 20:36

bill_starr's Perl version won't work well for embedded newlines (only copes with spaces). For those on e.g. Solaris where you don't have the GNU tools, a more complete version might be (using sed)...

find -type f | sed 's/./\\&/g' | xargs grep string_to_find

adjust the find and grep arguments or other commands as you require, but the sed will fix your embedded newlines/spaces/tabs.

score 1 · Answer 17 · answered Mar 27 '15 at 14:49

If find and xarg versions on your system doesn't support -print0 and -0 switches (for example AIX find and xargs) you can use this terribly looking code:

 find . -name "*foo*" | sed -e "s/'/\\\'/g" -e 's/"/\\"/g' -e 's/ /\\ /g' | xargs cp /your/dest

Here sed will take care of escaping the spaces and quotes for xargs.

Tested on AIX 5.3

score 1 · Answer 18 · edited Jul 29 '18 at 20:58

I played with this a little, started contemplating modifying xargs, and realised that for the kind of use case we're talking about here, a simple reimplementation in Python is a better idea.

For one thing, having ~80 lines of code for the whole thing means it is easy to figure out what is going on, and if different behaviour is required, you can just hack it into a new script in less time than it takes to get a reply on somewhere like Stack Overflow.

See https://github.com/johnallsup/jda-misc-scripts/blob/master/yargs and https://github.com/johnallsup/jda-misc-scripts/blob/master/zargs.py.

With yargs as written (and Python 3 installed) you can type:

find .|grep "FooBar"|yargs -l 203 cp --after ~/foo/bar

to do the copying 203 files at a time. (Here 203 is just a placeholder, of course, and using a strange number like 203 makes it clear that this number has no other significance.)

If you really want something faster and without the need for Python, take zargs and yargs as prototypes and rewrite in C++ or C.

score 1 · Answer 19 · edited Jul 29 '18 at 20:37

I used Bill Star's answer slightly modified on Solaris:

find . -mtime +2 | perl -pe 's{^}{\"};s{$}{\"}' > ~/output.file

This will put quotes around each line. I didn't use the '-l' option although it probably would help.

The file list I was going though might have '-', but not newlines. I haven't used the output file with any other commands as I want to review what was found before I just start massively deleting them via xargs.

Guenther Brunthaler · Answer 20 · 2018-03-17T18:09:17.520

I created a small portable wrapper script called "xargsL" around "xargs" which addresses most of the problems.

Contrary to xargs, xargsL accepts one pathname per line. The pathnames may contain any character except (obviously) newline or NUL bytes.

No quoting is allowed or supported in the file list - your file names may contain all sorts of whitespace, backslashes, backticks, shell wildcard characters and the like - xargsL will process them as literal characters, no harm done.

As an added bonus feature, xargsL will not run the command once if there is no input!

Note the difference:

$ true | xargs echo no data
no data

$ true | xargsL echo no data # No output

Any arguments given to xargsL will be passed through to xargs.

Here is the "xargsL" POSIX shell script:

#! /bin/sh
# Line-based version of "xargs" (one pathname per line which may contain any
# amount of whitespace except for newlines) with the added bonus feature that
# it will not execute the command if the input file is empty.
#
# Version 2018.76.3
#
# Copyright (c) 2018 Guenther Brunthaler. All rights reserved.
#
# This script is free software.
# Distribution is permitted under the terms of the GPLv3.

set -e
trap 'test $? = 0 || echo "$0 failed!" >& 2' 0

if IFS= read -r first
then
        {
                printf '%s\n' "$first"
                cat
        } | sed 's/./\\&/g' | xargs ${1+"$@"}
fi

Put the script into some directory in your $PATH and don't forget to

$ chmod +x xargsL

the script there to make it executable.

score 0 · Answer 21 · answered Jun 26 '13 at 16:02

0

You might need to grep Foobar directory like:

find . -name "file.ext"| grep "FooBar" | xargs -i cp -p "{}" .

answered Jun 26 '13 at 16:02

fred

9,663
3
24
34

1

Per the man page, `-i` is deprecated, and `-I` should be used instead. – Asclepius Aug 25 '14 at 16:02

score -1 · Answer 22 · edited Jul 27 '18 at 14:23

If you are using Bash, you can convert stdout to an array of lines by mapfile:

find . | grep "FooBar" | (mapfile -t; cp "${MAPFILE[@]}" ~/foobar)

The benefits are:

It's built-in, so it's faster.
Execute the command with all file names in one time, so it's faster.
You can append other arguments to the file names. For cp, you can also:
```
find . -name '*FooBar*' -exec cp -t ~/foobar -- {} +
```
however, some commands don't have such feature.

The disadvantages:

Maybe not scale well if there are too many file names. (The limit? I don't know, but I had tested with 10 MB list file which includes 10000+ file names with no problem, under Debian)

Well... who knows if Bash is available on OS X?

How can I use xargs to copy files that have spaces and quotes in their names?

22 Answers22

Linked