194

I want to find a bash command that will let me grep every file in a directory and write the output of that grep to a separate file. My guess would have been to do something like this

ls -1 | xargs -I{} "grep ABC '{}' > '{}'.out"

but, as far as I know, xargs doesn't like the double-quotes. If I remove the double-quotes, however, then the command redirects the output of the entire command to a single file called '{}'.out instead of to a series of individual files.

Does anyone know of a way to do this using xargs? I just used this grep scenario as an example to illustrate my problem with xargs so any solutions that don't use xargs aren't as applicable for me.

Jesse Shieh
  • 4,660
  • 5
  • 34
  • 49

4 Answers4

234

Do not make the mistake of doing this:

sh -c "grep ABC {} > {}.out"

This will break under a lot of conditions, including funky filenames and is impossible to quote right. Your {} must always be a single completely separate argument to the command to avoid code injection bugs. What you need to do, is this:

xargs -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}

Applies to xargs as well as find.

By the way, never use xargs without the -0 option (unless for very rare and controlled one-time interactive use where you aren't worried about destroying your data).

Also don't parse ls. Ever. Use globbing or find instead: http://mywiki.wooledge.org/ParsingLs

Use find for everything that needs recursion and a simple loop with a glob for everything else:

find /foo -exec sh -c 'grep "$1" > "$1.out"' -- {} \;

or non-recursive:

for file in *; do grep "$file" > "$file.out"; done

Notice the proper use of quotes.

lhunath
  • 120,288
  • 16
  • 68
  • 77
  • Upvoted but a doubt regd. not using `xargs` without `-0`: this only applies when you pipe `find`'s output with `xargs`, right? when I do `xargs -a ` how would I use this? Most commands like `grep` outputs with `\n` and not `\0.` The only way I can think of to work around this is to use `tr` again to fix that perhaps. But why is it important to use it only with `-0`? – legends2k Dec 26 '14 at 12:44
  • 4
    @legends2k because when you don't use `-0`, `xargs` will take your filenames and break all the spaces, quotes and backslashes in them. You should just forget about `xargs` as a tool. If you have lines, use a bash loop to iterate the lines: `while read line; do "$REPLY"; done < file-with-lines`, or `command | while ...` – lhunath Dec 27 '14 at 17:15
  • But when I try `$ xargs -a temp.txt -I{} echo *{}*` it prints `*billy tommy*` and `*catty*` i.e. it seems to break arguments only with `\n` as a delimiter. However, I'm using MSYS2 on Windows, so it may not be the behaviour on *nix. Also this behaviour may be only for the `-a` option and not for others, since the man page clearly says under `-0` _Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally)._ – legends2k Dec 27 '14 at 17:19
  • @legends2k `-a` doesn't even EXIST on my system, which says something about compatibility. Also, I never claimed `xargs` uses spaces as delimiters. I said it breaks the spaces in your input files. Try two spaces, leading spaces, trailing spaces, tabs, a quote in a line, etc. Eg. http://stuff.lhunath.com/temp.txt – lhunath Dec 28 '14 at 19:06
  • Perhaps specifying the right delimiter would address that concern: see [here](http://ideone.com/DPlwyV). – legends2k Dec 28 '14 at 19:35
  • @legends2k My `xargs` doesn't have a `-d`, so again, that is an incompatible option. See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html for what options are standard. Also, the *right* delimiter is NOT a newline, it is a NULL byte. You cannot correctly express a list of filenames as a list of lines because technically the WHOLE LIST of 517 lines could be 1 single file name (yes, file names can be multiple lines long). – lhunath Dec 29 '14 at 21:20
  • 1
    Wow, didn't know about that, thanks for the detail! So for portability (since not all `xargs` are GNU's), `xargs` needs to be avoided unless one can use it with `-0`. Thank you. – legends2k Dec 29 '14 at 23:37
  • 2
    Although I appreciate the detailed explanation for this specific use case, the question is about redirecting output of ```xargs```, which doesn't always involve parsing ```ls``` or using ```sh -c```. This doesn't answer the question in the slightest, but is the first google result for the question, only adding to the confusion. – pandasauce Jun 29 '17 at 09:27
  • 1
    @pandasauce read the first sentence of the question again. This is what was answered. In addition, the second code quote in the answer is exactly what you're looking for and is non-specific to ls. I've amended the answer a little to make it more clear, if that helps. – lhunath Jun 30 '17 at 12:33
  • 3
    @Ihunath, Hi, your answer works well for me. But could you give some detailed explanation or links about `xargs -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}`? Especially, the rules of embedded (double) quotes and the "--" symbol at the end. Thank you – Scott Yang Aug 07 '19 at 07:25
  • 2
    @lhunath There's actually nothing special about the -- symbol. When executing sh with a command, the parameters provided are assigned to $0, $1, etc. When executing sh with a script (e.g. ```sh test.sh```), the name of the script is assigned to $0 and the first parameter to $1 and so on. So, the same command saved in a script and executed ends up having all it's parameters shifted. Using the -- (or _, or turnip) acts as a throw-away variable to allow you to start parameters at offset 1 uniformly. The command could have been written without the -- and used $0 instead. – Derek Greer Feb 22 '21 at 13:46
  • 1
    The find command you've suggested does not execute, it appears to be waiting for the rest of command or something similar. – Alireza Mar 03 '21 at 04:55
  • @ScottYang The `-- {}` is passing `{}` as the first argument, then `$1` is referencing that first argument. You need the double quotes around `"$1"` for it to be expanded. – wisbucky Dec 04 '21 at 23:11
51

A solution without xargs is the following:

find . -mindepth 1 -maxdepth 1 -type f -exec sh -c "grep ABC '{}' > '{}.out'" \;

...and the same can be done with xargs, it turns out:

ls -1 | xargs -I {} sh -c "grep ABC '{}' > '{}.out'"

Edit: single quotes added after remark by lhunath.

Tudor Timi
  • 7,453
  • 1
  • 24
  • 53
Stephan202
  • 59,965
  • 13
  • 127
  • 133
  • He said he wants to use xargs. I posted a solution without it too, but deleted once I saw that he needed xargs. – Zifre May 10 '09 at 19:01
  • 1
    You're right. Reason I posted my answer was that it's better to have an alternative solution to get the job done than none at all. Turns out that it put me on the right track to find the desired answer (that is, the sh -c trick). – Stephan202 May 10 '09 at 19:03
15

I assume your example is just an example and that you may need > for other things. GNU Parallel http://www.gnu.org/software/parallel/ may be your rescue. It does not need additional quoting as long as your filenames do not contain \n:

ls | parallel "grep ABC {} > {}.out"

If you have filenames with \n in it:

find . -print0 | parallel -0 "grep ABC {} > {}.out"

As an added bonus you get the jobs run in parallel.

Watch the intro videos to learn more: http://pi.dk/1

The 10 seconds installation will try to do a full installation; if that fails, a personal installation; if that fails, a minimal installation:

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh

If you need to move it to a server, that does not have GNU Parallel installed, try parallel --embed.

Ole Tange
  • 31,768
  • 5
  • 86
  • 104
3

Actually, most of the answers here do not work with all filenames (if they contain double and single quotes), including the answer by lhunath and Stephan202.

This solution works with filenames with single and double quotes:

find . -mindepth 1 -print0 | xargs -0 -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}

Here's a test with filename with both single and double quotes:

echo ABC > "I'm here.txt"

# lhunath solution (hangs waiting for input)

$ find . -exec sh -c 'grep "$1" > "$1.out"' -- {} \;

# Stephan202 solutions

$ find . -mindepth 1 -maxdepth 1 -type f -exec sh -c "grep ABC '{}' > '{}.out'" \;
grep: ./Im: No such file or directory
grep: here.txt > ./Im here.txt.out: No such file or directory

$ ls -1 | xargs -I {} sh -c "grep ABC '{}' > '{}.out'"
xargs: unterminated quote

# this solution
$ find . -mindepth 1 -print0 | xargs -0 -I{} sh -c 'grep ABC "$1" > "$1.out"' -- {}

$ ls -1
"I'm here.txt"
"I'm here.txt.out"
wisbucky
  • 33,218
  • 10
  • 150
  • 101