How do I include a pipe | in my linux find -exec command?

Question

This isn't working. Can this be done in find? Or do I need to xargs?

find -name 'file_*' -follow -type f -exec zcat {} \| agrep -dEOE 'grep' \;

score 385 · Answer 1 · edited May 08 '14 at 16:08

385

the solution is easy: execute via sh

... -exec sh -c "zcat {} | agrep -dEOE 'grep' " \;

edited May 08 '14 at 16:08

Simon O'Hanlon

58,647
14
142
184

answered Nov 20 '08 at 22:40

flolo

15,148
4
32
57

27

What the OP was trying to accomplish can be met with the suggestions above, but this is the one which actually answers the question asked. There are reasons to do it this way - exec is a lot more powerful than just operating on the files returned by find, especially when combined with test. For instance: find geda-gaf/ -type d -exec bash -c 'DIR={}; [[ $(find $DIR -maxdepth 1 |xargs grep -i spice |wc -l) -ge 5 ]] && echo $DIR' \; Will return all directories in the search path which contain more than 5 lines total among all the files in that directory containing the word spice – swarfrat May 27 '12 at 19:00
6

Best answer. Grepping the whole output (as other answers suggest) isn't the same as grep each file. Tip: instead of sh, you can use any other shell you want (I tried that with bash and it's running ok). – pagliuca Nov 23 '12 at 18:15
5

Make sure to not overlook the `-c` option. Otherwise you will get a puzzling `No such file or directory` error message. – asmaier Dec 06 '13 at 12:22
here's a great ps replacement that makes use of find with piping inside an exec'd shell: /usr/bin/find /proc -mindepth 1 -maxdepth 1 -type d -regex '.*/[0-9]+' -print -exec bash -c "cat {}/cmdline | tr '\\0' ' ' ; echo" \; – parity3 Aug 02 '14 at 02:11
2

Example of finding files and renaming them with **sed** using regular expression `find -type f -name '*.mdds' -exec sh -c "echo {} | sed -e 's/_[0-9]\+//g' | xargs mv {}" \;` – Rostfrei Nov 04 '15 at 07:06
On AIX the {} inside the quotes is not replaced with the filename, therefore this solution doesn't work. – Roland Jan 18 '22 at 14:09
1

@Roland try to double quote the brackets and single quotes on the outer scope `-type f -exec sh -c 'echo "{}"' | grep something\;` – Pablo Bianchi Jun 22 '22 at 22:56
This doesn't guaranteed that whatever `{}` expands to is quoted properly for the shell. Better to pass `{}` as an argument, e.g. `-exec sh -c 'zcat "$1" | agrep -dEoE "grep" ' _ {} \;` (where `_` is just a dummy argument used to set `$0` in the shell that gets executed). – chepner Mar 21 '23 at 21:44

score 165 · Accepted Answer · edited Sep 18 '14 at 12:56

165

The job of interpreting the pipe symbol as an instruction to run multiple processes and pipe the output of one process into the input of another process is the responsibility of the shell (/bin/sh or equivalent).

In your example you can either choose to use your top level shell to perform the piping like so:

find -name 'file_*' -follow -type f -exec zcat {} \; | agrep -dEOE 'grep'

In terms of efficiency this results costs one invocation of find, numerous invocations of zcat, and one invocation of agrep.

This would result in only a single agrep process being spawned which would process all the output produced by numerous invocations of zcat.

If you for some reason would like to invoke agrep multiple times, you can do:

find . -name 'file_*' -follow -type f \
    -printf "zcat %p | agrep -dEOE 'grep'\n" | sh

This constructs a list of commands using pipes to execute, then sends these to a new shell to actually be executed. (Omitting the final "| sh" is a nice way to debug or perform dry runs of command lines like this.)

In terms of efficiency this results costs one invocation of find, one invocation of sh, numerous invocations of zcat and numerous invocations of agrep.

The most efficient solution in terms of number of command invocations is the suggestion from Paul Tomblin:

find . -name "file_*" -follow -type f -print0 | xargs -0 zcat | agrep -dEOE 'grep'

... which costs one invocation of find, one invocation of xargs, a few invocations of zcat and one invocation of agrep.

edited Sep 18 '14 at 12:56

Perception

79,279
19
185
195

answered Nov 20 '08 at 22:42

Rolf W. Rasmussen

1,909
1
12
9

2

Another advantage of xargs would be, that you can speed it with modern multi core cpu even more up, by using the -P switch (-P 0). – flolo Nov 20 '08 at 23:03
1

Yes, the -P swich is indeed a nice way to speed up execution in general. Unfortunately, you run the risk of the output of parallel zcat processes being piped into agrep interleaved, which would affect the result. This effect can be demonstrated using: echo -e "1\n2" | xargs -P 0 -n 1 yes | uniq – Rolf W. Rasmussen Nov 20 '08 at 23:57
@Adam, I've made your suggested change. – Paul Tomblin Nov 21 '08 at 15:09
for which you can install the splendid xjobs command (originally from Solaris) – sehe Apr 13 '11 at 22:26
What a beast the printf with sh. You rock. – Javier Diaz Jan 22 '13 at 11:29
7

A simpler and more general answer is at http://stackoverflow.com/a/21825690/42973: `-exec sh -c "… | … " \;`. – Eric O. Lebigot Apr 26 '15 at 01:04
while this is handy to know, it's not what the question asked. – Calvin Taylor Apr 03 '23 at 22:12

Paul Tomblin · Answer 3 · 2008-11-21T15:09:47.003

16

find . -name "file_*" -follow -type f -print0 | xargs -0 zcat | agrep -dEOE 'grep'

edited Nov 21 '08 at 15:09

answered Nov 20 '08 at 22:04

Paul Tomblin

179,021
58
319
408

Hoping to avoid -print and xargs for efficiency reasons. Maybe that's really my problem: find cannot handle piped commands through -exec – someguy Nov 20 '08 at 22:09
This doesn't work with files with spaces in their names; to fix, replace -print with -print0 and add the -0 option to xargs – Adam Rosenfield Nov 20 '08 at 22:44
2

@someguy - Wha? Avoiding xargs for efficiency reasons? Calling one instance of zcat, and passing it a list of multiple files, is *far* more efficient than exec-ing a new instance of it for each found file. – Sherm Pendley Nov 20 '08 at 22:44
@Adam - I've made your suggested change. 99% of the time when I'm doing finds, it's in my source code directories, and none of the files there have spaces so I don't bother with print0. Now my documents directory, on the other hand, I remember the print0. – Paul Tomblin Nov 21 '08 at 15:12
Would be a great answer with a little explanation about `-print0`, `xargs -0`, and filename with spaces. – Pablo Bianchi Jun 22 '22 at 22:54

simbo1905 · Answer 4 · 2015-02-18T22:15:32.450

You can also pipe to a while loop that can do multiple actions on the file which find locates. So here is one for looking in jar archives for a given java class file in folder with a large distro of jar files

find /usr/lib/eclipse/plugins -type f -name \*.jar | while read jar; do echo $jar; jar tf $jar | fgrep IObservableList ; done

the key point being that the while loop contains multiple commands referencing the passed in file name separated by semicolon and these commands can include pipes. So in that example I echo the name of the matching file then list what is in the archive filtering for a given class name. The output looks like:

/usr/lib/eclipse/plugins/org.eclipse.core.contenttype.source_3.4.1.R35x_v20090826-0451.jar /usr/lib/eclipse/plugins/org.eclipse.core.databinding.observable_1.2.0.M20090902-0800.jar org/eclipse/core/databinding/observable/list/IObservableList.class /usr/lib/eclipse/plugins/org.eclipse.search.source_3.5.1.r351_v20090708-0800.jar /usr/lib/eclipse/plugins/org.eclipse.jdt.apt.core.source_3.3.202.R35x_v20091130-2300.jar /usr/lib/eclipse/plugins/org.eclipse.cvs.source_1.0.400.v201002111343.jar /usr/lib/eclipse/plugins/org.eclipse.help.appserver_3.1.400.v20090429_1800.jar

in my bash shell (xubuntu10.04/xfce) it really does make the matched classname bold as the fgrep highlights the matched string; this makes it really easy to scan down the list of hundreds of jar files that were searched and easily see any matches.

on windows you can do the same thing with:

for /R %j in (*.jar) do @echo %j & @jar tf %j | findstr IObservableList

note that in that on windows the command separator is '&' not ';' and that the '@' suppresses the echo of the command to give a tidy output just like the linux find output above; although findstr is not make the matched string bold so you have to look a bit closer at the output to see the matched class name. It turns out that the windows 'for' command knows quite a few tricks such as looping through text files...

enjoy

score 5 · Answer 5 · edited Sep 16 '20 at 12:10

5

If you are looking for a simple alternative, this can be done using a loop:

for i in $(find -name 'file_*' -follow -type f); do
  zcat $i | agrep -dEOE 'grep'
done

or, more general and easy to understand form:

for i in $(YOUR_FIND_COMMAND); do
  YOUR_EXEC_COMMAND_AND_PIPES
done

and replace any {} by $i in YOUR_EXEC_COMMAND_AND_PIPES

edited Sep 16 '20 at 12:10

Clément Moulin - SimpleRezo

317
2
10

answered Oct 16 '19 at 11:15

Louis Gagnon

129
1
3

score 5 · Answer 6 · answered Nov 28 '19 at 06:24

5

I found that running a string shell command (sh -c) works best, for example:

find -name 'file_*' -follow -type f -exec bash -c "zcat \"{}\" | agrep -dEOE 'grep'" \;

answered Nov 28 '19 at 06:24

Andrew Khoury

408
3
9

Daniel Kaplan · Answer 7 · 2023-03-21T21:18:13.513

Here's what you should do:

find -name 'file_*' -follow -type f -exec sh -c 'zcat "$1" | agrep -dEOE "grep"' sh {} \;

I tried a couple of these answers and they didn't work for me. @flolo's answer doesn't work correctly if your filenames have special characters. According to this answer:

The find command executes the command directly. The command, including the filename argument, will not be processed by the shell or anything else that might modify the filename. It's very safe.

You lose that safety if you put the {} inside the sh command string.

There is a potential problem with @Rolf W. Rasmussen's answer. Yes, it handles special characters (as far as I know), but if the find output is too long, you won't be able to execute xargs -0 ...: there is a command line character limit set by the kernel and sometimes your shell. Coincidentally, every time I want to pipe commands from a find, I run into this limit.

But, they do bring up a valid point regarding the performance limitations. I'm not sure how to overcome that, though personally, I've never run into a situation where my example is too slow.

How do I include a pipe | in my linux find -exec command?

7 Answers7

Linked