13

Consider:

command1 | command2

Is the output of command1 used as standard input of command2 or as command line arguments to command2?

For example,

cat test.sh | grep "hehe"

What is its equivalent form without using a pipe?

I tried

grep "hehe" $(cat test.sh)

and it seems not to be correct.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Tim
  • 1
  • 141
  • 372
  • 590
  • 1
    Your question is a little confusing - hopefully either input redirection or simply supplying the filename as argument is what you wanted. On the other hand, if you're asking how to take the stdout of one command and give it to another command's stdin without using a pipe... that's the definition of a pipe. – Cascabel Oct 01 '09 at 16:18

4 Answers4

16
grep "hehe" < test.sh

Input redirection - works for a single file only, of course, whereas cat works for any number of input files.


Consider the notations:

grep "hehe" $(cat test.sh)
grep "hehe" `cat test.sh`

These are equivalent in this context; it is much easier to use the '$(cmd)' notation in nested uses, such as:

x=$(dirname $(dirname $(which gcc)))
x=`dirname \`dirname \\\`which gcc\\\`\``

(This gives you the base directory in which GCC is installed, in case you are wondering.)

In the grep example, what happens is that the contents of test.sh is read and split into white-space separated words, and each such word is provided as an argument to grep. Since grep treats the words after "hehe" (where grep, of course, does not see the double quotes - and they are not needed in this case; as a general rule, use single quotes rather than double quotes, especially around complex strings like regular expressions which often use shell metacharacters)... As I was saying, grep treats the words after "hehe" as file names, and tries to open each file, usually failing dismally because the files do not exist. This is why the notation is not appropriate in this context.


After revisiting the question, there is more that could be said - that hasn't already been said.

First off, many Unix commands are designed to work as 'filters'; they read input from some files, transform it in some way, and write the result onto standard output. Such commands are designed for use within command pipelines. Examples include:

  • cat
  • grep
  • troff and relatives
  • awk (with caveats)
  • sed
  • sort

All these filters have the same general behaviour: they take command line options to control their behaviour, and then they either read the files specified as command line arguments or, if there are no such arguments, they read their standard input. Some (like sort) can have options to control where their output goes instead of standard output, but that is relatively uncommon.

There are a few pure filters - tr is one such - that strictly read standard input and write to standard output.

Other commands have different behaviours. Eric Raymond provides a taxonomy for command types in "The Art of UNIX Programming".

Some commands generate lists of file names on standard output - the two classics are ls and find.

Sometimes, you want to apply the output from a file name generator as command line arguments for a filter. There's a program that does that automatically - it is xargs.

Classically, you would use:

find . -name '*.[chyl]' | xargs grep -n magic_name /dev/null

This would generate a complete list of files with the extensions '.c', '.h', '.y' and '.l' (C source, headers, Yacc and Lex files). As the list is read by xargs, it would create command lines with grep -n magic_name /dev/null at the start and each word (separated by white space) as an argument.

In the old days, Unix file names didn't include spaces. Under the influence of Mac and Windows, such spaces are now common-place. The GNU versions of find and xargs have complementary options to deal with this problem:

find . -name '*.[chyl]' -print0 | xargs -0 grep -n magic_name /dev/null

The '-print0' option means "print file names terminated by a NUL '\0'" (because the only characters that cannot appear in a (simple) file name are '/' and NUL, and obviously, '/' can appear in path names). The corresponding '-0' tells xargs to look for names terminated by NUL instead of space separated names.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
6

Another form of redirection is process substitution.

grep "hehe" <(cat test.sh)

is equivalent to:

grep "hehe" test.sh

which both look at the contents of test.sh itself.

While, as it has been noted, this command:

grep "hehe" $(cat test.sh)

looks for filenames in test.sh and uses them as arguments for grep. So if test.sh consists of:

scriptone
scripttwo

then grep is going to look for "hehe" in the contents of each of those files.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • 2
    I don't like your use of 'is equivalent' on line 3: `grep "hehe" test.sh` is, ultimately, using the contents of test.sh as stdin whereas `grep "hehe" <(cat test.sh)` is using the output of the command `cat test.sh` as stdin. Consider the difference in results if you run the commands as follows: `grep -H "hehe" <(cat test.sh)` and `grep -H "hehe" test.sh` – Isaac Kleinman Jan 02 '13 at 18:31
2

What is the equivalent of a bash pipe using command line arguments?

Pipes and command line arguments are different forms of input that are not interchangeable. If a program allows you to have equivalent forms of both, that is the choice of that program alone. (In source code, command line arguments appear as text in a variable, while pipes appear as open files, including stdin and stdout. Bash I/O redirection syntax, as used here lateron, technically does not belong to command line arguments, even though written right next to them on the command line …)

But let's be pedantic and also answer this:

What is the equivalent of a bash pipe without using a bash pipe character?

Answer: cat test.sh | grep "hehe" is equivalent to

grep "hehe" < <(cat test.sh)

Explanations:

  • Pipes redirect stdout of one command to stdin of another. To set the source of stdin, we can use input redirection ( < …) instead of using the pipe character.

  • However, just using input redirection (grep "hehe" < test.sh) is not the equivalent to pipes because it uses a file as the source for stdin, while pipes use the output a command (cat test.sh). So in addition, we add process substitution <(…) to replace input from a file to stdin with input from a command to stdin.

  • Of course, our example here is confusing because the two variants have the same effects:

      grep "hehe" < test.sh
      grep "hehe" < <(cat test.sh)
    

    But technically, input from stdin from a file is still a different mechanism than input from stdin from the output of a command that gets its input from a file.

  • For an even more detailed explanation, I recommend two other answers: here and here.

Source: Advanced Bash Scripting Manual, section on process substitution (start reading at "Some other usages").

tanius
  • 14,003
  • 3
  • 51
  • 63
  • Can a script differentiate between input from a file and input from a command? (and if yes, how?) And could you please add an example where the two variants don't have the same effect? – winklerrr Jul 01 '21 at 09:47
  • 1
    @winklerrr: "Input from a file" vs. "input from the output of a command": that's a bit sloppy wording on my side, I'll fix that. More precisely, `grep … < file.ext` is "input from a file via stdin" and `grep … < <(…)` is "input from the output of a command via stdin". Since in both cases, the input comes via `stdin` which a script sees as `/dev/fd/0`, a script cannot differentiate between these cases. – tanius Jul 01 '21 at 18:32
0

It's used as the stdin.

Try:

grep "hehe" - $(cat test.sh)

That might be wrong; I can't test it out on this computer. If you do it without the pipe like you tried, grep treats the last argument as a filename, ie, looks for a file called [contents of test.sh]. If you pass it a - (or don't put a last argument), you tell it to use stdin as the file.

You can also just pass grep a file to scan through:

grep "hehe" test.sh

...but you seem to be asking more of a generalized bash question, not really a grep usage question, so that's probably not too helpful.

Jarett Millard
  • 5,802
  • 4
  • 41
  • 48
  • The takes each word in test.sh and looks for a file with the same name as that word, and greps on those files (usually with very limited success). – Jonathan Leffler Oct 01 '09 at 16:11
  • The backticks are command substitution just like `$()`, just not nestable and easier to mess up. The latter form is probably what Tim is looking for. – Cascabel Oct 01 '09 at 16:13
  • Again, command substitution is not going to work. grep's arguments are not strings to search within; they are files to search within. Your first form will now grep through both standard input and all files with names given in test.sh. – Cascabel Oct 01 '09 at 16:17
  • Thanks! Yes my question was intended more general. Is it true that many bash commands are implemented in this way that they can take the same input from stdin as well as from command line argument? grep is just one example I tried to figure out this question. – Tim Oct 01 '09 at 16:31
  • @Jefromi: Ah, I see. Is there a way to do that that you know of? I was gonna put a \n after the -, but I wasn't sure that would work either. @Tim: Lots of commands use - as the stdin "file"; it might even be implemented that way in bash (ie, it'll work for all commands), but don't quote me on that. – Jarett Millard Oct 01 '09 at 16:42