111

I need to assign the results from a grep to an array... for example

grep -n "search term" file.txt | sed 's/:.*//'

This resulted in a bunch of lines with line numbers in which the search term was found.

1
3
12
19

What's the easiest way to assign them to a bash array? If I simply assign them to a variable they become a space-separated string.

ks1322
  • 33,961
  • 14
  • 109
  • 164
ceiling cat
  • 5,501
  • 9
  • 38
  • 51
  • See [this question](http://stackoverflow.com/questions/971162/how-to-initialize-a-bash-array-with-output-piped-from-another-command) – beerbajay Feb 26 '12 at 00:40
  • space-separated strings are easily traversable in bash. – Shiplu Mokaddim Feb 26 '12 at 00:42
  • ooh... should have searched more thoroughly. Thanks. – ceiling cat Feb 26 '12 at 00:46
  • As an aside, it is a common antipattern use `grep` to get the line numbers of something and eventually pass them to a tool which has regex support of its own. Then it can find the same lines as `grep` found, without taking the detour of figuring out the line numbers of the matches. – tripleee Nov 22 '18 at 06:05
  • Does this answer your question? [How to initialize a bash array with output piped from another command?](https://stackoverflow.com/questions/971162/how-to-initialize-a-bash-array-with-output-piped-from-another-command) – zypA13510 Nov 27 '19 at 01:32

3 Answers3

179

To assign the output of a command to an array, you need to use a command substitution inside of an array assignment. For a general command command this looks like:

arr=( $(command) )

In the example of the OP, this would read:

arr=($(grep -n "search term" file.txt | sed 's/:.*//'))

The inner $() runs the command while the outer () causes the output to be an array. The problem with this is that it will not work when the output of the command contains spaces. To handle this, you can set IFS to \n.

IFS=$'\n' arr=($(grep -n "search term" file.txt | sed 's/:.*//'))

You can also cut out the need for sed by performing an expansion on each element of the array:

arr=($(grep -n "search term" file.txt))
arr=("${arr[@]%%:*}")
Noel Yap
  • 18,822
  • 21
  • 92
  • 144
jordanm
  • 33,009
  • 7
  • 61
  • 76
  • to me it worked only _without_ outer brackets ```dvmList=`grep -RohP '(?<=oramds:).+\.dvm' myFolder``` ` – tuxErrante Jan 24 '19 at 09:23
  • @bastaPasta That is not not an array assignment. Array assignment may not have worked for you if you were not using a shell that supported arrays. – jordanm Jan 24 '19 at 14:55
  • You're right I was using sh instead of bash, but it works fine looping in a for – tuxErrante Jan 24 '19 at 15:00
  • @bastaPasta yes it will, as long as your "elements" do not contain spaces. There are other ways to simulate arrays for shells that do not have them via `set` and the special array `$@`. – jordanm Jan 24 '19 at 15:10
  • 3
    Note that `array=( $(command) )` is considered an antipattern, and is the topic of [BashPitfalls #50](http://mywiki.wooledge.org/BashPitfalls#hosts.3D.28_.24.28aws_....29_.29). – Charles Duffy Nov 16 '20 at 14:07
  • 1
    As a better-practice approach, still compatible with bash 3.x, consider `read -r -d '' -a arr < <(grep ... | sed ... && printf '\0')`, which avoids globbing and string-splitting your `command`'s output, and _also_ passes through a nonzero exit status from the pipeline (coalescing them all to 1, because the lack of a trailing delimiter makes `read` fail) – Charles Duffy Nov 16 '20 at 14:10
  • NB: If you have IFS=$'\n' set in general or for all of these lines (to handle terms with whitespace) then the append command will not keep the original array structure. It will just make array contain only 1 item (a string list), with all the changed strings. As seen if you run: `declare -p array` afterwards. – Magne Apr 28 '21 at 13:03
  • I'm seeing an extraneous whitespace character at the end of every element of my array when using the `IFS=$'\n' ` approach. Curious if I'm alone in this or if there's a known solution. – levigroker Oct 13 '21 at 19:22
  • @levigroker my guess is dos line endings, which are `\r\n` instead of just `\n` – jordanm Oct 13 '21 at 19:58
  • 1
    @jordanm Thanks for the reply. Coming back to this today has helped me find the root cause, which is *not* related to setting the IFS or the proposed solution. The issue was a dangling space in my input which I did not account for in my regular expression upstream of this. (for anyone curious, beware `agvtool` leaves trailing whitespace on every line of output) – levigroker Oct 14 '21 at 20:57
13

Space-separated strings are easily traversable in bash.

# save the ouput
output=$(grep -n "search term" file.txt | sed 's/:.*//')

# iterating by for.
for x in $output; do echo $x; done;

# awk
echo $output | awk '{for(i=1;i<=NF;i++) print $i;}'

# convert to an array
ar=($output)
echo ${ar[3]} # echos 4th element

if you are thinking space in file name use find . -printf "\"%p\"\n"

Scott Weldon
  • 9,673
  • 6
  • 48
  • 67
Shiplu Mokaddim
  • 56,364
  • 17
  • 141
  • 187
  • 5
    But broken for files with spaces. – jordanm Feb 26 '12 at 01:01
  • @jordanm use `find . -printf "\"%p\"\n"` – Shiplu Mokaddim Feb 26 '12 at 01:17
  • @ShipluMokaddim, that doesn't work. `for x in $output` with such a `find` command will cause a file named `two words` to be iterated over as one value `"two` and another value `words"`. – Charles Duffy Nov 16 '20 at 14:07
  • @ShipluMokaddim, ...the same thing is true if you use `ar=( $output )` with `output` assigned in that way, because parameter expansion and command substitution results don't go through the quote removal parsing phase. – Charles Duffy Nov 16 '20 at 14:08
3

@Charles Duffy linked the Bash anti-pattern docs in a comment, and those give the most correct answer:

readarray -t arr < <(grep -n "search term" file.txt | sed 's/:.*//')

His comment:

Note that array=( $(command) ) is considered an antipattern, and is the topic of BashPitfalls #50. – Charles Duffy Nov 16, 2020 at 14:07

Asfand Qazi
  • 6,586
  • 4
  • 32
  • 34