reuse command output multiple times in bash script

Question

I have a bash script that operates several similar find commands and reads the output into variables:

f_list=`find /foo -name bar | sort`
f_number=`find /foo -name bar | wc -l`
f_first=`find /foo -name bar | sort | head -n 1`

All this works as expected.

Assuming the "find" command to be expensive (time wise), is there a clever way of re-using one of the results for the others?

What I tried (and failed with) is:

f_list=`find /foo -name bar | sort`
f_number=`echo "$f_list" | wc -l`
f_first=`echo "$f_list" | head -n 1`

This doesn't work, but I hope it shows what I want to achieve.

It seems that putting the results into a variable spoils some format of the original output, which breaks stuff when sent again to the other commands.

Is there some clever way of achieving what I want?

EDIT

I created a fully working example you could recreate. In my working dir I have a folder "foo" with 3 files "bar1", "bar2", "bar3".

find_result=`find ./foo -type f -iname "bar*" | sort`
find_count1=`echo "$find_result" | wc -l`
echo "$find_result"
echo $find_count1
find_count2=`find ./foo -type f -iname "bar*" | wc -l`
find ./foo -type f -iname "bar*"
echo $find_count2

results in the expected

./foo/bar1
./foo/bar2
./foo/bar3
3
./foo/bar3
./foo/bar2
./foo/bar1
3

but when the result is empty (I modified the search criteria to find nothing)

find_result=`find ./foo -type f -iname "bor*" | sort`
find_count1=`echo "$find_result" | wc -l`
echo "$find_result"
echo $find_count1
find_count2=`find ./foo -type f -iname "bor*" | wc -l`
find ./foo -type f -iname "bor*"
echo $find_count2

the two results differ (notice the empty result line in front of the "1")

1
0

And thus I thought the culprint to be the extra line break in the echo command. Therefore I removed that (notice the "-n" in the second line):

find_result=`find ./foo -type f -iname "bor*" | sort`
find_count1=`echo -n "$find_result" | wc -l`
echo "$find_result"
echo $find_count1
find_count2=`find ./foo -type f -iname "bor*" | wc -l`
find ./foo -type f -iname "bor*"
echo $find_count2

which solves the problem for empty results

0
0

but now when there are results, the "wc -l" counts the wrong way so

find_result=`find ./foo -type f -iname "bar*" | sort`
find_count1=`echo -n "$find_result" | wc -l`
echo "$find_result"
echo $find_count1
find_count2=`find ./foo -type f -iname "bar*" | wc -l`
find ./foo -type f -iname "bar*"
echo $find_count2

yields in

./foo/bar1
./foo/bar2
./foo/bar3
2
./foo/bar3
./foo/bar2
./foo/bar1
3

So the problem is one line break that I thought must have an easy resolution to avoid being different between empty and non-empty find results. I have the feeling of missing something very simple.

`This doesn't work` Doesn't it? It looks like it should work. How do you know it doesn't work? But anyway, do not use \` backticks. Use `$(..)` instead. — KamilCuk, Mar 04 '21 at 14:12
"It modifies the output" is usually because of an unquoted expansion (see [I just assigned a variable, but echo $variable shows something else](https://stackoverflow.com/q/29378566/3266847)), but that doesn't seem to be the case here. — Benjamin W., Mar 04 '21 at 14:18
@KamilCuk: I've tried repalcing the backticks and the result is exactly the same. Is there a reason why $(...) is better than \`...\`? — , Mar 04 '21 at 19:08
@Benjamin W.: I edited my question, adding a minimal example. I think there is no unquoted expansion or funny character involved. — , Mar 04 '21 at 19:09

Socowi · Answer 1 · 2021-03-04T14:23:44.810

2

Using a variable to reuse find's output should work. Maybe there are backslashes in your filenames that get interpreted by echo (although that should not happen by default). If so, then use printf %s "$f_list" instead. But without a complete verifiable example we cannot know for sure.

However, in this specific case you can also switch to the following command which is also safe if no file matches and safe in the very unusual case where some files have multi-line filenames.

shopt -s globstar dotglob nullglob
list=(/foo/**/bar)
number=${#list[@]}
first=${list[0]}

edited Mar 04 '21 at 14:23

answered Mar 04 '21 at 14:10

Socowi

25,550
3
32
54

Notice that this differs from the `find` output in that it doesn't match hidden directories – you have to either use `list=(/foo/{,.}**/bar)` for that, or `shopt -s dotglob`. – Benjamin W. Mar 04 '21 at 14:15
Good catch. Thank you for the comment. I'd go with `dotglob` as `/foo/{,.}**/bar` matches `/foo/bar` and `/foo/./bar` at the same time. – Socowi Mar 04 '21 at 14:23
That's the same path and would be normalized away, no? Only the first one would actually be in the expansion. – Benjamin W. Mar 04 '21 at 14:29
@BenjaminW. It is the same path and it will be normalized, but both versions will be in the expansion (because the there are actually two expansions). There is even another problem I didn't recognize so far: The parent directory of `foo` might also be included! You can try this using `shopt -s globstar && mkdir -p a/{b,c}/c && echo a/b/{,.}**/c`. This prints `a/b/c a/b/./c a/b/../c`. – Socowi Mar 04 '21 at 16:42
1

Ah, if it has the same name! True. `dotglob` and `nullglob` is much cleaner then. – Benjamin W. Mar 04 '21 at 17:00

score 1 · Answer 2 · answered Mar 04 '21 at 14:23

1

An alternative would be to use a temporary file. That would be something like:

temp=$(mktemp /tmp/prefixXXXXXXXXXXX)
find /foo -name bar | sort > $temp
f_number=`wc -l $temp`
f_first=`head -n 1 $temp`

rm -f $temp

answered Mar 04 '21 at 14:23

Ljm Dullaart

4,273
2
14
31

your solution works for me but I have to use for example `wc -l <$temp` instead of `wc -l $temp` otherwise wc would add unwanted extra output – Mar 09 '21 at 16:10

score 0 · Accepted Answer · answered Mar 04 '21 at 15:45

Use an array to store the null delimited list returned by find -print0 | sort -z:

#!/usr/bin/env bash

# map null delimited output of command-group into the f_list array
mapfile -t f_list < <(find /foo -name bar -print0 | sort --zero-terminated)

# Number of entries in the f_list array
f_number=${#f_list[@]}

# First entry of the f_list array
f_first=${f_list[0]}

reuse command output multiple times in bash script

3 Answers3