Looping over files in UNIX

Question

Im trying to loop over all the files in a directory, whose name is given as a command line argument (e.g. myfolder). For each file the grep command should run on the folder, and count the number of times the phrase (e.g. myphrase) is found in the text file.

When I run my code, similar to below, I get the error "No such file or directory". I've tried calling the script using ./myscript.sh myfolder and ./myscript.sh /[fullpath]/myfolder and they both result in the same error.

for f in "$1"
do
  echo "processing $f file"
  grep -o '<myphrase>' "$f" | wc -l
done

Any ideas as to what's going wrong? If it helps the script is being run from within the same folder as the text files, and the command must be called with the folder name as an argument - both annoying requirements I must follow.

Edit: running ls -ld for this folder gives drwxr-xr-x@ 829 user staff 28186 7 Feb 17:19 my folder

Answer 1

As anubhava mentioned above:

In your for loop use:

for f in "$1"/*
do
...
done

You can then check if f is a file with:

[[ -f $f ]]

And perform your necessary logic inside the loop:

[[ -f $f ]] && grep -o '<myphrase>' "$f" | wc -l

So in summary:

for f in "$1"/*
do
 echo "processing $f file"
 [[ -f $f ]] && grep -o '<myphrase>' "$f" | wc -l
done

Answer 2

You need to add a wildcard match to your target directory.
Compare the two for loops in foo.sh ("original" is commented out).

Also I modified the grep command to just be an echo so you have an
easy way to preview what it is going to attempt to execute.

sample output from foo.sh

edit: I added -r (below, in foo.sh) to check for read permission on dir $1. That would be hard to trouble shoot; when I tested against a no-read directory it just ran looking like nothing was in the directory (even though it had a *.java and a *.class file, as shown above).

To be more clear, running against a non-readable dir looks like this (this is without the -r check):

$ chmod a-r tmp
$ ./foo.sh tmp
$1="tmp"
processing tmp/* file
grep -o '<myphrase>' "tmp/*" | wc -l
$ 

Note the "processing tmp/* file" line above. The for loop is feeds the literal characters "tmp/*" into the $f variable.
Which is fine, that is exactly how wildcard expansion is supposed to work if the pattern doesn't match anything.
But we're not checking for error codes from grep, so it could be hard
to notice grep complaining about "file tmp/* not found".

foo.sh output (revised)

$ chmod a+r tmp
$ ./foo.sh
$1=""
Error: no directory specified.
$ ./foo.sh foo.sh
$1="foo.sh"
Error: "foo.sh" is not a dir.
$ ./foo.sh tmp
$1="tmp"
processing tmp/Foo.class file
grep -o '<myphrase>' "tmp/Foo.class" | wc -l
processing tmp/Foo.java file
grep -o '<myphrase>' "tmp/Foo.java" | wc -l
$ chmod a-r tmp
$ ./foo.sh tmp
$1="tmp"
Error: no read permissions on dir "tmp".
$ 

foo.sh

#!/bin/bash

echo "\$1=\"$1\""
if [ -z "$1" ]; then
   echo "Error: no directory specified."
   exit 1
fi
if [ ! -e "$1" ]; then
   echo "Error: dir \"$1\" does not exist."
   exit 1
fi
if [ ! -d "$1" ]; then
   echo "Error: \"$1\" is not a dir."
   exit 1
fi
if [ ! -r "$1" ]; then
   echo "Error: no read permissions on dir \"$1\"."
   exit 1
fi
# maybe default to "." if $1 is empty ?
# original: for f in "$1"
for f in "$1"/*
do
  echo "processing $f file"
  echo "grep -o '<myphrase>' \"$f\" | wc -l"
  # maybe  change myphrase to $2 ?
done

Also, as pointed out elsewhere, you are writing your own version of the "find" command. Another possibility follows...

So what about find ?

I'm adding this just to encourage you to look at the find command at some point.

Disclaimer: Writing your own script is perfectly ok. There is value in the research and understanding how to roll-your-own.

find is indeed complex, but it is well worth climbing find's learning curve for the long run.

Note that 99% of the following is just comments.

ezfind.sh (an example)

#!/bin/bash

# example ussage:
# ezfind.sh "$HOME/my_dir"  "foo.*bar"
#  "$1" is the start point, any directory path (relative or absolute).
#  -type f limits matches to regular files (e.g. probably dont
#  want to run grep dirs or devices).
#  exec args ar funky, see below.
#  Optional: see bottom of this script for notes about -depth
#  to limit how deep find will search.
#------------------------------------------------------
find "$1" -type f -exec grep -i -o   "$2" '{}' ';'
#                       \________/  \___/ \__/ \_/
#                           |        |     |   |
#   command ----------------+        |     |   |
#   pattern for grep ----------------+     |   |
#   find replaces {} w/filename------------+   |
#   find exepects a semicolon for end-of-cmd---+

# Quoting is funky for -exec arguments.
# The values have to to survive current bash interpretation,
# so they can be passed to find's argument list.
# Then find turns around and passes them to grep's argument list.
# The semicolon is normally a bash statement separator so we
# need to quote it (or escape it) so it gets passed to find
# as part of the arg list.

# find -exec will replace {} with current filename.
# find gives you a crazy amount of file name, file type and date range options.
#    search for all file names in /what/ever matching "*.txt"
#    search for all file names matching "*.sh" modified in the last hour.
# for more on find, see here:
# http://www.softpanorama.org/Tools/Find/using_exec_option_and_xargs_in_find.shtml
# 
# What if I don't want to search every sub-folder, all the way down?
# Also about not traversing subdirectories, consider the -depth modifer.
# To just search the target directory, modify the above to read:
#     find "$1" -maxdepth 0  ....
# To just search the target directory and immediate subdirectories...
#     find "$1" -maxdepth 1  ....
# For a nice summary of -depth, see here:
#     http://www.tech-recipes.com/rx/31/limit-the-depth-of-search-using-find/

Answer 3

From the Directory details, it seems that there is a space in the directory name:

drwxr-xr-x@ 829 user staff 28186 7 Feb 17:19 my folder

try to run the script as below or remove the space from the directory name :)

./myscript.sh "my folder"

Answer 4

You can just do:

find <directory> -type f -exec egrep -c '<regular expression>' {} +

Note that there is no need to check for file existence because find does not find non-existing files. Also note that find traverses sub-directories found (tunable behaviour with -maxdepth 1).

E.g.:

find /usr/include -type f -exec egrep -c 'int128' {} +