1

I have a folder (/test) that contains several subfolder and files (at various level). Inside /test, I want to iterate only text files containing a particular string ($string), ignoring files and entire directories specified in a file ($to_skip) by path. This "exclusion list" contains both files and folder paths as shown below:

/test/ex1/fileA
/test/bob/ex1/fileB
/test/ex1/subfolder
/test/jerry/ex2

that is, one path per line. What I've done follows, but it didn't worked (the various options are required for other reasons):

grep -nriFlI "$string" "/test" | grep -vFf "$to_skip" > "$out"

The 1st grep actually give me the correct list of paths (any text files containing $string), but the 2nd grep doesn't filter as expected since $out contains all the items produced by the 1st grep except the occurrences corresponding the last pattern.

The strange thing is that it works correctly only for the last pattern. For example, if /test, among the others data, has the following files/folders

/test/jerry/ex2/f1
/test/jerry/ex2/f2
/test/jerry/ex2/foo/f3
/test/jerry/ex2/bar

and /test/jerry/ex2/ is the last pattern specified in $to_skip, the paths above (and in general any data under /test/jerry/ex2/) are being excluded correctly!

How can I achieve my target?

Many thanks!

user9952796
  • 15
  • 4
  • 1
    If you put absolute paths into the file `$to_skip` and use `"$(realpath "$root")"` instead of `"$root"` in your command, it should produce the desired output. – Ruslan Osmanov Jun 13 '23 at 11:12
  • What about this? https://stackoverflow.com/questions/4210042/how-do-i-exclude-a-directory-when-using-find – Dominique Jun 13 '23 at 11:17
  • @RuslanOsmanov unfortunately, following you tip doesn't solve the problem. Anyway, I added some details to my question about a "strange" behavior. – user9952796 Jun 13 '23 at 12:33
  • 2
    This sounds like a job for which one might want to use `find` to pass filenames to `grep`. – Charles Duffy Jun 13 '23 at 12:53
  • @Dominique yes, but I must include the paths to be skipped inside a file, not hardcoded. – user9952796 Jun 13 '23 at 12:53
  • @user9952796: turn the content of that file into a format, understood by `find -prune`. – Dominique Jun 13 '23 at 12:56
  • @CharlesDuffy I tried yesterday something like https://stackoverflow.com/questions/22558245/exclude-list-of-files-from-find, but it doesn't work for me – user9952796 Jun 13 '23 at 12:57
  • 1
    Please [edit] your question and add all information to the question instead of using comments for clarification. If you did "something like" a solution, show the exact commands you used. With the information in the question it is difficult to reproduce the problem. Create a [mre]. Show an example of the files and their contents where necessary, the values of the variables, contents of the file `$to_skip` and the expected result. If your ecample needs a directory structure with several files it would be good to add a shell script that creates all these. Better use `/tmp` instead of `/test`. – Bodo Jun 13 '23 at 14:59

1 Answers1

0

As suggested by Dominique, putting -prune to find command will be a common method to exclude specified dirs/files out of the search result. Would you please try the following:

#!/bin/bash

# create the option to "find" such as "-path /test/foo -o -path /test/bar -o ..."
while IFS= read -r f; do
    (( ${#opts[@]} > 0 )) && opts+=("-o")       # 2nd argument and thereafter
    opts+=("-path" "$f")
done < "$to_skip"

find /test \( "${opts[@]}" \) -prune -o -type f -exec grep -niFlI "$string" '{}' +
  • The array opts contains the exclusion list created from "$to_skip"
  • The pair of \( .. \) is needed for the precedence.
  • The -r option to grep is dropped because the argument list is already expanded as recursively traversed files.
tshiono
  • 21,248
  • 2
  • 14
  • 22
  • Thank you, I don't understand why you use the -o option – user9952796 Jun 14 '23 at 09:48
  • The `-o` option is for `or`. We want to exclude the path if it matches **either** of the paths in the exclusion list. Then the paths should be combined with `-o`. Is that clear? – tshiono Jun 14 '23 at 10:49
  • BTW is my script working? – tshiono Jun 14 '23 at 10:55
  • Sorry, I didn't explain myself well: I'm talking about the `-o` preceding `-type f` – user9952796 Jun 14 '23 at 12:19
  • When we use `-prune` action, the another action should be put after `-o`. The logical flow control is similar to `if CONDITION then prune; else ACTION`, where the `ACTION` is `-type f -exec grep ...` meaning `if the path is a file, do grep`. – tshiono Jun 14 '23 at 12:32
  • A shell shortcut statement for `if CONDITION; then ACTION1; else ACTION2` is `CONDITION && ACTION1 || ACTION2`. It demonstrates why `-o` (OR) works as `else`. – tshiono Jun 14 '23 at 12:47