ls | grep -v in script

Question

I have a bash script where I want to do this

    (in pseudo)
    for each item in excludelist
        append to string: | grep -v $item

    for each file in directory 
    do
        stuff
    done

and now

   (in real code, the creation and concatenation of f works, the problem is not here I believe)
   for i in $EXCLUDE
   do
       echo $i
       f="$f | grep -v $i"
   done

   for d in `ls $AWSTATSCONF/awstats*conf $f`
   do
       echo $d
   done

output

   ls: |: No such file or directory
   ls: grep: No such file or directory

Any and all help really appreciated to get this to work.

Greetings,

Shadow

but that part works. The problem lies in the execution of the ls. — ShadowFlame, May 07 '12 at 08:30
Considering the `ls` tries do access '`grep`', I don't think that part works so well. — VonC, May 07 '12 at 08:32
what I meant was that the building of the string containing the series of "grep -v"'s works quite well. I do realize that adding it to the ls-command does not work. — ShadowFlame, May 07 '12 at 08:33
Sure. Gite my concatenation proposition a try though, and see if the `ls` works better. — VonC, May 07 '12 at 08:36
See this: http://stackoverflow.com/questions/2005192/how-to-execute-a-bash-command-stored-as-a-string-with-quotes-and-asterisk — Vinko Vrsalovic, May 07 '12 at 08:48
Please read http://mywiki.wooledge.org/ParsingLs for an explanation of why `ls | grep` is a bad idea to start with. — Charles Duffy, May 07 '12 at 14:08

score 2 · Answer 1 · answered May 07 '12 at 13:36

Since you're using bash, you can use pattern matching to exclude your EXCLUDE list:

pattern=$(awk ' BEGIN {OFS="|"; printf("!(")} 
                {$1=$1; printf("%s",$0)} 
                END {print ")"}
              ' <<< $EXCLUDE)
shopt -s extglob
for d in $pattern; do
  echo $d
done

score 2 · Answer 2 · answered May 07 '12 at 14:06

There are many ways to deal with it.

There are, rightly, misgivings about parsing the output of ls. How justified those are depends on whether you have to deal with all possible file names (ones containing spaces and newlines are particularly problematic), or whether you have normal file names that use just the portable filename character set (which, according to POSIX, are the (Latin) alphabet, the digits, plus ., -, and _).

Assuming that you only have to deal with portable filenames, then you could get the effect you wanted with:

for i in $EXCLUDE
do
    echo $i
    f="$f | grep -v $i"
done

for d in `eval ls $AWSTATSCONF/awstats*conf $f`
do
    echo $d
done

Using eval like that forces the shell to treat the pipes in the string $f as pipes in the shell. Beware: eval is a powerful and therefore dangerous tool.

The chosen mechanism of running 20 greps in sequence if there are 20 terms to exclude uses a lot of processes. It probably won't be too serious for demonstration purposes, but if you're dealing with production work and thousands of files...not so good.

You'd probably want to look at building an egrep (or grep -E) command:

f="antidisestablishmentarianism"
for i in $EXCLUDE
do
    f="$f|$i"
done

ls $AWSTATSCONF/awstats*conf |
egrep -v "$f" |
while read d
do echo $d
done

I assume you don't have any stats files with names containing 'antidisestablishmentarianism' lurking around (yes, it's a real word; it is mostly used here as a joke, though).

The while read formulation is another way of writing the for loop. Be wary of subprocesses not being able to affect the parent shell's variables. That is, the while loop is processed by a sub-shell, not the main shell, so if you are relying on the loop to set variables in the main shell, it won't work — and you can/should go back to the for loop. However, you don't need eval again:

for d in $(ls $AWSTATSCONF/awstat*conf | egrep -v "$f")
do
    echo $d
done

Using $(...) instead of back-quotes is generally a good idea.

score 0 · Answer 3 · answered May 07 '12 at 08:45

0

The problem is that foo whatever whatever else is executed as plain escaped arguments to the ls command (of course | and grep are not found on the current directory). eval is the command that executes a string as a bash command

You need to do something like the following

for d in `eval "ls $AWSTATSCONF/awstats*conf $f"`
do
   echo $d
done

Keep in mind that this opens a big can of worms security wise, check the following FAQ to understand more and for other alternatives

answered May 07 '12 at 08:45

Vinko Vrsalovic

330,807
53
334
373

no problem. it is for internal use by professionals only. no web frontend or the likes. but thanks for the warning though :-) – ShadowFlame May 07 '12 at 09:26

score 0 · Answer 4 · answered May 07 '12 at 09:14

0

Don't use for i in $(ls). You should use globbing.

Don't use concatenation to build a long string of grep -v.

Where does $EXCLUDE come from? It should be an array instead of a string.

for d in "$AWSTATSCONF"/awstats*conf
do
    for e in "${exclude[@]}"
    do
        if [[ $d != $e ]]
        then
            echo "$d"
        fi
    done
done

answered May 07 '12 at 09:14

Dennis Williamson

346,391
90
374
439

EXCLUDE="string1 string2 string3" – ShadowFlame May 07 '12 at 09:33

score -1 · Answer 5 · answered May 07 '12 at 08:22

-1

You need to append the *output of grep to $f.

$f = "$f $(grep -v $i)"

answered May 07 '12 at 08:22

Benjamin Bannier

55,163
11
60
80

I believe I have misformulated my question. The part with the problem I believe is the ls, not the creation of the greps.. – ShadowFlame May 07 '12 at 08:31

ls | grep -v in script

5 Answers5