Bash Script to process data containing input string

Question

I am trying to create a script that will find all the files in a folder that contain, for example, the string 'J34567' and process them. Right now I can process all the files in the folder with my code, however, my script will not just process the contained string it will process all the files in the folder. In other words once I run the script even with the string name ./bashexample 'J37264' it will still process all the files even without that string name. Here is my code below:

#!/bin/bash

directory=$(cd `dirname .` && pwd)
tag=$1

echo find: $tag on $directory

find $directory . -type f -exec grep -sl "$tag"  {} \;

for files in $directory/*$tag*
do
    for i in *.std
    do
    /projects/OPSLIB/BCMTOOLS/sumfmt_linux < $i > $i.sum
    done

    for j in *.txt
    do
    egrep "device|Device|\(F\)" $i > $i.fail
    done
    echo $files
done

Is a .txt files generate after you process a .std file with path? On what type of file should the `$ARGUMENTS` be used to? .std, .txt or any? — konsolebox, Aug 14 '13 at 05:23
The .txt files generate at the same time with the .std files for all the files containing the input string. $ARGUMENTS should be used on any file, there are only two files anyways. — tiger_groove, Aug 14 '13 at 16:20
I updated my current code, right now it finds all the files that contain the input string, but it is still processing all the files in the current directory instead of just the ones the user input. — tiger_groove, Aug 14 '13 at 20:08
Hey guys, the code works great, just one minor change I would need help with. Is there a way to restrict user input that contains no string? for example if the user inputs -> xl-irv-05{kmoslehp}305: ./bashexample2 (no string). The command allows me to use the code with no string, however, if there is a lot of data in the datalog I don't want it to generate .fails or.sum files for all the files (since that would be a lot of space wasted), but simply output some warning saying something like "please input the input string" — tiger_groove, Aug 20 '13 at 22:42

Neum2012 · Answer 1 · 2013-08-17T15:46:35.187

1

Kevin, you could try the following:

#!/bin/bash

directory='/home'
tag=$1


for files in $directory/*$tag*
do
    if [ -f $files ]
    then
            #do your stuff
            echo $files
    fi 
done

where directory is your directory name (you could pass it as a command-line argument too) and tag is the search term you are looking for in a filename.

edited Aug 17 '13 at 15:46

answered Aug 14 '13 at 02:16

Neum2012

11
2

For some reason this is not working and still generating all the files inside the folder when I do the following: I input this -> xl-irv-05{kmoslehp}305: ./bashexample2 J42427 and get this output -> /home/*J42427* – tiger_groove Aug 14 '13 at 19:26
Kevin, sorry for the delay. I added an if-statement because you get that output when no files with that tag exist. -f $files checks to see if the file exists. – Neum2012 Aug 17 '13 at 15:47

konsolebox · Answer 2 · 2013-08-15T06:00:23.163

0

This is my temporary solution. Please check if it follows your intention.

#!/bin/bash

directory=$(cd `dirname .` && pwd)  ## Should this be just directory=$PWD ?
tag=$1

echo "find: $tag on $directory"

find "$directory" . -type f -exec grep -sl "$tag" {} \;  ## Shouldn't you add -maxdepth 1 ? Are the files listed here the one that should be processed in the loop below instead?

for file in "$directory"/*"$tag"*; do
    if [[ $file == *.std ]]; then
        /projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
    fi

    if [[ $file == *.txt ]]; then
        egrep "device|Device|\(F\)" "$file" > "${file}.fail"
    fi

    echo "$file"
done

Update 1

#!/bin/bash

directory=$PWD  ## Change this to another directory if needed.
tag=$1

echo "find: $tag on $directory"

while IFS= read -rd $'\0' file; do
    echo "$file"
    case "$file" in
    *.std)
        /projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
        ;;
    *.txt)
        egrep "device|Device|\(F\)" "$file" > "${file}.fail"
        ;;
    *)
        echo "Unexpected match: $file"
        ;;
    esac
done < <(exec find "$directory" -maxdepth 1 -type f -name "*${tag}*" \( -name '*.std' -or -name '*.txt' \) -print0)  ## Change or remove the maxdepth option as wanted.

Update 2

#!/bin/bash

directory=$PWD
tag=$1

echo "find: $tag on $directory"

while IFS= read -rd $'\0' file; do
    echo "$file"
    /projects/OPSLIB/BCMTOOLS/sumfmt_linux < "$file" > "${file}.sum"
done < <(exec find "$directory" . -maxdepth 1 -type f -name "*${tag}*" -name '*.std' -print0)

while IFS= read -rd $'\0' file; do
    echo "$file"
    egrep "device|Device|\(F\)" "$file" > "${file}.fail"
done < <(exec find "$directory" -maxdepth 1 -type f -name "*${tag}*" -name '*.txt' -print0)

edited Aug 15 '13 at 06:00

answered Aug 14 '13 at 20:15

konsolebox

72,135
12
99
105

Yes the files that are listed on that line are the ones that should be processed, and I am not familiar with the command 'maxdepth', can you explain this command. – tiger_groove Aug 14 '13 at 20:20
maxdepth limits the levels of directories that `find` would search for files. Anything wrong with the script besides that? – konsolebox Aug 14 '13 at 20:39
Nope everything else works great! Thanks a lot konsolebox! it wouldnt let me vote you up since my reputation hasnt reached 15 yet.. – tiger_groove Aug 14 '13 at 23:27
Hey konsolebox, is there a way to restrict user input that contains no string? for example if the user inputs -> xl-irv-05{kmoslehp}305: ./bashexample2, because right now the command allows me to use the code with no string, however, if there is a lot of data in the datalog i don't want it to generate .fails or.sum files for all the files, but simply output some warning saying something like "please input the input string" – tiger_groove Aug 20 '13 at 22:37
@kkmoslehpour Try to test with [[ -z $VAR ]] perhaps? If you like you could post the code. :) Perhaps starting a new thread about it would be helpful too, since it's a little different already. – konsolebox Aug 23 '13 at 15:43
@kkmoslehpour Try to test with [[ -z $VAR ]] perhaps? If you like you could post the code. :) Perhaps starting a new thread about it would be helpful too, since it's a little different already. – konsolebox Aug 23 '13 at 15:44

score 0 · Answer 3 · edited May 23 '17 at 12:20

Following script will give you the list of files that contain (inside the file, not in file name) the given pattern.

#!/bin/bash

directory=`pwd`
tag=$1

for file in $(find "$directory" -type f -exec grep -l "$tag" {} \;); do
    echo $file
    # use $file for further operations
done

What is the relevance of .std, .txt, .sum and .fail files to the files containing given pattern?

Its assumed there are no special characters, spaces, etc. in file names.
If that is the case following should help working around those.
How can I escape white space in a bash loop list?
Capturing output of find . -print0 into a bash array

There are multiple issues in your script.

Following is not required to set the operating directory to current directory.

directory=$(cd `dirname .` && pwd)

find is executed twice for the current directory due to $directory and ..

find $directory . -type f -exec grep -sl "$tag"  {} \;

Also, result/output of above find is not used in for loop.
For loop is run for files in the $directory (sub directories not considered) with their file name having the given pattern.

for files in $directory/*$tag*

Following for loop will run for all .txt files in current directory, but will result in only one output file due to use of $i from previous loop.

for j in *.txt
do
    egrep "device|Device|\(F\)" $i > $i.fail
done

Bash Script to process data containing input string

3 Answers3