0

I have a directory that has daily files that contains over 500 fields per line and 600,000 rows.

I want to look at 1 file and find all lines that contain B2 on field #351.

Then search all files for any lines that match the values in fields 282, 341, 314, and 348 in the output from the first file.

Right now I have the following but it produces blank output:

ARCHIVEDIR=/appl/dir/archive
file1_tmp=$$.tmp
zcat ${ARCHIVEDIR}/FILE_12162019.gz | awk 'BEGIN{FS=OFS="|"} $351 == "B2"{gsub(/ /,""); print $282,$341,$314,$348}' > "$file1_tmp"
for fname in ${ARCHIVEDIR}/FILE_*; do
    zcat "$fname" | awk -v fname="$fname" '
        BEGIN { FS=OFS=SUBSEP="|" }
        NR==FNR { tgts[$0]; next }
        ($282,$341,$314,$348) in tgts { print fname, $0 }
    ' "$file1_tmp" -
done

For example, file1 has 130,000 records containing B2 in field 351. I want to find any records from all files (including the original in file1) matching fields 282, 341, 314, and 348.

ORIGINAL POST Below - reposted to try and clear up some confusion

I gave up trying and ended up with the following in a for loop:

echo -e "$FILENAME|\c"
zcat $FILENAME | grep "$SYSTEM" | grep "$RECORDNUM" | grep "$LOCATION" | grep "$PENGUINS"

The output is:

FILENAME|{each Line the matches all 4 search variables}

I'm looking for an awk command that will clean it that output efficiently.

I have tried:

zcat $FILENAME | awk -v FILENAME=$FILENAME -v SYSTEM=$SYSTEM -v RECORDNUM=$RECORDNUM -v LOCATION=$LOCATION -v PENGUINS=$PENGUINS -v FS="|" -v OFS='|' '/SYSTEM/ && /RECORDNUM/ && /LOCATION/ && /PENGUINS/ {print FILENAME,$0}'`

and because the positional values will always be the same, I even tried the following:

zcat $FILENAME | awk -v FILENAME=$FILENAME -v SYSTEM=$SYSTEM -v RECORDNUM=$RECORDNUM -v LOCATION=$LOCATION -v PENGUINS=$PENGUINS -v FS="|" -v OFS='|' '($282 == SYSTEM) && ($341 == RECORDNUM)  && ($314 == LOCATION) && ($348 == PENGUINS) {print FILENAME,$0}'

SAMPLE INPUT FILE: (for testing purposes, I created 4 copies of the following and gzipped the files) sh-4.2$ zcat file1 SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED SYSTEM2|SPACER|88083|SPACER|SPACER|FLORIDA|SPACER|SPACER|SPACER|MOUNTED SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED SYSTEM2|SPACER|123141|SPACER|SPACER|NOCAL|SPACER|SPACER|SPACER|MOUNTED SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED SYSTEM2|SPACER|90391|SPACER|SPACER|TEXAS|SPACER|SPACER|SPACER|MOUNTED SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED SYSTEM2|SPACER|354295|SPACER|SPACER|FLORIDA|SPACER|SPACER|SPACER|MOUNTED

sh-4.2$ ls -ls
total 32
4 -rw-rw-rw- 1 host pdx  170 Dec 20 06:10 file1.gz
4 -rw-rw-rw- 1 host pdx  170 Dec 20 06:10 file2.gz
4 -rw-rw-rw- 1 host pdx  170 Dec 20 06:10 file3.gz
4 -rw-rw-rw- 1 host pdx  170 Dec 20 06:10 file4.gz
4 -rwxrwxrwx 1 host pdx  727 Dec 20 06:15 testawk
4 -rwxrwxrwx 1 host pdx  626 Dec 20 06:16 testgrep

Then created 2 scripts: testawk

 for FILENAME in `ls file1.gz`
do
   zcat $FILENAME | awk -v FS='|' -v OFS='|' '{if ($10 == "STUFFED") print $1,$3,$6,$10}' | tr -d " " >> $$.tmp
done

for TMPR in `cat $$.tmp`
do
   SYSTEM=`echo $TMPR | awk -v FS='|' '{print $1}'`; export SYSTEM
   RECORDNUM=`echo $TMPR | awk -v FS='|' '{print $2}'`; export RECORDNUM
   LOCATION=`echo $TMPR | awk -v FS='|' '{print $3}'`; export LOCATION
   PENGUINS=`echo $TMPR | awk -v FS='|' '{print $4}'`; export PENGUINS
   for FILENAME in `ls fil*`
   do
      export FILENAME
      zcat $FILENAME | awk -v FILENAME=$FILENAME -v SYSTEM=$SYSTEM -v RECORDNUM=$RECORDNUM -v LOCATION=$LOCATION -v PENGUINS=$PENGUINS -v FS="|" '/SYSTEM/ && /RECORDNUM/ && /LOCATION/ && /PENGUINS/'
   done
done

and

testgrep

for FILENAME in `ls file1.gz`
do
   zcat $FILENAME | awk -v FS='|' -v OFS='|' '{if ($10 == "STUFFED") print $1,$3,$6,$10}' | tr -d " " >> $$.tmp
done

for TMPR in `cat $$.tmp`
do
   SYSTEM=`echo $TMPR | awk -v FS='|' '{print $1}'`; export SYSTEM
   RECORDNUM=`echo $TMPR | awk -v FS='|' '{print $2}'`; export RECORDNUM
   LOCATION=`echo $TMPR | awk -v FS='|' '{print $3}'`; export LOCATION
   PENGUINS=`echo $TMPR | awk -v FS='|' '{print $4}'`; export PENGUINS
   for FILENAME in `ls fil*`
   do
      echo -e "$FILENAME|\c"; zcat $FILENAME | grep "$SYSTEM" | grep "$RECORDNUM" | grep "$LOCATION" | grep "$PENGUINS"
   done
done

When I execute testawk, the output is blank.

When I execute testgrep, the output contains all the lines where $PENGUIN=STUFFED with the filename in the beginning of each line.

sh-4.2$ ./testgrep
file1.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|1435|SPACER|SPACER|PHILLY|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|80128312|SPACER|SPACER|SOCAL|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|12|SPACER|SPACER|GEORGIA|SPACER|SPACER|SPACER|STUFFED
file1.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file2.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file3.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file4.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED
file.gz|SYSTEM1|SPACER|124910|SPACER|SPACER|VIRGINIA|SPACER|SPACER|SPACER|STUFFED

BREAKDOWN of what I'm doing and trying to do: The following portion of the scripts are the same, it will create a file called $$.tmp for any lines in file1.gz that have "STUFFED" in field 10. This file will only contain the values from fields 1, 3, 6, and 10. (this is used in the next portion of the scripts and it currently works)

for FILENAME in `ls file1.gz`
do
   zcat $FILENAME | awk -v FS='|' -v OFS='|' '{if ($10 == "STUFFED") print $1,$3,$6,$10}' | tr -d " " >> $$.tmp
done

The next portion of the script assigns variables for each of the 4 fields and exports the variables to be used in awk (not sure if the export is needed).

for TMPR in `cat $$.tmp`
do
   SYSTEM=`echo $TMPR | awk -v FS='|' '{print $1}'`; export SYSTEM
   RECORDNUM=`echo $TMPR | awk -v FS='|' '{print $2}'`; export RECORDNUM
   LOCATION=`echo $TMPR | awk -v FS='|' '{print $3}'`; export LOCATION
   PENGUINS=`echo $TMPR | awk -v FS='|' '{print $4}'`; export PENGUINS

This portion of the script will start my for loop to check all files starting with fil for matches: (I've included both the awk and grep commands by commented them out)

   for FILENAME in `ls fil*`
   do
      export FILENAME
#      zcat $FILENAME | awk -v FILENAME=$FILENAME -v SYSTEM=$SYSTEM -v RECORDNUM=$RECORDNUM -v LOCATION=$LOCATION -v PENGUINS=$PENGUINS -v FS="|" '/SYSTEM/ && /RECORDNUM/ && /LOCATION/ && /PENGUINS/'
#      echo -e "$FILENAME|\c"; zcat $FILENAME | grep "$SYSTEM" | grep "$RECORDNUM" | grep "$LOCATION" | grep "$PENGUINS"
   done

And then I end the original for loop: done

Unix_pharmacy
  • 15
  • 7
  • What is not working? – Inian Dec 20 '19 at 12:49
  • It's not finding the lines that match all 4. The 4 greps work. I'll try to put together a sample file if needed. – Unix_pharmacy Dec 20 '19 at 12:51
  • 1
    @Unix_pharmacy, It is always recommended to post sample of input and expected output, so kindly do add the same in your question and let us know then. – RavinderSingh13 Dec 20 '19 at 12:52
  • I have added information to the post with sample scripts/input. Hope this helps. – Unix_pharmacy Dec 20 '19 at 13:36
  • Your shell script has a lot of issues by the way - as a start you should run it through http://shellcheck.net and correct everything it warns you about. See also https://stackoverflow.com/questions/673055/correct-bash-and-shell-script-variable-capitalization, https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice, https://mywiki.wooledge.org/Quotes, https://unix.stackexchange.com/questions/321697/why-is-looping-over-finds-output-bad-practice, and http://mywiki.wooledge.org/BashFAQ/082. – Ed Morton Dec 20 '19 at 13:56
  • Will do, my original script is much larger and the output is over 500 fields vs just 10. I tried to create a quick script for the purposes of this question. Also, I had to change all the variable names and locations due to confidentiality. – Unix_pharmacy Dec 20 '19 at 14:00
  • i'm trying to replace the 2 line grep command (echo -e + 4 greps) with an awk command. – Unix_pharmacy Dec 20 '19 at 14:02
  • I agree there probably is. I have 12 files with over 600,000 lines and over 500 fields on each line. Let's say this file has sales information. 1 line for the sale (S1 populated in field 251) and 1 line for a return (R1 populated in field 251). I'm trying to find all returns and print the sales that match those returns. To do this, I'm matching on 4 key fields that should be unique to the record regardless if sold/returned. (sorry, I feel like this is complicating things) – Unix_pharmacy Dec 20 '19 at 14:14

2 Answers2

1

You can't match variables in the pattern that way. Try this:

awk -v file=$FILENAME -v system=$SYSTEM -v ... '$0 ~ system && ... {print file,$0}'
steffen
  • 16,138
  • 4
  • 42
  • 81
1

This is how to do what I THINK you're trying to do, i.e. print lines from several gzipped files whose names start with fil* that have values in fields $282,$341,$314,$348 that match values in fields $1,$3,$6,$10 from file1.gz:

file1_tmp=$(mktemp)
zcat file1.gz | awk 'BEGIN{FS=OFS="|"} $10 == "STUFFED"{gsub(/ /,""); print $1,$3,$6,$10}' > "$file1_tmp"
for fname in fil*; do
    zcat "$fname" | awk -v fname="$fname" '
        BEGIN { FS=OFS=SUBSEP="|" }
        NR==FNR { tgts[$0]; next }
        ($282,$341,$314,$348) in tgts { print fname, $0 }
    ' "$file1_tmp" -
done

but I don't know why your script has a loop on 1 file (for FILENAME in `ls file1.gz`) nor do I understand if you REALLY want to process that file again in the second loop where file1.gz will match the globbing pattern fil*.

If the above doesn't do exactly what you want then update your question to clarify what it is you're trying to do (as opposed to how you're trying to do it).

Ed Morton
  • 188,023
  • 17
  • 78
  • 185
  • Thank you @Ed Morton. I decided to wipe out my question and re-ask in a different way. I believe your script will work, but I'm missing something since the output is blank. I hope my explanation of what I'm trying to do helps . – Unix_pharmacy Dec 20 '19 at 15:21
  • I wish you hadn't done that as now my answer doesn't make any sense since it's the same script as in your question and I'll probably start getting downvotes. In any case, sounds like I guessed right but we can't help you debug it without something to test against. You need to in your question provide a **small** (ie. about 5 lines each of 5 fields per line) example of plain text files `file1` and at least 2 files `file2` and `file3` that can be used to test the script against using `cat` instead of `zcat`. – Ed Morton Dec 20 '19 at 15:38
  • I saved the post prior to wiping it out. I can add it back to the bottom of my current post if you want. – Unix_pharmacy Dec 20 '19 at 16:35
  • Just revert your post to the previous version and add the new stuff at the top. – Ed Morton Dec 20 '19 at 17:54