Looping through a set of files linux based on filepath

Question

I have directory with lots of compressed data with with a couple of file names. I have two file types als.sumstats.lmm.chr and als.sumstats.meta.chr. After chr there is a number 1-22. I want to loop through only the als.sumstats.meta.chr. However, my code is not working. I keep getting gzip: /ALSsummaryGWAS/Summary_Statistics_GWAS_2016/als.sumstats.meta.chr*.txt.gz: no such file or directory, suggesting my files are not being found with my loop. Can someone help. This is what I have right now.

#!/bin/bash

FILES=/ALSsummaryGWAS/Summary_Statistics_GWAS_2016/als.sumstats.meta.chr*.txt.gz
for f in $FILES;
do
 echo "$FILES"
 echo "extracting columns 2,1,3,9"
 gunzip -c $f | awk '{print $2, $1, $3, $14+$15}' >> ALSGWAS.txt
done

Can you please be more specific than "not working"? At least give the exact directory listing, the expected result and actual output/result. — kaylum, Aug 02 '20 at 22:55
You seem to have a typo in `Summary_Statisitics_GWAS_2016` (`Statistics`) : can you check that's not your problem? — sal, Aug 02 '20 at 23:07
Hi Kaylum, I have 22 files I want to loop through that have this als.sumstats.meta.chr1.txt.gz, als.sumstats.meta.chr2.txt.gz, ..... The output I think is clear in the awk statement. The error as I pointed out in the question is this: no such file or directory — Genetics, Aug 02 '20 at 23:37
gzip: /ALSsummaryGWAS/Summary_Statistics_GWAS_2016/als.sumstats.meta.chr*.txt.gz: no such file or directory — Genetics, Aug 02 '20 at 23:38

score 1 · Accepted Answer · answered Aug 03 '20 at 10:43

In your script snippet, wildcard '*' pattern is stored as a string in the $FILES variable which needs to be evaluated at some point to get the list of matching files.

In order to evaluate it, you can use eval like this:

FILES="ls -1 /ALSsummaryGWAS/Summary_Statistics_GWAS_2016/als.sumstats.meta.chr*.txt.gz"
for f in $(eval $FILES);
do
    echo "$FILES"
    echo "processing $f"
    echo "extracting columns 2,1,3,9"
    gunzip -c $f | awk '{print $2, $1, $3, $14+$15}' >> ALSGWAS.txt
done

But eval is not a recommended way to do such operations (eval is dangerous), so you can try it like this:

FILES=$(ls -1 /ALSsummaryGWAS/Summary_Statistics_GWAS_2016/als.sumstats.meta.chr*.txt.gz)
for f in $FILES;      
do
    echo "$FILES"
    echo "processing $f"
    echo "extracting columns 2,1,3,9"
    gunzip -c $f | awk '{print $2, $1, $3, $14+$15}' >> ALSGWAS.txt
done

Looping through a set of files linux based on filepath

1 Answers1