Nested for loop with cat EOF issues

Question

Here is my code, I have two files where one contains file locations and other contains input name. I want to create a for-loop that works to incorporate both those files. I tried that here

#!/bin/bash -l
x=`cat x_file_locations`
y=`cat y_list`
for i in $x
do
    for j in $y
    do
            cat << EOF >> ./algo/job_$j.sh
            #!/bin/bash -l
            #
            #$ -N $i
            #$ -cwd
            #$ -pe smp 10
            #$ -l mem_free=20G

            algo --input_se $i --output $j --threads=10
            EOF
    done
done

I get this error

line 24: warning: here-document at line 10 delimited by end-of-file (wanted `EOF')
line 25: syntax error: unexpected end of file

what am i doing wrong?

Try https://shellcheck.net – Jetchisel May 06 '20 at 12:25 — Jetchisel, May 06 '20 at 12:25

KamilCuk · Accepted Answer · 2020-05-06T13:18:00.113

1

When you do << EOF the exactly whole line has to be EOF with no spaces, nor tabs, not before nor after the EOF string.

Do:

            cat << EOF >> ./algo/job_$j.sh
            #!/bin/bash -l
            #
            #$ -N $i
            #$ -cwd
            #$ -pe smp 10
            #$ -l mem_free=20G

            algo --input_se $i --output $j --threads=10
EOF
   ^ no spaces after
^ no spaces in front

Alternatively you can use << -EOF and leading tabs (and tabs only!) will be ignored.

Notes:

Do not use backticks `. Use $(..) intead. https://wiki-dev.bash-hackers.org/scripting/obsolete
for i in cat is an antipattern. How to read a file line by line in bash.

how i would use while read instead to read a file line by line in my case?

while IFS= read -r x; do
    while IFS= read -r y; do
        something something
    done < y_list
done < x_file_locations

edited May 06 '20 at 13:18

answered May 06 '20 at 12:26

KamilCuk

120,984
8
59
111

okay so I took your suggestions, the code executes but the cat function does multiple cat into one file. can you give me an example of how i would use while read instead to read a file line by line in my case? – Mohamed Momin May 06 '20 at 13:01
1

The `./algo/job_$j.sh` is the same for each `i`, so what did you expect? So write into something like `./algo/job_$i_$j.sh`. The examples how to read a file line by line are showed all in the link. – KamilCuk May 06 '20 at 13:16
@MohamedMomin, when you use `>>` redirector, the output is appended to the file, instead of substituted. If you want it not to be appended, then use a simple `>` operator, instead. – Luis Colorado May 07 '20 at 12:38

score 0 · Answer 2 · answered May 06 '20 at 14:32

Use arrays.

#!/bin/bash -l
readarray x < x_file_locations
readarray y < y_list
for i in "${x[@]}"
do for j in "${y[@]}"
   do  boilerplate=( "#!/bin/bash -l" "#" "#$ -N $i" "#$ -cwd" "#$ -pe smp 10" "#$ -l mem_free=20G" "" )
       printf "%s\n" "${boilerplate[@]}" "algo --input_se $i --output $j --threads=10" >| "./algo/job_$i-$j.sh" # need both i and j to avoid output stacking
   done
done

Luis Colorado · Answer 3 · 2020-05-07T12:33:23.080

Look at the bash (or sh(1)) doc, EOF must be left aligned to the first column of the line. You cannot indent it. If you do, it will not be recognised. This will work:

#!/bin/bash -l
x=`cat x_file_locations`
y=`cat y_list`
for i in $x
do
    for j in $y
    do
            cat << EOF >> ./algo/job_$j.sh
            #!/bin/bash -l
            #
            #$ -N $i
            #$ -cwd
            #$ -pe smp 10
            #$ -l mem_free=20G

            algo --input_se $i --output $j --threads=10
            # if you indent here, all your jobs will be
            # indented.
EOF
#^^ left aligned, don't indent it.
    done
done

You cannot even add characters after it in the same line (even a comment is disallowed) If you use FOO, the shell looks in the text for the regexp ^FOO$ (as a regexp) to match the input. Also, if you quote or delimit the pattern after the << operator, then no variable substitution will be made in the redirected text (somewhat arbitrary, but this is a legacy that predates upto version 7 UNIX or more)

This is not according to the way the shell normally parses the input, but it's very safe. Look in the doc of bash(1), for these two forms to specify the eof token (depending on if you quote it or not, after the << operator).

By the way, as commented to this answer, you had better to use three nested loops, (as you'll see about the utility of the eof token) (look at the places of the redirections, as if you place the input redirect next to the read command, you will open the file for each read (and not for the whole while loop)

#!/bin/sh

file_locations="x_file_locations"
y_list="y_list"

while read loc
do
        while read output
    do
                        # you have better not to indent thes
                        # data of the file, or you'll get an
                        # unknown reason all indented file.

            cat << EOF >> ././././job_${loc}_${output}.sh
#!/bin/bash -l
#
#$ -N ${loc}
#$ -cwd
#$ -pe smp 10
#$ -l mem_free=20G

algo --input_se "${loc}" --output "${output}" --threads=10
EOF

    done < "${y_list}"
done < "${file_locations}"

but if you want the x_file_locations and the y_list contents also be included in the shell script:

#!/bin/sh

DATE=$(date +%Y%m%d-%H%M)

while read loc
do
        while read output
    do
                # you have better not to indent thes
                # data of the file, or you'll get an
                # unknown reason all indented file.

        cat << EOF >> "././././job_${loc}_${output}.sh"
#!/bin/bash -l
#
#$ -N '${loc}'
#$ -cwd
#$ -pe smp 10
#$ -l mem_free=20G

scripts/${output}.sh --input_se "${loc}" --output "${output}.log" --threads=10
EOF

    done << EOF
Population
Overall-Temp
Rain
Vacation-Places
EOF

done << EOF
Paris
Amsterdam
Oviedo
Geneve
Zurich
Helsinki
Warsaw
Oulu
EOF

@CharlesDuffy, I'm not talking about any form of `for` loop. The OP is questioning about the shell not recognizing the `EOF` delimiter he has put in the script indented, and the shell is complainting about the EOF not found, and the command input delimited by the shell script end of file. My answer has nothing to do with using a `for` loop, but with the error he is getting from the shell. But it's the correct answer to the problem. — Luis Colorado, May 07 '20 at 11:43
Yes, it's a correct answer to the problem. My point was only that it was showcasing bad practices. (The other point I didn't have time to make at the moment is that it's a very *duplicative* question/answer pair, and should be closed rather than answered as per the "Answer Well-Asked Questions" section of [How to Answer](https://stackoverflow.com/help/how-to-answer)). — Charles Duffy, May 07 '20 at 14:39
@CharlesDuffy, as you comment, I considered your point and addressed the case of the `for` loops in my answer, and even included `here-document` versions of the `while read` loops. — Luis Colorado, May 07 '20 at 20:14

Nested for loop with cat EOF issues

3 Answers3