Shell script while read loop executes only once

Question

I am writing a shell script to batch process .mov files off my camera through Handbrake to save HD space. The script searches a directory with 'find' and then runs Handbrake on each .mov file found, matching the creation date of the resulting file to the source file's date with 'touch'.

I originally did this with a for loop:

for i in $(find "$*" -iname '*.mov') ; do
  ~/Unix/HandbrakeCLI --input "$i" --output "$i".mp4 --preset="Normal"
  touch -r "$i" "$i".mp4
done

This worked, but failed if the input files had spaces in their file names. So I tried a while loop instead:

find "$*" -iname '*.mov' | while read i ; do
  ~/Unix/HandbrakeCLI --input "$i" --output "$i".mp4 --preset="Normal"
  touch -r "$i" "$i".mp4
done

The problem with this loop is that it works for the first file in the directory, and then exits the loop. Note that if I substitute an "echo $i" in the body of the while loop, it prints all of the .mov files in the directory, so the loop is structured correctly.

I believe there is a partial answer to my question on this stackoverflow thread. But the solution is specific to ssh and doesn't solve the general problem. Seems to have something do do with stdin being used by a sub-process, but I don't exactly understand how this works.

Any advice?

I'm on OSX 10.6

This can't be answered without knowing what $* is supposed to be. Is it supposed to be a list of files or directories? Or the current directory? Looks very much like you just use the wrong `file` argument with the wrong shell special parameter (e.g. use `file "$@"` instead of `"$*"`). — Jens, Aug 18 '11 at 21:15
`"$*"` is quite certainly wrong; you definitely want `"$@"` instead. — Charles Duffy, Jul 01 '13 at 21:53

score 3 · Answer 1 · edited May 23 '17 at 12:16

3

Taken from this answer: I now echo nothing into HandbrakeCLI to ensure it's not using the same stdin as my script:

find . -name "*.mkv" | while read FILE
do
    echo "" | handbrake-touch "$FILE"

    if [ $? != 0 ]
    then
        echo "$FILE had problems" >> errors.log  
    fi
done

...and it works as intended/expected.

edited May 23 '17 at 12:16

Community

1
1

answered Jul 01 '13 at 21:47

Rodrigo Polo

4,314
2
26
32

l0b0 · Answer 2 · 2013-07-09T08:29:06.080

2

One possibility is to use the safe find:

while IFS= read -r -d '' -u 9
do
    ~/Unix/HandbrakeCLI --input "$REPLY" --output "$REPLY".mp4 --preset="Normal"
    touch -r "$REPLY" "$REPLY".mp4
done 9< <( find "$@" -type f -print0 )

This should be POSIX compatible, but only works if neither HandbrakeCLI nor touch read from standard input and no file names contain newlines:

find "$@" -type f -print0 | while IFS= read -r
do
    ~/Unix/HandbrakeCLI --input "$REPLY" --output "$REPLY".mp4 --preset="Normal"
    touch -r "$REPLY" "$REPLY".mp4
done

edited Jul 09 '13 at 08:29

answered Apr 06 '11 at 12:27

l0b0

55,365
30
138
223

I tried running this and got an error: "syntax error near unexpected token `<' I should note that I don't understand what the first two lines mean, so I'm not able to fix it myself. – beibei2 Apr 08 '11 at 10:46
that means that you are probably not on bash. Please report output of 'echo $0' in the context of the script? Or, what version of bash is this (`bash --version`) – sehe Apr 08 '11 at 11:13
bash version: GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin10.0) echo $0: /Volumes/Macintosh HD/Users/N/Unix/bin/nsqueezemovs – beibei2 Apr 11 '11 at 04:06
@beibei Does your script start with `#!/bin/sh` or `#!/bin/bash`? It needs to be the latter for `<()` to work. – Charles Duffy Jul 01 '13 at 21:54

score 1 · Answer 3 · answered Apr 05 '11 at 09:21

1

You are probably running with a shellopt '-e' (exit on errors)

Try

set +e

answered Apr 05 '11 at 09:21

sehe

374,641
47
450
633

1

Setting +e did not make any difference. Handbrake executes once on the first file in the while-read loop and then breaks out of the loop and executes the remainder of the script. I don't thing the problem has to do with errors, because handbrake completes successfully. – beibei2 Apr 08 '11 at 10:44
Could you try to do the Handbrake step in a subshell (`Handbrake`) – sehe Apr 08 '11 at 11:14
I tried running handbrake in a subshell, but it had no effect on the problem I'm having. I tried subshelling the entire loop, and also just the call to handbrake. Same result both ways. – beibei2 Apr 11 '11 at 04:06

score 1 · Answer 4 · edited May 23 '17 at 11:53

1

This link fixed the problem for me on Ubuntu Linux 11.04 .

Preventing a child process (HandbrakeCLI) from causing the parent script to exit

Here is the code that works for me:

ls -t *.mpeg | tail -n +2 | while read name
do echo "$name" ; in="$name" ; out=coded/"`echo $name | sed 's/.mpeg$/.mkv/'`"
echo "" | HandBrakeCLI -i "$in" -o "$out" -a '1,2' ; mv -v "$name" trash/"$name"
done

edited May 23 '17 at 11:53

Community

1
1

answered Aug 18 '11 at 19:02

gabkdlly

111
3

score 0 · Answer 5 · answered Oct 07 '12 at 22:08

I've got the same problem. I think HandbrakeCLI is consuming stdin. I don't have my actual system here, but my testing shows that's what is probably happening.

"output" is a file with the numbers 1-10, each on their own line.

while read line ; do echo "MAIN: $line"; ./consumer.sh; done < output

consumer.sh:

#!/bin/sh

while read line ; do
  echo "CONSUMER: $line"
done

Results:

MAIN: 1
CONSUMER: 2
CONSUMER: 3
CONSUMER: 4
CONSUMER: 5
CONSUMER: 6
CONSUMER: 7
CONSUMER: 8
CONSUMER: 9
CONSUMER: 10

Changing the outer while loop will fix the problem:

while read line ; do echo "MAIN: $line"; ./consumer.sh < /dev/null; done < output

Results:

MAIN: 1
MAIN: 2
MAIN: 3
MAIN: 4
MAIN: 5
MAIN: 6
MAIN: 7
MAIN: 8
MAIN: 9
MAIN: 10

score 0 · Answer 6 · answered Jan 08 '13 at 21:08

0

Consider just using find to invoke a shell, instead of wrapping find in a while loop.

find "$@" -iname '*.mov' -exec sh -c '
    ~/Unix/HandbrakeCLI --input "$1" --output "$1".mp4 --preset="Normal"
    touch -r "$1" "$1".mp4
' _ {} ';'

answered Jan 08 '13 at 21:08

Josh Cartwright

771
4
7

Thanks Josh. Any chance you could elaborate on the meaning of the final line? – beibei2 Jan 09 '13 at 22:05
Sure! It is, in a clean way, passing arguments to the inline script. Consider `sh -c 'echo "$1"' _ "hello,world"`. `sh` will run the basic, inline script `echo "$1"` passing as its arguments `_` and `hello,world`. By convention, the 0th argument should hold the pathname to the command itself, which in this case doesn't make sense, so `_` is just a useless placeholder. Hopefully that helps! – Josh Cartwright Jan 10 '13 at 00:03

Shell script while read loop executes only once

6 Answers6

Linked

Related