First to mention that I do not have lot of experiences with Bash scripting.
Here is the problem that I observe:
When I execute read
command and inside the cycle I run background processes, the read
command misses some of the arguments in some very rare cases.
For example: if I read
the output of ls -la
for big number of video files and on each of them I execute ffmpeg
command in a different sub-process, then in some very rare cases there are missing some of the first parameters of read
command.
In that case the rest of parameters of the ls
are wrong (having partial of their real values or wrongly assigned).
I most of the cases I have an output like this (which is correct):
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_453.mp4'
but for very few lines I have not correct output and it is like this:
p1: 'an'; p2: '1080519'; p3: 'Jan'; p4: '27'; p5: '05:49'; p6: 'origVideo_454.mp4'; p7: ''; p8: ''; p9: ''
Here p1
and p2
are missing and p3
should be "uman"
but is just "an"
. And p3
becomes p1
, p4
becomes p2
, etc, in this way p7
, p8
and p9
remain without values.
Here is my bash script:
#!/bin/bash
#src_dir=/tmp/text_files
src_dir=/tmp/video_files
dest_dir=/tmp/video_files_dest
mkdir -p $dest_dir
handle_video() {
echo "handling file: '$1'"
ffmpeg -loglevel error -i $src_dir/$1 -acodec copy -vcodec copy $dest_dir/$1
}
generate_text() {
str=''
for k in {1..512}
do
random=$(openssl rand -hex 20)
str="${str}${random} "
if [ $(( $k % 4 )) -eq 0 ]; then
str="${str} ${new_line}"
fi
done
echo "${str}" > $src_dir/$1
}
while read p1 p2 p3 p4 p5 p6 p7 p8 p9; do
echo "p1: '$p1'; p2: '$p2'; p3: '$p3'; p4: '$p4'; p5: '$p5'; p6: '$p6'; p7: '$p7'; p8: '$p8'; p9: '$p9'"
if test -f $src_dir/$p9; then
handle_video $p9 &
# generate_text $p9 &
fi
done << EOF
$(ls -la $src_dir)
EOF
**When I run the `handle_video` not in background but in same thread** I do not have such problem (remove `&` from line 33).
First I thought the issue might be in the output of the command `la -ls` and I tried with other commands, but I saw the same kind of results - in most of the executions `read` has correct parameters but in very few cases they are wrong.
I also tried the script instead with handle_video
(which uses ffmpeg
invocation) to run different function that is executed inside read
cycle: the generate_text
.
To do this I comment lines 3 and 33 and uncomment lines 4 and 34.
And the interesting thing is that when executing it with handle_video
problem exists but when executing with generate_text
there is no such problem at all. At least I have never observed it in all my tests.
When executing it with handle_video
I put 1200 video .mp4 files (1.1 MB each) in directory /tmp/video_files
and run bash script.
When executing it with generate_text
I generate 1200 empty files in directory /tmp/text_files
and run the bash script.
I also tried to execute the read
command with piping like this, but the result is the same:
ls -la $src_dir | while read p1 p2 p3 p4 p5 p6 p7 p8 p9; do
echo "p1: '$p1'; p2: '$p2'; p3: '$p3'; p4: '$p4'; p5: '$p5'; p6: '$p6'; p7: '$p7'; p8: '$p8'; p9: '$p9'"
if test -f $src_dir/$p9; then
handle_video $p9 &
# generate_text $p9 &
fi
done
Bash version is: 5.2.15(1)-release
ffmpeg version 5.0.2
Guest OS: Fedora version: "36 (Workstation Edition)"
VirtualBox 7.0.2
Host OS: is Windows 10 version: 21H2
Once again when I do not run the function handle_video
in background (at line 33 remove the ampersand &
) there are no problems.
And when I use instead of handle_video
the function generate_text
again there are no problems.
So I wonder is there problem in the read
method and how it gets the arguments, or is there problem with bash how it is being executing multiple processes, or there is something that I do not understand.
Any help and tips are appreciated.
Here is a snippet of real output:
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_453.mp4'
handling file: 'origVideo_448.mp4'
handling file: 'origVideo_449.mp4'
handling file: 'origVideo_44.mp4'
handling file: 'origVideo_450.mp4'
handling file: 'origVideo_451.mp4'
handling file: 'origVideo_452.mp4'
p1: 'an'; p2: '1080519'; p3: 'Jan'; p4: '27'; p5: '05:49'; p6: 'origVideo_454.mp4'; p7: ''; p8: ''; p9: ''
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_455.mp4'
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_456.mp4'