0

I want to iterate through a list of videos and convert them from .avi to .mp4 using a script, which takes about ~1 min for each file.

find . -name "*.avi" -type f | while read FILE; do
  HandbrakeCLI [flags] $FILE # execute 30 sec ~ 2 min function here
done

problem is, this seems to be only running on the first file, as though successive invocations are blocks.

How can I spawn asynchronous threads to handle each case?

This is a duplicate of that other issue which is trying to solve new processes of ssh, and the solution is based on arguments available to ssh. I am asking for a way to spawn new bash threads in an iterator, maybe theres a duplicate ticket that can direct me that way?

user2167582
  • 5,986
  • 13
  • 64
  • 121
  • 1
    Where do you use variable $FILE? – Cyrus Sep 17 '16 at 08:36
  • Just a side note. The loop is not required. You can use the `-exec` option of `find`: `find . -name "*.avi" -type f -exec HandbrakeCLI ... {} \;` – hek2mgl Sep 17 '16 at 08:37
  • With ampersand `&` as last character in the commandline: `HandbrakeCLI &`. That starts a new thread for every command. – PeterMmm Sep 17 '16 at 08:47
  • You can achieve that using [GNU Parallel][1]. For example, here is command to run 2 instances in parallel. ` find . -name "*.avi" -type f | parallel -j 2 "HandbrakeCLI {}"`. You may change `-j 2` parameter to change number of parallel instances. [1]: https://www.gnu.org/software/parallel/ – alexander Sep 17 '16 at 09:02
  • @Cyrus sorry for being too concise. – user2167582 Sep 17 '16 at 10:37

1 Answers1

2

Assuming your issue is the HandbreakCLI script is only called once, the root cause might be it behaves differently depending on whether the standard input is a terminal or not, or maybe does it drains its stdin. In such case, here is a way to overcome the issue from within find:

find . -type f -name  "*.avi" -exec sh -c 'FILE="$1" HandbrakeCLI &' sh {} \;

To trace what this command is running, you can modify it that way:

find . -type f -name  "*.avi" -exec sh -c 'set -x; FILE="$1" HandbrakeCLI &' sh {} \;
jlliagre
  • 29,783
  • 6
  • 61
  • 72
  • Not really. Trying out answering with a phone... Not easy. – jlliagre Sep 17 '16 at 08:46
  • Phone or not, the point is that this is doing the same as the while loop in the question. – hek2mgl Sep 17 '16 at 08:49
  • @hek2mgl Well, to be honest, my first answer wasn't even doing the same as the OP loop as the script seems to pick its argument from a variable name, not a parameter. My first guess was the script was draining its standard input and preventing remaining occurrences to occur, but it looks I misunderstood the "are blocks" statement. – jlliagre Sep 17 '16 at 09:05
  • @hek2mgl Please do not edit my reply to change the script behavior. The OP script is passing the file to process as an environment variable, not as an argument. – jlliagre Sep 17 '16 at 09:14
  • Well, I've overlooked that particular point. I was mainly focussing on fixing syntax errors. What is the purpose of using backticks? What is the purpose of the `sh {} \;` at the end of the line? – hek2mgl Sep 17 '16 at 09:23
  • @hek2mgl backticks were an issue with my phone keyboard, thanks for pointing it, fixed. The `sh {} \;` at the end of the script is a portable way to pass the filename as `{}` is only guaranteed to supported when alone in the line, per the POSIX standard. – jlliagre Sep 17 '16 at 09:28
  • Can you give a link to that POSIX document? I'm pointing to the additional `sh`, just to make that clear. – hek2mgl Sep 17 '16 at 09:31
  • The "sh" is a convention, any string would work here although `sh` is less confusing that say `fubar`. It is passed as argument 0 to the shell. See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117 , "sh -c" description and in particular "Set the value of special parameter 0". – jlliagre Sep 17 '16 at 09:37
  • Oh, I see your point, however, I strongly recommend to use `-` for that. (Or `--` if you want). That's far better to read. You see, at least I was confused by `sh` – hek2mgl Sep 17 '16 at 09:38
  • 1
    Using `sh` is probably the most used convention. `-` would suggest it is a login shell, which it is not. – jlliagre Sep 17 '16 at 09:40
  • Ok, then it is probably me who is confusing others. :) The answer looks good now. +1 I would still recommend to not answer from a phone. Or learn typing! :P – hek2mgl Sep 17 '16 at 09:42
  • 1
    @hek2mgl Thanks, I'm learning by practice ;) – jlliagre Sep 17 '16 at 09:45
  • @jlliagre I am getting empty results from running the command you gave me and getting empty lines. – user2167582 Sep 18 '16 at 03:10
  • This command is not supposed to display anything by itself. If there are empty lines, either HandbrakeCLI is silent, or there are no `.avi` files to process. Answer updated to allow tracing the script execution. – jlliagre Sep 18 '16 at 08:30