256

I am trying to use xargs to call a more complex function in parallel. 

#!/bin/bash
echo_var(){
    echo $1
    return 0
}
seq -f "n%04g" 1 100 |xargs -n 1 -P 10 -i echo_var {} 
exit 0

This returns the error

xargs: echo_var: No such file or directory

Any ideas on how I can use xargs to accomplish this, or any other solution(s) would be welcome.

codeforester
  • 39,467
  • 16
  • 112
  • 140
fac3
  • 2,681
  • 2
  • 14
  • 5
  • 3
    Danger, user1148366, Danger! Don't use bash for parallel programming- you will run into so many problems. Use C/C++ and pthreads, or Java threads, or anything that makes you think long and hard about what you're doing, because parallel programming takes a lot of thought to get right. – David Souther Jun 12 '12 at 19:44
  • 38
    @DavidSouther If the tasks are independent, such as convert all these picture files to png, then don't worry. It is when you have synchronisation (beyond wait for all to finish) and communication that it gets messy. – ctrl-alt-delor Feb 26 '14 at 15:41
  • 2
    @DavidSouther - I am a long time Java dev and I have been working in groovy of late. And I continue to tell people: Friends don't let friends write bash script. And yet, I find myself looking at this post/solution because (sad face :( ) I am engaged in parallel processing in bash. I could readily do it in groovy/java. Bad! – Christian Bongiorno Feb 08 '19 at 00:50
  • Also discussed in https://unix.stackexchange.com/questions/158564/how-to-use-defined-function-with-xargs – Joshua Goldberg Feb 28 '19 at 17:38

4 Answers4

245

Exporting the function should do it (untested):

export -f echo_var
seq -f "n%04g" 1 100 | xargs -n 1 -P 10 -I {} bash -c 'echo_var "$@"' _ {}

You can use the builtin printf instead of the external seq:

printf "n%04g\n" {1..100} | xargs -n 1 -P 10 -I {} bash -c 'echo_var "$@"' _ {}

Also, using return 0 and exit 0 like that masks any error value that might be produced by the command preceding it. Also, if there's no error, it's the default and thus somewhat redundant.

@phobic mentions that the Bash command could be simplified to

bash -c 'echo_var "{}"'

moving the {} directly inside it. But it's vulnerable to command injection as pointed out by @Sasha.

Here is an example why you should not use the embedded format:

$ echo '$(date)' | xargs -I {} bash -c 'echo_var "{}"'
Sun Aug 18 11:56:45 CDT 2019

Another example of why not:

echo '\"; date\"' | xargs -I {} bash -c 'echo_var "{}"'

This is what is output using the safe format:

$ echo '$(date)' | xargs -I {} bash -c 'echo_var "$@"' _ {}
$(date)

This is comparable to using parameterized SQL queries to avoid injection.

I'm using date in a command substitution or in escaped quotes here instead of the rm command used in Sasha's comment since it's non-destructive.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • 17
    A bit more discussion: xargs executes a completely new instance of the process named. In this case, you provide the name `echo_var`, which is a function in this script, not a process (program) in your PATH. What Dennis' solution does is export the function for child bash processes to use, then forks to the subprocess and executes there. – David Souther Jun 12 '12 at 19:39
  • 10
    what is the significance of `_` and ``\`` , without them it wasn't working for me – Hashbrown Oct 19 '13 at 08:16
  • 14
    @Hashbrown: The underscore (`_`) provides a place holder for `argv[0]` (`$0`) and almost anything could be used there. I think I added the backslash-semicolon (`\;`) because of its use in terminating the `-exec` clause in `find`, but it works for me without it here. In fact, if the function were to use `$@` instead of `$1` then it would see the semicolon as a parameter, so it should be omitted. – Dennis Williamson Oct 19 '13 at 11:35
  • 4
    -i argument to xargs is since been deprecated. Use -I (capital i) instead. – Nicolai S Jul 07 '15 at 20:44
  • 13
    You can simplify this by including the argument from xargs in the command string for bash with `bash -c 'echo_var "{}"'`. So you do not need the _ {} at the end. – phobic Jun 02 '16 at 08:57
  • 2
    Adding to the `-i` vs. `-I` issue mentioned by @Nicolai: [POSIX does not even mention](https://man.openbsd.org/POSIX-2013/xargs) `-i`, providing a stronger reason than "just" deprecation. – benaryorg Jul 12 '17 at 23:53
  • 1
    Would you be so kind to explain me this syntax `-I {} bash -c 'echo_var "$@"' _ {}`, please? – Daniele Aug 09 '17 at 09:41
  • 1
    @Daniele: `-I {}` sets the replacement string to a set of curly braces. In the command processed by `xargs`, the replacement string will be replaced by the arguments, each in turn. the rest of the part that you quoted is invoking Bash to run the function `echo_var` with all the arguments it receives (`$@`). The underscore at the end stands for `argv[0]` as mentioned in an earlier comment. Finally, the curly braces at the end is where the arguments passed to `xargs` (by `seq` or `printf` in this case) are substituted and then passed to the `bash` command. – Dennis Williamson Jun 14 '18 at 20:09
  • 4
    @phobic, `bash -c 'echo_var "{}"'` is bad idea. If the input line contains `"`, it won't be properly escaped. E.g. piping the `hello"; rm -rf "/` string to `xargs -I {} bash -c 'echo_var "{}"'` would lead to `bash -c 'echo_var "hello"; rm -rf "/"'`. – Sasha Aug 17 '19 at 18:17
  • I had confused with the arguments `_ {}` in the command, now I understand it, it is explained in the bash man page, see explanation for `-c` option. – Jinlxz Liu Nov 06 '20 at 08:40
  • @poige: Why do you say that? `type -a printf` outputs `printf is a shell builtin \n printf is /usr/bin/printf \n printf is /bin/printf` (which varies depending on the OS). It _is_ a builtin and it's _also_ an external executable. It's also documented in the Shell Builtin Commands section of `man bash`. The builtin is preferentially run (as evidenced by the order of the output of `type -a`). In order to run the external executable, you must specify the path (e.g. `/usr/bin/printf '%s\n' "$string"`) or use other techniques (e.g.`env printf ...`). – Dennis Williamson Oct 30 '22 at 13:19
  • @DennisWilliamson, yeah, already noted, your reply was literally one minute earlier than I came to drop the comment. ) – poige Oct 30 '22 at 13:22
16

Something like this should work also: 

function testing() { sleep $1 ; }
echo {1..10} | xargs -n 1 | xargs -I@ -P4 bash -c "$(declare -f testing) ; testing @ ; echo @ "
Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985
Eremite
  • 7,995
  • 2
  • 13
  • 6
  • 1
    In general, this would break on shell special characters (e.g. `|`, `#`) and ignore white space in input. Instead of letting bash treat the input as code, I suggest letting xargs pass them as-is. `echo {1..10} | xargs -n 1 -P4 bash -c "$(declare -f testing);"' testing "$@"; echo "$@";' argv0` – EndlosSchleife Oct 20 '21 at 16:18
4

Seems I can't make comments :-(

I was wondering about the focus on

bash -c 'echo_var "$@"' _ {}
vs
bash -c 'echo_var "{}"'

The 1st substitutes the {} as an arg to bash while the 2nd as an arg to the function. The fact that example 1 doesn't expand the $(date) is simply a a side effect.

If you don't want the functions args expanded , use single single quotes rather than double. To avoid messy nesting , use double quote (expand args on the other one)

$ echo '$(date)' | xargs -0 -L1 -I {} bash -c 'printit "{}"'
Fri 11 Sep 17:02:24 BST 2020

$ echo '$(date)' | xargs -0 -L1 -I {} bash -c "printit '{}'"
$(date)
graeme vetterlein
  • 49
  • 1
  • 1
3

Maybe this is bad practice, but you if you are defining functions in a .bashrc or other script, you can wrap the file or at least the function definitions with a setting of allexport:

set -o allexport

function funcy_town {
  echo 'this is a function'
}
function func_rock {
  echo 'this is a function, but different'
}
function cyber_func {
  echo 'this function does important things'
}
function the_man_from_funcle {
  echo 'not gonna lie'
}
function funcle_wiggly {
  echo 'at this point I\'m doing it for the funny names'
}
function extreme_function {
  echo 'goodbye'
}

set +o allexport
xdhmoore
  • 8,935
  • 11
  • 47
  • 90