GNU Parallel and Bash functions: How to run the simple example from the manual

Question

I'm trying to learn GNU Parallel because I have a case where I think I could easily parallelize a bash function. So in trying to learn, I went to the GNU Parallel manual where there is an example...but I can't even get it working! To wit:

(232) $ bash --version
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
(233) $ cat tpar.bash
#!/bin/bash

echo `which parallel`
doit() {
  echo Doing it for $1
  sleep 2
  echo Done with $1
}
export -f doit
parallel doit ::: 1 2 3
doubleit() {
  echo Doing it for $1 $2
  sleep 2
  echo Done with $1 $2
}
export -f doubleit
parallel doubleit ::: 1 2 3 ::: a b

(234) $ bash tpar.bash
/home/mathomp4/bin/parallel
doit: Command not found.
doit: Command not found.
doit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.

As you can see, I can't even get the simple example to run. Thus, I'm probably doing something amazingly stupid and basic...but I'm at a loss.

ETA: As suggested by commenters (chmod +x, set -vx):

(27) $ ./tpar.bash

echo `which parallel`
which parallel
++ which parallel
+ echo /home/mathomp4/bin/parallel
/home/mathomp4/bin/parallel

doit() {
  echo Doing it for $1
  sleep 2
  echo Done with $1
}
export -f doit
+ export -f doit
parallel doit ::: 1 2 3
+ parallel doit ::: 1 2 3
doit: Command not found.
doit: Command not found.
doit: Command not found.
doubleit() {
  echo Doing it for $1 $2
  sleep 2
  echo Done with $1 $2
}
export -f doubleit
+ export -f doubleit
parallel doubleit ::: 1 2 3 ::: a b
+ parallel doubleit ::: 1 2 3 ::: a b
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.
doubleit: Command not found.

ETA2: Note, I can, in the script, just call 'doit 1', say, and it will do that. So the function is valid, it just isn't...exported?

Are you sure you have GNU parallel, and not the version by Tollef Fog Heen? — Fred Foo, May 22 '14 at 18:27
@larsmans, It is GNU parallel: (247) $ parallel --version GNU parallel 20140522 — Fortran, May 22 '14 at 18:28
It works fine on my Mac under OSX. Have you made `tpar.bash` executable with `chmod +x tpar.bash` — Mark Setchell, May 22 '14 at 18:32

Gilles 'SO- stop being evil' · Accepted Answer · 2014-05-22T21:39:52.113

You cannot call a shell function from outside the shell where it was defined. A shell function is a concept inside the shell. The parallel command itself has no way to access it.

Calling export -f doit in bash exports the function via the environment so that it is picked up by child processes. But only bash understands bash functions. A (grand)*child bash process can call it, but not other programs, for example not other shells.

Going by the message “Command not found”, it appears that your preferred shell is (t)csh. You need to tell parallel to invoke bash instead. parallel invokes the shell indicated by the SHELL environment variable¹, so set it to point to bash.

export SHELL=$(type -p bash)
doit () { … }
export -f doit
parallel doit ::: 1 2 3

If you only want to set SHELL for the execution of the parallel command and not for the rest of the script:

doit () { … }
export -f doit
SHELL=$(type -p bash) parallel doit ::: 1 2 3

I'm not sure how to deal with remote jobs, you may need to pass --env=SHELL in addition to --env=doit (note that this assumes that the path to bash is the same everywhere).

And yes, this oddity should be mentioned more prominently in the manual. There's a brief note in the description of the command argument, but it isn't very explicit (it should explain that the command words are concatenated with a space as a separator and then passed to $SHELL -c), and SHELL isn't even listed in the environment variables section. (I encourage you to report this as a bug; I'm not doing it because I hardly ever use this program.)

¹ _{which is bad design, since SHELL is supposed to indicate a user interface preference for an interactive command line shell, and not to change the behavior of programs.}

@larsmans `export` at the top of the script is simpler, but indeed setting `SHELL` just for the `parallel` command (and its subprocesses) is also an option. — Gilles 'SO- stop being evil', May 22 '14 at 21:36

Ole Tange · Answer 2 · 2019-10-11T07:01:54.063

15

Since version 20160722 you can instead use env_parallel:

doit() { echo "$@"; }
echo world | env_parallel doit Hello

You just need to activate env_parallel by adding it to .bashrc. You can add it to .bashrc by running this once:

env_parallel --install

edited Oct 11 '19 at 07:01

answered Jan 10 '18 at 23:31

Ole Tange

31,768
5
86
104

Note that `env_parallel --install` will append a line that runs `which env_parallel.bash` to `.bashrc`, so it only needs to be run once. – sboysel Oct 10 '19 at 23:39

GNU Parallel and Bash functions: How to run the simple example from the manual

2 Answers2

Linked

Related