xargs not working with built in shell functions

Question

I am trying to speed up the processing of a database. I migrated towards xargs. But I'm seriously stuck. Piping a list of arguments to xargs does not work if the command invoked by xargs isn't a built in. I can't figure out why. Here is my code:

#!/bin/bash

list='foo
bar'

test(){
    echo "$1" 
}

echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' 'test' {}

So there is no output at all. And test function never gets executed. But if I replace "test" in the "xargs" command with "echo" or "printf" it works fine.

xargs takes an executable as an argument (including custom scripts) rather than a function defined in the environment. It might help to explain the bigger problem. — Alastair McCormack, Apr 13 '15 at 17:55
Thanks. Putting my commands inside a script solved the problem. — Sgt.Doakes, Apr 13 '15 at 18:01
Incidentally, don't make the mistake of calling your own things `test`; it's a shell builtin (also known as `[`). — tripleee, Apr 13 '15 at 18:09
Your problem description is diametrically wrong, by the way. `xargs` cannot use a builtin, but it can use any external command such as `ls` or `echo` (which is also a built-in in many modern shells, but still available as `/bin/echo` in order for, well, *this* to work) or `printf`. — tripleee, Apr 13 '15 at 18:10

tripleee · Answer 1 · 2017-05-30T17:10:53.257

You can't pass a shell function to xargs directly, but you can invoke a shell.

printf 'foo\0bar\0' |
xargs -r -0 sh -c 'for f; do echo "$f"; done' _

The stuff inside sh -c '...' can be arbitrarily complex; if you really wanted to, you could declare and then use your function. But since it's simple and nonrecursive, I just inlined the functionality.

The dummy underscore parameter is because the first argument after sh -c 'script' is used to populate $0.

Because your question seems to be about optimization, I imagine you don't want to spawn a separate shell for every item passed to xargs -- if you did, nothing would get faster. So I put in the for loop and took out the -I etc arguments to xargs.

Notice also the `-r` to prevent `xargs` from running anything if it receives empty input. — tripleee, Aug 08 '17 at 05:17

score 5 · Accepted Answer · answered Apr 13 '15 at 18:04

5

xargs takes an executable as an argument (including custom scripts) rather than a function defined in the environment.

Either move your code to a script or use xargs to pass arguments to an external command.

answered Apr 13 '15 at 18:04

Alastair McCormack

26,573
8
77
100

score 5 · Answer 3 · answered Mar 04 '17 at 15:47

5

Change from:

echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' 'test' {}

To:

export -f test
echo "$list" | tr '\012' '\000' | xargs -0 -n1 -I '{}' sh -c 'test {}'

answered Mar 04 '17 at 15:47

Kur00Hazama

619
6
4

Dick Guertin · Answer 4 · 2017-12-11T22:15:27.590

I've seen a solution from 'jac' on the bbs.archlinux.org web-site that uses a primary and secondary (slave) pair of scripts that are very efficients. Instead of an internal 'function' that normally would accept a single $1 parameter, the primary sends a list of parameters to its secondary where a while-loop handles each member of the list as consecutive $1 values. Here's a sample pair I'm using to apply the 'file' command to a bunch of executables, which in my case all begin with "em" in the filename. Make changes as necessary:

#!/bin/bash
# primary:  showfil
ls -l em* | grep  '^-rwx' | awk '{$1=$2=$3=$4=$5=$6=$7=$8=""; print $0}' | xargs -I% ~/showfilf "%"
~/showfilf fixmstr spisort trc
exit 0

#!/bin/bash
# secondary:  showfilf
myarch=$(uname -s | grep 'arwin')
while [[ -n "$1" ]]; do
  if [ -x "$1" ]; then
     if [ -n "$myarch" ]; then
        file "./$1"
     else
        myfile=$(file "./$1" | awk '{print $1" "$3" "$10" "$11" "$12}')
        myfile=${myfile%(uses}
        myfile=${myfile%for}
        echo "$myfile"
     fi
  fi
  shift
done
exit 0

This code works on Darwin (Mac) and Linux, and probably other systems. The 'grep' in the primary retains only executable files, not directories or symlinks. The 'awk' eliminates the first eight fields of 'ls' and retains just the filename,which is passed to 'xargs', which builds a list of quoted filenames to send to 'showfilf'. There's a separate invocation of 'showfilf' with three other filenames in the list. 'showfilf' has a while-loop which processes the list. Note that there is system-dependent code here, determined by 'uname -s' and 'grep'. Lastly, make these scripts executable, and place them on your $PATH, such as $HOME. If your $PATH doesn't include your $HOME, I recommend you modify it in your .bashrc or .bash_login something like this: export PATH=$PATH:$HOME

xargs not working with built in shell functions

4 Answers4

Linked