xargs with multiple arguments

Question

I have a source input, input.txt

a.txt
b.txt
c.txt

I want to feed these input into a program as the following:

my-program --file=a.txt --file=b.txt --file=c.txt

So I try to use xargs, but with no luck.

cat input.txt | xargs -i echo "my-program --file"{}

It gives

my-program --file=a.txt
my-program --file=b.txt
my-program --file=c.txt

But I want

my-program --file=a.txt --file=b.txt --file=c.txt

Any idea?

Answer 1

Don't listen to all of them. :) Just look at this example:

echo argument1 argument2 argument3 | xargs -l bash -c 'echo this is first:$0 second:$1 third:$2'

Output will be:

this is first:argument1 second:argument2 third:argument3

Answer 2

None of the solutions given so far deals correctly with file names containing space. Some even fail if the file names contain ' or ". If your input files are generated by users, you should be prepared for surprising file names.

GNU Parallel deals nicely with these file names and gives you (at least) 3 different solutions. If your program takes 3 and only 3 arguments then this will work:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo a2.txt; echo b2.txt; echo c2.txt;) |
parallel -N 3 my-program --file={1} --file={2} --file={3}

Or:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo a2.txt; echo b2.txt; echo c2.txt;) |
parallel -X -N 3 my-program --file={}

If, however, your program takes as many arguments as will fit on the command line:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo d1.txt; echo e1.txt; echo f1.txt;) |
parallel -X my-program --file={}

Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Answer 3

How about:

echo $'a.txt\nb.txt\nc.txt' | xargs -n 3 sh -c '
   echo my-program --file="$1" --file="$2" --file="$3"
' argv0

Answer 4

It's simpler if you use two xargs invocations: 1st to transform each line into --file=..., 2nd to actually do the xargs thing ->

$ cat input.txt | xargs -I@ echo --file=@ | xargs echo my-program
my-program --file=a.txt --file=b.txt --file=c.txt

Answer 5

You can use sed to prefix --file= to each line and then call xargs:

sed -e 's/^/--file=/' input.txt | xargs my-program

Answer 6

Here is a solution using sed for three arguments, but is limited in that it applies the same transform to each argument:

cat input.txt | sed 's/^/--file=/g' | xargs -n3 my-program

Here's a method that will work for two args, but allows more flexibility:

cat input.txt | xargs -n 2 | xargs -I{} sh -c 'V="{}"; my-program -file=${V% *} -file=${V#* }'

Answer 7

I stumbled on a similar problem and found a solution which I think is nicer and cleaner than those presented so far.

The syntax for xargs that I have ended with would be (for your example):

xargs -I X echo --file=X

with a full command line being:

my-program $(cat input.txt | xargs -I X echo --file=X)

which will work as if

my-program --file=a.txt --file=b.txt --file=c.txt

was done (providing input.txt contains data from your example).

---

Actually, in my case I needed to first find the files and also needed them sorted so my command line looks like this:

my-program $(find base/path -name "some*pattern" -print0 | sort -z | xargs -0 -I X echo --files=X)

Few details that might not be clear (they were not for me):

• some*pattern must be quoted since otherwise shell would expand it before passing to find.
• -print0, then -z and finally -0 use null-separation to ensure proper handling of files with spaces or other wired names.

Note however that I didn't test it deeply yet. Though it seems to be working.

Answer 8

xargs doesn't work that way. Try:

  myprogram $(sed -e 's/^/--file=/' input.txt)

Answer 9

It's because echo prints a newline. Try something like

echo my-program `xargs --arg-file input.txt -i echo -n " --file "{}`

Answer 10

Actually, it's relatively easy:

... | sed 's/^/--prefix=/g' | xargs echo | xargs -I PARAMS your_cmd PARAMS

The sed 's/^/--prefix=/g' is optional, in case you need to prefix each param with some --prefix=.

The xargs echo turns the list of param lines (one param in each line) into a list of params in a single line and the xargs -I PARAMS your_cmd PARAMS allows you to run a command, placing the params where ever you want.

So cat input.txt | sed 's/^/--file=/g' | xargs echo | xargs -I PARAMS my-program PARAMS does what you need (assuming all lines within input.txt are simple and qualify as a single param value each).

Answer 11

There is another nice way of doing this, if you do not know the number of files upront:

my-program $(find . -name '*.txt' -printf "--file=%p ")

Answer 12

I was looking for a solution for this exact problem and came to the conclution of coding a script in the midle.

to transform the standard output for the next example use the -n '\n' delimeter

example:

 user@mybox:~$ echo "file1.txt file2.txt" | xargs -n1 ScriptInTheMiddle.sh

 inside the ScriptInTheMidle.sh:
 !#/bin/bash
 var1=`echo $1 | cut -d ' ' -f1 `
 var2=`echo $1 | cut -d ' ' -f2 `
 myprogram  "--file1="$var1 "--file2="$var2 

For this solution to work you need to have a space between those arguments file1.txt and file2.txt, or whatever delimeter you choose, one more thing, inside the script make sure you check -f1 and -f2 as they mean "take the first word and take the second word" depending on the first delimeter's position found (delimeters could be ' ' ';' '.' whatever you wish between single quotes . Add as many parameters as you wish.

Problem solved using xargs, cut , and some bash scripting.

Cheers!

if you wanna pass by I have some useful tips http://hongouru.blogspot.com

Answer 13

Nobody has mentioned echoing out from a loop yet, so I'll put that in for completeness sake (it would be my second approach, the sed one being the first):

for line in $(< input.txt) ; do echo --file=$line ; done | xargs echo my-program

Answer 14

Old but this is a better answer:

cat input.txt | gsed "s/\(.*\)/\-\-file=\1/g" | tr '\n' ' ' | xargs my_program

# i like clean one liners gsed is just gnu sed to ensure syntax matches version brew install gsed or just sed if your on gnu linux already...

test it:

cat input.txt | gsed "s/\(.*\)/\-\-file=\1/g" | tr '\n' ' ' | xargs echo my_program

Answer 15

I discovered a more straightforward way. You can even customize the order of positional parameters to your liking, like in the demonstration below.

set $(cat input.txt)
my-program --file=$1 --file=$3 --file=$2

This works well in scenarios where you know the order and magnitude of arguments that are in the input.

Answer 16

check this out too :

while IFS= read -r file; do
  my-program --file="$file"
done < input.txt

the while loop reads each line from the input.txt file and passes it as an argument to the my-program command with the --file flag!

Answer 17

Most answers don't take into account that some characters in the input lines can break xargs and unquoted $(...). Here are a few solutions that do the job correctly (i.e. robustly).

---

1. PURE BASH

You can load the input lines into a bash array and prepend --file= to each element with a parameter expansion. WARNING! You can't use this method for generating more than one argument per array element (for e.g. ‑f xxx.txt).

readarray -t files < input.txt

my-program "${files[@]/#/--file=}"

note: if your bash doesn't have readarray/mapfile then you can use IFS=$'\n' read -r -d '' -a files < input.txt instead.

---

2. AWK | XARGS -0

You can use awk for prepending --file= to each line and generate a NUL-delimited stream that'll be safely processed by xargs -0:

awk '{printf("--file=%s%c",$0,0)}' input.txt |
xargs -0 my-program

---

3. SED | XARGS

If you ever need to be POSIX compliant:

sed -e 's/"/"\\""/g' -e 's/.*/--file="&"/' input.txt |
xargs my-program