0

I have a program that outputs a list of paths. Program 1 can be simulated with

echo '"Calibre Library" "VirtualBox VMs"'

or a C++ program

#include <iostream>
int main() {
    std::cout << R"("Calibre Library" "VirtualBox VMs")";
}

That means that calling program 1 generates the output

"Calibre Library" "VirtualBox VMs"

I want to pass the the output of this program to another program. The call of program 2 can be simulated with

ls $(echo '"Calibre Library" "VirtualBox VMs"')

or the C++ program

ls $(./a.out)

The paths are correct. That means

ls "Calibre Library" "VirtualBox VMs"

works and two arguments are passed to the program call. But

ls $(echo '"Calibre Library" "VirtualBox VMs"')

and

ls $(./a.out)

don't work

ls: cannot access '"Calibre': No such file or directory
ls: cannot access 'Library"': No such file or directory
ls: cannot access '"VirtualBox': No such file or directory
ls: cannot access 'VMs"': No such file or directory

because 4 arguments are passed to the call.

Why does it behave differently and is it possible to solve it without changing both programs?

3 Answers3

0

You can use program1 | xargs program2 to solve your problem

echo '"Calibre Library" "VirtualBox VMs"' | xargs -exec ls

echo '"Calibre Library" "VirtualBox VMs"' | xargs ls
Victor Lee
  • 2,467
  • 3
  • 19
  • 37
  • 1
    What is `-exec` supposed to do? My `man xargs` doesn't list it. My `xarg` interprets `-exec` as `-e xec` which is `-eof=xec` and causes `xargs` to ignore every argument after `xec`. – Socowi Apr 08 '21 at 12:51
  • 1
    Yes, it works with `echo '"Calibre Library" "VirtualBox VMs"' | xargs -exec ls` and `a.out | xargs -exec ls`. But it also works without `-exec`: `echo '"Calibre Library" "VirtualBox VMs"' | xargs ls` and `a.out | xargs ls` –  Apr 08 '21 at 12:53
  • I confuse it, `-exec` often used with `find`.More info from https://stackoverflow.com/questions/896808/find-exec-cmd-vs-xargs – Victor Lee Apr 08 '21 at 13:48
0

Double quotes only work when they are literally present in the outermost command being run. If you include some string expression that just contains double quotes, those are just ordinary characters in the value as far as the shell is concerned.

While this sees two arguments:

set -- "first word" "second word"; echo $#
#=> 2

This sees four:

set -- $(echo "first word" "second word"); echo $#
#=> 4

And this sees only one:

set -- "$(echo "first word" "second word")"; echo $#
#=> 1

And making the echo output literal quotation marks doesn't change any of that; you just get quotation marks in your parameters:

set -- $(echo '"first word" "second word"')
echo $#
#=> 4
echo $1
#=> "first 

set -- "$(echo '"first word" "second word"')"
echo $#
#=> 1
echo $1
#=> "first word" "second word"

The upshot is that if you want the output of a command to create words that are easily separable by the shell, you should not output double-quoted strings; that's going to require an extra level of evaluation on the output, which is possible but opens you up to all sorts of potential mischief by the users running your script. Instead, have the command output plain strings, with no quotation marks around them, but separated by some character that doesn't appear inside any of the individual elements. A newline is great if none of your strings contain newlines:

$ ./a.out
Calibre Library
VirtualBox VMs

But you can use any convenient character that doesn't appear in any of the values; NUL (character code 0, usually represented in strings as "\0") is a popular choice because it's almost never needed inside a string and often isn't even allowed, depending on the program you're passing it to.

Once you have a command that does that, you can do a couple of things. If all you want is to turn its output into arguments that get passed to another command directly, you can use xargs:

./a.out | xargs ls 

If it outputs NUL-separated strings instead of newline-separated ones, use the -0 parameter:

./a.out | xargs -0 ls

More generally, if you want to keep a list of strings around in the shell, you should store them in an array. You can set one to literal values like this:

files=("Calibre Library" "VirtualBox VMs")

Or you can use mapfile to set it from the output of a command that prints out strings suitable for input to xargs:

mapfile -t files < <(./a.out)

Or, if they're NUL-delimited:

IFS=$'\0' mapfile -t files < <(./a.out)

Then you can use the special expression "${files[@]}" to get the individual elements of the array back out and pass them to another command:

ls "${filenames[@]}"

The ${arrayName[@]} expression is somewhat magical in that the double-quotes around it are effectively inherited by each individual element of the array instead of just grouping the whole thing into one big word. (If you want it all as one big word, which is generally less useful, you can still get that by using [*] instead of [@]).

Mark Reed
  • 91,912
  • 16
  • 138
  • 175
0

ls $(echo '"Calibre Library" "VirtualBox VMs"') and ls $(./a.out) don't work.

... and there is no way to make it work with only a subshell $(). Subshells expand to

  • either one argument if you quote $():
    ls "$(echo everything in here is one argument for ls)"
  • or one argument per word if you don't quote $():
    ls $(echo 'this yields five words " ')
    How bash splits the output of echo into words is controlled by IFS. Quotes in echo's output have no special meaning at this point. But globs like * still do, so that might result in even more problems.

If I understood you correctly, you want to write a program (here a.out) that generates a list of arguments and feed that list to another program (here ls). Of course you could do one of the following things:

  • Output one argument per line, then read those lines and pass them to ls,
    e.g. by using echo $'Calibre Library\nVirtualBox VMs' | sed 's/./\\&/' | xargs ls.
  • Output the arguments in correctly quoted manner, then use eval (dangerous!)
    eval ls "$(echo '"Calibre Library" "VirtualBox VMs"')"

But both approaches put a big burden on the user because they have to use $(), eval and so on. Programs like xargs and GNU parallel do very similar things as you plan to do. Both of these take the command to be executed (e.g. ls) as an argument and then execute it themself (for instance by using system() in C++) instead of relying on the user to parse their output into arguments. I think this would be the way to go here. Usage could be something like ...

yourProgram program to be executed {placeholder for args to be inserted} :::: other arguments for your program
Socowi
  • 25,550
  • 3
  • 32
  • 54