How to pipe the content of a variable as STDIN in a qx{} statement in Perl?

Question

I basically would like to do this:

$_ = "some content that need to be escaped &>|\"$\'`\s\\";
qx{echo $_ | foo}

There are two problems here. First the content of $_ needs to be escaped as it can contain binary data. Second, invoking echo might be slightly inefficient.

How can I simply pipe some content as STDIN to a command in Perl?

So you want to use `foo` to change the content of `$_`. Is this in a one-liner? — simbabque, Oct 21 '16 at 09:39
Well, yes it looks like a one-liner, but I want to do in in a Perl script. I am processing some files through some filters before doing something else with the content. — nowox, Oct 21 '16 at 09:43
In a one-liner I'd probably invoke perl twice. Something like `$ echo "foobar" | perl -pe 's/f/ff/' | wc -c | perl -pe '$_ *= 2'` so you don't have to care about all of the escaping. — simbabque, Oct 21 '16 at 09:45
@simbabque Yes of course as one-liner it's easy. My current workaround is to save `$_` to a temporary file then use `qx{cat $temp | foo}` which is ugly. — nowox, Oct 21 '16 at 09:51
@PerlDuck yes, it would. You came too late, the answer where we discussed that in a comment has already been deleted and you can't see those yet. — simbabque, Oct 21 '16 at 09:57
Regarding the temporary file, why not do `qx{foo <$temp_filename}`? Then you don't need `cat`. — simbabque, Oct 21 '16 at 09:58

ikegami · Answer 1 · 2016-10-21T19:41:11.047

The following assume @cmd contains the program and its arguments (if any).

my @cmd = ('foo');

If you want to capture the output, you can use any of the following:

use String::ShellQuote qw( shell_quote );
my $cmd1 = shell_quote('printf', '%s', $_);
my $cmd2 = shell_quote(@cmd);
my $output = qx{$cmd1 | $cmd2};

use IPC::Run3 qw( run3 );
run3(\@cmd, \$_, \my $output);

use IPC::Run qw( run );
run(\@cmd, \$_, \my $output);

If you don't want to capture the output, you can use any of the following:

use String::ShellQuote qw( shell_quote );
my $cmd1 = shell_quote('printf', '%s', $_);
my $cmd2 = shell_quote(@cmd);
system("$cmd1 | $cmd2");

system('/bin/sh', '-c', 'printf "%s" "$0" | "$@"', $_, @cmd);

use String::ShellQuote qw( shell_quote );
my $cmd = shell_quote(@cmd);
open(my $pipe, '|-', $cmd);
print($pipe $_);
close($pipe);

open(my $pipe, '|-', '/bin/sh', '-c', '"$@"', 'dummy', @cmd);
print($pipe $_);
close($pipe);

use IPC::Run3 qw( run3 );
run3(\@cmd, \$_);

use IPC::Run qw( run );
run(\@cmd, \$_);

If you don't want to capture the output, but you don't want to see it either, you can use any of the following:

use String::ShellQuote qw( shell_quote );
my $cmd1 = shell_quote('printf', '%s', $_);
my $cmd2 = shell_quote(@cmd);
system("$cmd1 | $cmd2 >/dev/null");

system('/bin/sh', '-c', 'printf "%s" "$0" | "$@" >/dev/null', $_, @cmd);

use String::ShellQuote qw( shell_quote );
my $cmd = shell_quote(@cmd);
open(my $pipe, '|-', "$cmd >/dev/null");
print($pipe $_);
close($pipe);

open(my $pipe, '|-', '/bin/sh', '-c', '"$@" >/dev/null', 'dummy', @cmd);
print($pipe $_);
close($pipe);

use IPC::Run3 qw( run3 );
run3(\@cmd, \$_, \undef);

use IPC::Run qw( run );
run(\@cmd, \$_, \undef);

Notes:

The solutions using printf will impose a limit on the size of the data to pass to the program's STDIN.
The solutions using printf are unable to pass a NUL to the program's STDIN.
The presented solutions that use IPC::Run3 and IPC::Run don't involve a shell. This avoids problems.
You should probably use system and capture from IPC::System::Simple instead of the builtin system and qx to get "free" error checking.

I like the `IPC::Run` solutions. Seems very simple! The first solution is using `printf` instead of `echo`, what would be the advantage of using `printf` instead of `echo`? — Håkon Hægland, Oct 21 '16 at 19:19
@Håkon Hægland, It's very hard to properly pass arbitrary values to `echo` because they could be interpreted as options. Also, some versions of `echo` recognize `\ ` escapes, requiring further system-dependent escaping. — ikegami, Oct 21 '16 at 19:24

score 3 · Accepted Answer · edited May 23 '17 at 12:16

3

This answer is a very naive approach. It's prone to deadlock. Don't use it!

ikegami explains in a comment below:

If the parent writes enough to the pipe attached to the child's STDIN, and if the child outputs enough to the pipe attached to its STDOUT before it reads from its STDIN, there will be a deadlock. (This can be as little as 4KB on some systems.) The solution involved using something like select, threads, etc. The better solution is to use a tool that has already solved the problem for you (IPC::Run3 or IPC::Run). IPC::Open2 and IPC::Open3 are too low-level to be useful in most circumstances

I'll leave the original answer, but encourage readers to pick the solution from one of the other answers instead.

You can use open2 from IPC::Open2 to read and write to the same process.

Now you don't need to care about escaping anything.

use IPC::Open2;
use FileHandle;

my $writer = FileHandle->new;
my $reader = FileHandle->new;

my $pid = open2( $reader, $writer, 'wc -c' );

# write to the pipe
print $writer 'some content that need to be escaped &>|\"$\'`\s\\';

# tell it you're done
$writer->close;

# read the out of the pipe
my $line = <$reader>;
print $line;

This will print 48.

Note that you can't use double quotes "" for the exact input you showed because the number of backslashes \ is wrong.

See perldoc open and perlipc for more information.

edited May 23 '17 at 12:16

Community

1
1

answered Oct 21 '16 at 09:56

simbabque

53,749
8
73
136

Good but I would prefer using only core-modules. Currently I use `File::Temp` but I don't like it – nowox Oct 21 '16 at 10:01
5

@nowox both IPC::Open2 and FileHandle have been _first released with perl 5_ according to my `corelist`. I believe that means since the initial release of Perl 5. It doesn't get more core-module than that. :) – simbabque Oct 21 '16 at 10:02
3

By the way, File::Temp came with 5.6.1. – simbabque Oct 21 '16 at 10:03
BAD! This "solution" suffers from a race condition. Your programs could deadlock. – ikegami Oct 21 '16 at 19:02
@ikegami can you explain how to fix it? – simbabque Oct 21 '16 at 19:04
3

If the parent writes enough to the pipe attached to the child's STDIN, and if the child outputs enough to the pipe attached to its STDOUT before it reads from its STDIN, there will be a deadlock. (This can be as little as 4KB on some systems.) The solution involved using something like `select`, threads, etc. The better solution is to use a tool that has already solved the problem for you (IPC::Run3 or IPC::Run). IPC::Open2 and IPC::Open3 are too low-level to be useful in most circumstances. – ikegami Oct 21 '16 at 19:08
1

If you didn't want to capture the output, using `open2('>&STDOUT', $writer, ...)` would avoid the issue. But then you could simply use `open(my $writer, '|-', ...)` – ikegami Oct 21 '16 at 19:15
@ikegami *"there will be a deadlock"*. Interesting, I created a follow-up question [here](http://stackoverflow.com/q/40189625/2173773) – Håkon Hægland Oct 22 '16 at 07:45

score 2 · Answer 3 · answered Oct 21 '16 at 18:27

I like the solution provided by @simbabque since it avoids calling the Shell. Anyway, for comparison, a shorter solution can be obtained using Bash (but avoiding echo) by using a Bash Here string:

$_ = q{some content that need to be escaped &>|\"$\'`\s\\};
$_ =~ s/'/'"'"'/g; # Bash needs single quotes to be escaped
system 'bash', '-c', "foo <<< '$_'";

And, if you need to capture the output of the command:

use Capture::Tiny 'capture_stdout';
my $res = capture_stdout { system 'bash', '-c', "foo <<< '$_'" };

How to pipe the content of a variable as STDIN in a qx{} statement in Perl?

3 Answers3

Linked