0

I am trying to do ssh on a server and then do a grep to get the count of different errors in the log file. Once i have those details logs those into a CSV file. But when i am trying to run the grep command i am getting error.

#!/usr/bin/perl
my $addr = "user\@servername";
my $text = qq|Internal Server Error|;
my $remote_path = "/data/logs/error";
my $cmd = `ssh $remote_addr "grep -a $text $remote_path | awk -F " " '{print $4}' | sort -nr | uniq -c | sort -nr 2>/dev/null"`;
print $cmd;

But i am getting below error when i am running the script

grep: Internal: No such file or directory
grep: Server: No such file or directory
grep: Error: No such file or directory

Is there any suggestion how we can do this in shell script.

Developer
  • 6,292
  • 19
  • 55
  • 115
  • Why not do most (possibly all) of that directly in Perl? – Shawn Nov 05 '18 at 22:43
  • @Shawn The only thing running in the local Perl program is the `ssh` command; everything else is executed by the shell on the remote host. – chepner Nov 06 '18 at 13:48
  • If you aren't willing to install the modules necessary to do this safely in perl (by using [zdim's great answer](https://stackoverflow.com/a/53163525/14122)), would you be interested in a shell-based answer that doesn't use perl at all? – Charles Duffy Nov 06 '18 at 20:13

1 Answers1

5

First, in order to avoid a quoting nightmare and the many chances for shell injection I'd suggest to use a module, like String::ShellQuote

Then, I don't see that you need all those external tools, while such a long pipe-line is tricky and expensive. It invokes a number of programs, for jobs done really well in Perl, and requires very precise syntax.

Apart from ssh-ing, one other thing that an external tool may be good for here is to extract lines of interest with grep, in case that file is huge (otherwise you can read it into a scalar).

use warnings;
use strict;
use feature 'say';

use List::Util qw(uniq);  # in List::MoreUtils prior to module v1.45
use String::ShellQuote qw(shell_quote);

my $remote_addr = ...
my $remote_path = ...
my $text = 'Internal Server Error';

my $remote_cmd = shell_quote('grep', '-a', $text, $remote_path);
my $cmd = shell_quote('ssh', $remote_addr, $remote_cmd);

my @lines = qx($cmd);
chomp @lines;

# Process @lines as needed, perhaps
my @result = sort { $b <=> $a } uniq map { (split)[3] } @lines;
say for @result;

Once it comes to running external commands, there are many options. In the first place consider using a module. They all simplify things a lot, in particular with error checking, and are in general more reliable, while some also make harder jobs far more manageable.

An example with IPC::System::Simple

use IPC::System::Simple qw(capturex); 

my @lines = capturex('ssh', $remote_addr, $remote_cmd);

Since ssh executes the command when ran with one it doesn't need a shell (for that part) so capturex is used. See documentation for more options and for how to check for errors.

Some other options, from simple to more powerful, are Capture::Tiny, IPC::Run3, IPC::Run.

For more on all this see links assembled in this post (and search for more).


I can't see a need to run that pipeline as it stands but if there is one (stay on the remote host?) then form commands as above and then assemble the full pipeline

my $cgrep = shell_quote('grep', '-a', $text, $remote_path);
my $cawk  = shell_quote('awk', '-F', ' ', '{print $4}');
my $csort = shell_quote('sort', '-nr');
my $cuniq = shell_quote('uniq', '-c');

my $remote_cmd = "$cgrep | $cawk | $csort | $cuniq | $csort 2>/dev/null";

Note that the needed shell functionalities (| and redirection) shouldn't be quoted away.

The mere space in the awk piece may be awkward-looking but since it gets escaped it winds up right for -F. For me it's another sign of trouble with running external programs in shell pipelines; I couldn't figure out that bare space, thanks to Charles Duffy for the comment.

In this case the sort and uniq parts of the pipeline can be just typed as one string since it's only program names and options, but as soon as changes are made or any variables make their way in that becomes tricky. So I use shell_quote, for consistency and as a template.


Modules are reported missing and hard to obtain. Then escape what need be escaped (until you figure out how to get the modules, that is). In this case there happen to be little to fix, but that bit can serve as an example of common hoops to go through with complicated pipelines.

The string with $text needs to reach grep as such, one string. Since it passes through shell, which would break it by space into words, we need to protect (quote/escape) those spaces. Not to forget, we also need to get it to the shell in the first place, through Perl's parsing rules.

One way

my $text_quoted = q(') . quotemeta($text) . q(');

where quotemeta quotes all kinds of other things as well.

We also should protect the filename pattern, as it may rely on shell metacharacters (like *)

my $remote_path_quoted = quotemeta $remote_path;

but again, you have to inspect whether this is suitable in each case.

NOTE &hairsp; If any dynamically generated variables (computed, come from user...) are interpolated in these commands they need be validated, with things carefully escaped and quoted.

Now your pipeline should work (it does in my simulated tests)

my $cmd = "ssh $remote_host grep -a $text_quoted $remote_path_quoted"
    . q( | awk F ' ' '{print $4}' | sort -nr | uniq | sort -nr 2>/dev/null);

This can be broken up into sensible components in their own variables, etc, but I really don't recommend such hand-patched solutions.

I suggest to only use the first part (ssh + grep) and do the rest in Perl though, as in the main part of the answer. And then to have those modules installed and switch to them.

No major computing tool works without (the many) libraries, and every production install contains a lot of "additional" stuff. As need for more libraries arises they are installed. Why would it have to be different with Perl? Yes, you can do it with builtins only but that may be much harder.


&hairsp; One good reason would be so to use the system sort when files are huge, since it doesn't have to load the whole file at once, and for its speed. However, in this pipeline it is fed data over the pipe and invoked repeatedly so these advantages don't apply.

zdim
  • 64,580
  • 5
  • 52
  • 81
  • @Developer Edited and added. Let me know if more/other is needed – zdim Nov 05 '18 at 23:54
  • With respect to the awk command, `shell_quote('awk', '-F', ' ', '{print $4}')` appears to Do The Right Thing. – Charles Duffy Nov 06 '18 at 13:31
  • I don;t have these modules installed which you are using and i am unable to install them because of permission – Developer Nov 06 '18 at 17:07
  • @CharlesDuffy Yes, thank you, edited. It makes sense (after the fact) -- the `-F` needs precisely a space, which does get escaped. – zdim Nov 06 '18 at 18:39
  • @Developer The module `String::ShellQuote` "merely" escapes what need be escaped for the shell to get the right string. You can do this by hand instead, but it's a chore and it can be easy to miss a spot and have "shell injection" happen behind your back (that doesn't have to imply anyone's bad intent, it can bite accidentally and can go down completely quietly). If you do this often I'd recommend to ask for it to be installed. You can also install a module as a user if that works for how your code is run. – zdim Nov 06 '18 at 18:45
  • @Developer As for the other modules mentioned (to run and manage external commands) -- they are _really_ handy and useful. I'd recommend to pick one or two and ask for them to be installed. (You can at least try them out by installing them for yourself, as a user.) – zdim Nov 06 '18 at 18:47
  • @Developer, ...do you have access to *your own home directory*? That's all you need -- write access to the system-wide perl installation isn't necessary. – Charles Duffy Nov 06 '18 at 20:14
  • @Developer Posted a hand-rolled solution (working in my tests) -- until you get those modules installed. Please figure out how to do that; life's much easier that way. (If your software can run using user-installed modules then that's straightforward). – zdim Nov 06 '18 at 20:30
  • @thanks for explaining and helping me to understand how we can fix this issue. – Developer Nov 07 '18 at 19:11
  • 1
    @Developer Welcome :) Let me know if questions come up. – zdim Nov 07 '18 at 20:15