1

The below command works fine when I run it manually but when I call it from perl script using backticks or system command, it gives this error

sh: -c: line 0: syntax error near unexpected token `('

Script snapshot:

#Find contents of myFile in zipfile and output the matched records to output.txt
$cmd = "awk -F\"|\" 'NR==FNR{hash[\$0]=1;next} \$237 in hash' $myFILE <(unzip -p $zipfile *XYZ*) >> output.txt";
$result=`$cmd`;

It seems that we cannot call a subshell i.e. (unzip ...) within a system call through perl. Please advise as I have been struggling since a couple of days.

Nainesh Vashi
  • 11
  • 2
  • 5
    Good grief ... no wonder it's a struggle. Who's going to get all those escapes right. Why that `awk`? This way you shell out (with backticks), then run a command (awk, for which you need to escape things), run another program in a subshell of that shell (unzip, for which you need to escape things), with redirections ... why not just do all processing (of unzip's return) in Perl, if the script is in Perl? – zdim Oct 13 '17 at 02:58
  • 1
    `<( ... )` is not valid `sh` syntax – ikegami Oct 13 '17 at 03:54
  • If you reframe your question as to exactly what your inputs and outputs are, we'll give you a perl only answer that's both simpler and less brittle. But why not instead use one of the various perl modules do to the job – Sobrique Oct 13 '17 at 09:56
  • Thanks for your answers. I do not want to add packages in perl – Nainesh Vashi Oct 14 '17 at 13:58
  • Thanks for your answers so far. @zdim - I do not want to add packages in perl as my client's environment does not allow that or at least it will take a few days and i need to work with this urgently. – Nainesh Vashi Oct 14 '17 at 14:04
  • @ikegamie - It works if I put the command in a shell script (just as it is, without backticks or system calls). – Nainesh Vashi Oct 14 '17 at 14:05
  • @NaineshVashi The construct `<(..)` works in a shell at the terminal, likely `bash`. But Perl's backticks, it uses `sh`, as the error shows you `sh: -c: ...`. In _that_ shell it's not valid syntax. – zdim Oct 14 '17 at 19:12
  • 1
    @NaineshVashi I didn't suggest to use packages. To repeat: call `unzip` out of your (Perl) script and process its output in that (Perl) script. Use Perl's (superb) native text-processing capabilities, instead of shelling out to use `awk`. (I get the idea: you have a command-line with `awk` that works and you'd like to "just use it" in the script; but that makes it really _much_ harder, error-prone, and brittle. And awkward.) – zdim Oct 14 '17 at 19:15
  • i can call unzip but the package has 10M lines x 3K fields per line of varying sizes. And i have to process 100s such zips with a parallel degree. Unzipping will need a lot of space or memory depending on whether i output the unzipped to a file or hold in memory. Hence calling unzip separately is not an option for me. – Nainesh Vashi Oct 15 '17 at 19:29
  • Given the constraints, i saw awk to be faster to search if 237th field of each line is present in a list in another file or not. With perl, I would do the same thing by creating an array using split i thought awk would be natively faster than what i woud write in perl.Most importantly, i have to stream in from a zip as i have storage issue and without installing the perl zip modules. Hence I researched to the above commadline. – Nainesh Vashi Oct 15 '17 at 19:36
  • Alright, so you do have a thought-out reason. Still, I have a hard time believing that this approach is necessary for speed, or that it is faster really. As for streaming, you can manage the `unzip`'s piped extraction (`-p`) in Perl just fine. – zdim Oct 15 '17 at 20:24
  • But if you insist, one way is to use `system` so that you can ask for `bash` (this way you get `sh` shell which doesn't know of `<(...` as ikegami told you). So, `system('bash', '-c', 'awk ...')`. This doesn't return the output but you redirect it to a file anyway; just open thet file then. Then go through all those escapes carefully – zdim Oct 15 '17 at 20:26
  • A quick search brings up [this post](https://stackoverflow.com/a/571383/4653379) for example, on how to use `bash` via `system`. Then you'll have to deal with escaping -- unless it's good as it stands? – zdim Oct 15 '17 at 20:37

1 Answers1

0

This is because the command line is most likely /bin/bash and perl's shell is /bin/sh which does not support this mean. Put the zip command before the awk with pipe (|). Something like this:

#Find contents of myFile in zipfile and output the matched records to output.txt
$cmd = "unzip -p $zipfile *XYZ* | awk -F\"|\" 'NR==FNR{hash[\$0]=1;next} \$237 in hash' $myFILE >> output.txt";
$result=`$cmd`;
niry
  • 3,238
  • 22
  • 34
  • Or better yet - don't inline awk, because you can do it all in perl and don't have escaping problems. – Sobrique Oct 13 '17 at 09:53
  • @sobrique - I cannot install unzip/zip packages in perl as it is my client's protected env or at least it will take days and i need it earlier. – Nainesh Vashi Oct 14 '17 at 14:08
  • Perl does most of it natively. The only thing you might need a module for is reading zip files. But perl modules can be installed in a user context too. – Sobrique Oct 15 '17 at 10:40