Shell script containing a Perl one-liner has blank results

Question

I have a Perl one-liner that works fine on the command line:

perl -nle 'm"\w+:x:\d+:\d+:\S+:/S+:(\S+)$" and $h{$1}++; END{ print "$_: $h{$_}" foreach sort { $h{$b} <=> $h{$a} } keys %h }' /etc/textfile

I've put this into a shell file called shell.sh so the next guy won't have to copy/paste and can just run it:

#!/bin/sh
perl -nle 'm"\w+:x:\d+:\d+:\S+:/S+:(\S+)$" and $h{$1}++; END{ print "$_: $h{$_}" foreach sort { $h{$b} <=> $h{$a} } keys %h }' /etc/textfile

I try running this on the command line and get no results; it just loads a fresh prompt with no output. Anyone see what I'm doing wrong?

Here are some system specs:

Linux version 2.6.32-220.13.1.el6.x86_64

(gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)

GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)

Here's a bit from the text file:

rfink:x:140:140:rat fink:/var/lib/rfink:/sbin/nologin                                 
edible:x:16252:10001:eric idle:/users/eidle/:/bin/bash                                       
tsawyer:x:30855:10001:tom sawyer:/users/tsawyer/:/bin/bash                                
karthur:x:30886:10001:King Arthur:/users/karthur/:/bin/bash                                         
karthur:x:30886:10001:king arthur:/users/karthur/:/bin/bash                                         
jcash:x:30887:10001:john cash:/users/jcash/:/bin/bash                              
hpotter:x:30887:10001:harry potter:/users/hpotter/:/bin/bash                              
triddle:x:30956:10001:tom riddle:/users/triddle/:/bin/bash

Added. I'm pretty sure the regex is sound because the one-liner works from the command line. Something about the shell script seems to be mucking it up. — kyoob, May 21 '12 at 15:28
Is all the trailing whitespace in the sample from your question in `/etc/textfile` verbatim? — Greg Bacon, May 21 '12 at 15:43
The last line nails it. Each line has 1 trailing whitespace (the multiple whitespaces on all other lines happened in c/p). — kyoob, May 21 '12 at 15:47
This won't help, but why not convert the perl one-liner to a perl script rather than a bash script? — gpojd, May 21 '12 at 15:51
That would be the easiest thing to do, but this has been requested specifically as a .sh file. So it goes. — kyoob, May 21 '12 at 15:53

score 3 · Accepted Answer · edited May 23 '17 at 12:04

Quick answer

perl -nle 'm"\w+:x:\d+:\d+:[^:]+:\S+:(\S+)\s*$" and $h{$1}++;
  END{ print "$_: $h{$_}" foreach sort { $h{$b} <=> $h{$a} } keys %h }' \
  /etc/textfile

Your regex had three issues.

The field after the group ID could have spaces, so replace that subpattern with [^:]+ to match one or more non-colon characters.
You used the wrong slash in your subpattern for matching the home directory.
Insert \s* before $ to allow optional trailing whitespace on each line.

Output:

/bin/bash: 7
/sbin/nologin: 1

Other approaches

Perl has an awk mode, which would allow

perl -F: -lane '++$sh{$F[-1]};
  END{print "$_: $sh{$_}" for sort { $sh{$b} <=> $sh{$a} } keys %sh}' \
  /etc/textfile

Having to remove trailing whitespace seems to cancel the syntactic benefit.

perl -F: -lane '($sh = pop @F) =~ s/\s+$//; ++$sh{$sh};
  END{print "$_: $sh{$_}" for sort { $sh{$b} <=> $sh{$a} } keys %sh}' \
  /etc/textfile

You could use a pipeline to get the best of all worlds:

perl -pe 's/[^\S\n]+$//' /etc/textfile |
  perl -F: -lane 'print $F[-1]' |
    sort | uniq -c | sort -nr

The output transposes the columns, but you get the same information.

Note the use of the regex double-negative technique in the pipeline’s first command for removing all whitespace except newlines.

      7 /bin/bash
      1 /sbin/nologin

As a shell script

Your question asks for a shell script, so—to vibe off daxim’s answer—that is

#! /bin/sh

perl -MUser::pwent -le \
  '$_->shell && print $_->shell while $_ = getpwent' |
  sort | uniq -c | sort -nr

Note that this does not handle the pathological case of a shell named 0.

If you don’t necessarily want to read the system /etc/passwd, then your script becomes

#! /bin/sh

if [ $# -eq 0 ]; then
  echo Usage: $0 passwd-file .. 1>&2
  exit 1
fi

perl -pe 's/[^\S\n]+$//' "$@" |
  perl -lne 'm|\w+:x:\d+:\d+:[^:]+:\S+:(\S+)$| && print $1' |
    sort | uniq -c | sort -nr

Different systems use different formats, so I recommend nailing down your expectation as in the above rather than blindly printing the last field, whatever it is. This may mean coping with the occasional empty output.

Neat, this worked! Actually I just lazied up that [^:]+ entry and changed that whole phrase between colons to another \S+. Still not sure why the one-liner as I had it would give me results from the command line but not from within a .sh file. — kyoob, May 21 '12 at 16:02
I'm glad it helped. I don't see how either could have generated output unless your users' home directories are /SSSSSSS, /SSSSS, and /SS (or there's a copy-paste error). — Greg Bacon, May 21 '12 at 16:24
Yeah, most probably a problem with the wetware. IIRC (this was all the way back on Friday so that's a longshot) I manually typed the code into the .sh script so I could easily have reversed that slash and broken the whole thing. Thanks! — kyoob, May 21 '12 at 16:26

score 2 · Answer 2 · answered May 21 '12 at 16:31

Avoid ad-hoc regex when a specialised parser exists.

perl -MUser::pwent=getpwent -e'
    while (my $pwent = getpwent) { $h{ $pwent->shell }++; }
    END { print "$_: $h{$_}\n" for sort { $h{$b} <=> $h{$a} } keys %h }
'

Avoid reg-ex when simpler constructs, like split, index/substr, unpack will do. Here I take advantage of autosplit:

perl -F: -lane'
    $h{ $F[-1] }++;
    END { print "$_: $h{$_}" for sort { $h{$b} <=> $h{$a} } keys %h }
' /etc/textfile

This makes for much shorter, more readable programs.

Shell script containing a Perl one-liner has blank results

2 Answers2

Quick answer

Other approaches

As a shell script