0

I want to filter lines of oc rsh du -shc output like this:

oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '$2=="total" {print $1}'

But I got no results. For local du -shc /some/dir 2>&1 I get desired output.

# locally
$  du -shc ~ 2>&1
du: cannot read directory '/home/xxxx/.cache/doc': Permission denied
du: cannot read directory '/home/xxxx/.cache/dconf': Permission denied
du: cannot read directory '/home/xxxx/.gvfs': Permission denied
52G /home/xxxx/
52G total

And filtering:

# filtering works; search the 2nd arg "total" and print arg 1
$  du -shc ~ 2>&1 | awk '$2=="total" {print $1}' 
52G

And printing:

$ du -shc ~ 2>&1 | awk '{print $1}'
du:
du:
du:
52G
52G

And $2:

$ du -shc ~ 2>&1 | awk '{print $2}'
cannot
cannot
cannot
/home/xxx
total

But remotely:

oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '$2=="total" {print $1}'
# no output

And if I don't use awk:

$ oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1
du: cannot read directory '/proc/tty/driver': Permission denied
du: cannot read directory '/proc/acpi': Permission denied
du: cannot read directory '/proc/scsi': Permission denied
du: cannot access '/proc/130224/task/130224/fd/3': No such file or directory
du: cannot access '/proc/130224/task/130224/fdinfo/3': No such file or directory
du: cannot access '/proc/130224/fd/4': No such file or directory
du: cannot access '/proc/130224/fdinfo/4': No such file or directory
du: cannot read directory '/run/cryptsetup': Permission denied
du: cannot read directory '/run/secrets/rhsm': Permission denied
du: cannot read directory '/sys/firmware': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/1': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/2': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/4': Permission denied
du: cannot read directory '/var/lib/yum/history/2021-12-02/3': Permission denied
du: cannot read directory '/var/lib/machines': Permission denied
du: cannot read directory '/var/cache/ldconfig': Permission denied
1.8G    /
1.8G    total
command terminated with exit code 1

And, if I only print $1:

$ oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '{print $1}'
du:
   du:
      du:
         du:
            du:
               du:
                  du:
                     du:
                        du:
                           du:
                              du:
                                 du:
                                    du:
                                       du:
                                          du:
                                             du:
                                                1.8G
                                                    1.8G
                                                        command

Why there are extra indentations? Seems only line break is done, but no return carriage to the beginning of the line???

If I print $2, we can see the 2 lines at the end are aligned; what is wrong here?

$ oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1 | awk '{print $2}'
cannot
      cannot
            cannot
                  cannot
                        cannot
                              cannot
                                    cannot
                                          cannot
                                                cannot
                                                      cannot
                                                            cannot
                                                                  cannot
                                                                        cannot
                                                                              cannot
                                                                                    cannot
                                                                                          cannot
                                                                                                /
total
terminated

Local awk version is:

GNU Awk 4.1.4, API: 1.1 (GNU MPFR 4.0.1, GNU MP 6.1.2)
Copyright (C) 1989, 1991-2016 Free Software Foundation.

, and remote, openshift awk version is

GNU Awk 4.0.2
Copyright (C) 1989, 1991-2012 Free Software Foundation.

and local du version is

du (GNU coreutils) 8.28
Copyright (C) 2017 Free Software Foundation, Inc.

, while remote, openshift pod du version is

du (GNU coreutils) 8.22
Copyright (C) 2013 Free Software Foundation, Inc.

Seems remote versions are behind local versions a lot, notice the years of copyright.

As per request of @Ed Morton:

$ oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | od -c
0000000   d   u   :       c   a   n   n   o   t       r   e   a   d    
                                                                       0000020   d   i   r   e   c   t   o   r   y       '   /   p   r   o   c
                                                                                                                                              0000040   /   t   t   y   /   d   r   i   v   e   r   '   :       P   e
                       0000060   r   m   i   s   s   i   o   n       d   e   n   i   e   d  \r
                                                                                              0000100  \n   d   u   :       c   a   n   n   o   t       r   e   a   d
                                                                                                                                                                     0000120       d   i   r   e   c   t   o   r   y       '   /   p   r   o
                                              0000140   c   /   a   c   p   i   '   :       P   e   r   m   i   s   s
                                                                                                                     0000160   i   o   n       d   e   n   i   e   d  \r  \n   d   u   :    
                                                                                                                                                                                            0000200   c   a   n   n   o   t       r   e   a   d       d   i   r   e
                                                                     0000220   c   t   o   r   y       '   /   p   r   o   c   /   s   c   s
                                                                                                                                            0000240   i   '   :       P   e   r   m   i   s   s   i   o   n       d
                     0000260   e   n   i   e   d  \r  \n   d   u   :       c   a   n   n   o
                                                                                            0000300   t       a   c   c   e   s   s       '   /   p   r   o   c   /
                                                                                                                                                                   0000320   2   8   7   2   7   /   t   a   s   k   /   2   8   7   2   7
                                            0000340   /   f   d   /   3   '   :       N   o       s   u   c   h    
                                                                                                                   0000360   f   i   l   e       o   r       d   i   r   e   c   t   o   r
                                                                                                                                                                                          0000400   y  \r  \n   d   u   :       c   a   n   n   o   t       a   c
                                                                   0000420   c   e   s   s       '   /   p   r   o   c   /   2   8   7   2
                                                                                                                                          0000440   7   /   t   a   s   k   /   2   8   7   2   7   /   f   d   i
                   0000460   n   f   o   /   3   '   :       N   o       s   u   c   h    
                                                                                          0000500   f   i   l   e       o   r       d   i   r   e   c   t   o   r
                                                                                                                                                                 0000520   y  \r  \n   d   u   :       c   a   n   n   o   t       a   c
                                          0000540   c   e   s   s       '   /   p   r   o   c   /   2   8   7   2
                                                                                                                 0000560   7   /   f   d   /   4   '   :       N   o       s   u   c   h
                                                                                                                                                                                        0000600       f   i   l   e       o   r       d   i   r   e   c   t   o
                                                                 0000620   r   y  \r  \n   d   u   :       c   a   n   n   o   t       a
                                                                                                                                        0000640   c   c   e   s   s       '   /   p   r   o   c   /   2   8   7
                 0000660   2   7   /   f   d   i   n   f   o   /   4   '   :       N   o
                                                                                        0000700       s   u   c   h       f   i   l   e       o   r       d   i
                                                                                                                                                               0000720   r   e   c   t   o   r   y  \r  \n   d   u   :       c   a   n
                                        0000740   n   o   t       r   e   a   d       d   i   r   e   c   t   o
                                                                                                               0000760   r   y       '   /   r   u   n   /   c   r   y   p   t   s   e
                                                                                                                                                                                      0001000   t   u   p   '   :       P   e   r   m   i   s   s   i   o   n
                                                               0001020       d   e   n   i   e   d  \r  \n   d   u   :       c   a   n
                                                                                                                                      0001040   n   o   t       r   e   a   d       d   i   r   e   c   t   o
               0001060   r   y       '   /   r   u   n   /   s   e   c   r   e   t   s
                                                                                      0001100   /   r   h   s   m   '   :       P   e   r   m   i   s   s   i
                                                                                                                                                             0001120   o   n       d   e   n   i   e   d  \r  \n   d   u   :       c
                                      0001140   a   n   n   o   t       r   e   a   d       d   i   r   e   c
                                                                                                             0001160   t   o   r   y       '   /   s   y   s   /   f   i   r   m   w
                                                                                                                                                                                    0001200   a   r   e   '   :       P   e   r   m   i   s   s   i   o   n
                                                             0001220       d   e   n   i   e   d  \r  \n   d   u   :       c   a   n
                                                                                                                                    0001240   n   o   t       r   e   a   d       d   i   r   e   c   t   o
             0001260   r   y       '   /   v   a   r   /   l   i   b   /   y   u   m
                                                                                    0001300   /   h   i   s   t   o   r   y   /   2   0   2   1   -   0   1
                                                                                                                                                           0001320   -   2   6   /   1   '   :       P   e   r   m   i   s   s   i
                                    0001340   o   n       d   e   n   i   e   d  \r  \n   d   u   :       c
                                                                                                           0001360   a   n   n   o   t       r   e   a   d       d   i   r   e   c
                                                                                                                                                                                  0001400   t   o   r   y       '   /   v   a   r   /   l   i   b   /   y
                                                           0001420   u   m   /   h   i   s   t   o   r   y   /   2   0   2   1   -
                                                                                                                                  0001440   0   1   -   2   6   /   2   '   :       P   e   r   m   i   s
           0001460   s   i   o   n       d   e   n   i   e   d  \r  \n   d   u   :
                                                                                  0001500       c   a   n   n   o   t       r   e   a   d       d   i   r
                                                                                                                                                         0001520   e   c   t   o   r   y       '   /   v   a   r   /   l   i   b
                                  0001540   /   y   u   m   /   h   i   s   t   o   r   y   /   2   0   2
                                                                                                         0001560   1   -   0   1   -   2   6   /   4   '   :       P   e   r   m
                                                                                                                                                                                0001600   i   s   s   i   o   n       d   e   n   i   e   d  \r  \n   d
                                                         0001620   u   :       c   a   n   n   o   t       r   e   a   d       d
                                                                                                                                0001640   i   r   e   c   t   o   r   y       '   /   v   a   r   /   l
         0001660   i   b   /   y   u   m   /   h   i   s   t   o   r   y   /   2
                                                                                0001700   0   2   1   -   0   1   -   2   6   /   3   '   :       P   e
                                                                                                                                                       0001720   r   m   i   s   s   i   o   n       d   e   n   i   e   d  \r
                                0001740  \n   d   u   :       c   a   n   n   o   t       r   e   a   d
                                                                                                       0001760       d   i   r   e   c   t   o   r   y       '   /   v   a   r
                                                                                                                                                                              0002000   /   l   i   b   /   m   a   c   h   i   n   e   s   '   :    
                                                       0002020   P   e   r   m   i   s   s   i   o   n       d   e   n   i   e
                                                                                                                              0002040   d  \r  \n   d   u   :       c   a   n   n   o   t       r   e
       0002060   a   d       d   i   r   e   c   t   o   r   y       '   /   v
                                                                              0002100   a   r   /   c   a   c   h   e   /   l   d   c   o   n   f   i
                                                                                                                                                     0002120   g   '   :       P   e   r   m   i   s   s   i   o   n       d
                              0002140   e   n   i   e   d  \r  \n   3   .   7   G  \t   /  \r  \n   3
                                                                                                     0002160   .   7   G  \t   t   o   t   a   l  \r  \n
0002173
xxxxxxx@elxag5zs8d3:~

and:

$ oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | awk '{print $1}' | od -c
0000000   d   u   :  \n   d   u   :  \n   d   u   :  \n   d   u   :  \n
*
0000100   3   .   7   G  \n   3   .   7   G  \n
0000112

If I change RS, I got stranger results.

$ oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | awk 'BEGIN {RS="\r\n";} {print $1}'
du:
   du:
      du:
         du:
            du:
               du:
                  du:
                     du:
                        du:
                           du:
                              du:
                                 du:
                                    du:
                                       du:
                                          du:
                                             du:
                                                3.7G
                                                    3.7G
                                                        xxxxxx@elxag5zs8d3:~
WesternGun
  • 11,303
  • 6
  • 88
  • 157
  • Please add the output of running it locally piped to awk and getting more than 1 line of output, e.g. `du -shc ~ 2>&1 | awk '{print $1}'`. so we can see if THAT gets indented too and so narrow down the problem. – Ed Morton Apr 14 '22 at 14:06
  • 1
    Aside: why are you doing `2>&1`? It seems like your real intent is to only match on a string like `total` in the stdout so do you REALLY want awk to parse the stderr as well as the stdout? – Ed Morton Apr 14 '22 at 14:09
  • If I don't use `2>&1`, permission errors come out; because awk only accepts stdin and will not process stderr, but I don't want to see stderr coming out. Later I may use the output of this command in a script, so I don't want anything unrelated. Updated. – WesternGun Apr 14 '22 at 14:26
  • 1
    No, use `2>/dev/null` to discard stderr instead of `2>&1` to add it into the stdout if you want to ignore the errors. – Ed Morton Apr 14 '22 at 14:36
  • Yes that works too I guess. – WesternGun Apr 14 '22 at 14:38
  • `2>&1` actually doesn't work as it clutters up the output and could produce false matches or other undesirable effects, e.g. maybe stderr includes some control chars that are causing what you're currently experiencing, idk, so let's eliminate that possibility by using `2>/dev/null` instead. – Ed Morton Apr 14 '22 at 14:39
  • Thanks for adding the local tool output. Now run `oc rsh broker-amq-1-2-fsbnd du -shc / 2>/dev/null | od -c` and `oc rsh broker-amq-1-2-fsbnd du -shc / 2>/dev/null | awk '{print $1}' | od -c` and add the output of those to your question so we can see which characters are actually being output by both commands. – Ed Morton Apr 14 '22 at 14:41
  • Both give `0000000`. – WesternGun Apr 14 '22 at 14:43
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/243906/discussion-between-westerngun-and-ed-morton). – WesternGun Apr 14 '22 at 14:51
  • For future reference: seems I just switched to another environment so no output for `oc rsh`, pod name mismatch. After getting pod names again, with `od -c`, I find out that remote `du` uses line seperator `\r\n` and `awk` will translate that into `\n`. Due to unknown reason it's print like no carriage return in my terminal. I have to set `ORS="\r\n"` for `awk` to get expected output(line break + carriage return), just like local. I don't know why. Local `du` also uses `\n`. – WesternGun Apr 14 '22 at 15:23

3 Answers3

0

It's very odd that your oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | od -c output shows no blanks or tabs, e.g. between cannot and read in:

0000000 d u : c a n n o t r e a d

while without od -c, i.e. when you run oc rsh broker-amq-1-2-fsbnd du -shc / 2>&1, there clearly are blanks:

du: cannot read directory '/proc/tty/driver': Permission denied

Anyway, as shown in the oc rsh broker-amq-1-15-snd64 du -shc / 2>/dev/null | od -c output your output lines end in \r\n, e.g. (emphasis mine):

0000000 d u : c a n n o t r e a d
0000020 d i r e c t o r y ' / p r o c
0000040 / t t y / d r i v e r ' : P e
0000060 r m i s s i o n d e n i e d **\r
0000100 \n** d u : c a n n o t r e a d
0000120 d i r e c t o r y ' / p r o
0000140 c / a c p i ' : P e r m i s s
0000160 i o n d e n i e d **\r \n**

You mentioned that locally du outputs \n-terminated lines so that just means it's probably rsh or oc that's changing that to \r\n - try it with some other command like oc rsh broker-amq-1-15-snd64 date to verify.

In any case, to handle that simply and portably, and assuming you don't actually want \r\n line endings in your final output, change your awk script from this:

awk '$2=="total" {print $1}'

to this which removes the \r at the end of each line before doing anything else with the input:

awk '{sub(/\r$/,"")} $2=="total" {print $1}'

The reason you weren't getting output with $2=="total" is that given input like:

foo total\r\n

$2 isn't "total", it's "total\r", and so the comparison fails.

You mentioned setting ORS="\r\n" to get the desired output - no, that just propagates the problem to the next command.

You could set RS="\r\n" but that would only work in an awk that accepts multi-char RS, e.g. GNU awk, in other awks that'd be treated the same as RS="\r" per the POSIX standard that says RS is a single char.

There are some platforms out there where when \r\n input is detected either:

  1. The \r automatically gets stripped by the underlying C primitives that awk calls to read input, or
  2. The RS automatically gets set to \r\n.

so just be aware of that and check what input awk is actually getting and what RS and ORS are set to if the sub() above doesn't do what you want (awk 'NR==1{ print "$0=<"$0">" ORS "RS=<"RS">" ORS "ORS=<"ORS">"; exit }' | od -c should tell you all you need to know).

For more information see Why does my tool output overwrite itself and how do I fix it?.

Ed Morton
  • 188,023
  • 17
  • 78
  • 185
  • Yeah I agree, with `oc rsh`, `date` also outputs `\r\n`, but when I enter the pod in Openshift, that gives `\n`. So yeah, your solution is right. I got the correct result (`3.7G`). But the only thing is that the next line after the correct result is indented, like `3.7G(line break here) xxxx@localhost:~`(notice the 4 spaces at the beginning of the second line, which matches the length of the output). So line break in my Ubuntu only needs `\n`, but why suddenly after `oc rsh` it needs `\r\n`? – WesternGun Apr 14 '22 at 19:22
  • And maybe there was some formatting error of StackOverflow, I edited the question to answer your question about space between `du:` and `cannot`: there are extra spaces; it's just the code format in the question with each line starting with 4 spaces is not clear, I changed to another format, now it's clear. – WesternGun Apr 14 '22 at 19:34
  • You;d have to ask whoever provides whatever `oc rsh` is why they chose to use DOS line endings. Please post a new followup question about the indenting with a [mcve] using your currently fixed script just about that remaining issue as there's already so much information in your question it's hard to tell what it's about any more and this may be something completely different from the DOS line endings. – Ed Morton Apr 14 '22 at 20:12
  • Thanks for ur patience a lot! `oc` is openshift client and `oc rsh` runs command in pods. According to [a bug report](https://bugzilla.redhat.com/show_bug.cgi?id=1668763), `--no-tty=true` will not insert CRLF at the end. – WesternGun Apr 15 '22 at 03:30
0

For completeness, as a side note:

oc rsh seems always to append CRLF after the output and there's some bug reports about that, see https://bugzilla.redhat.com/show_bug.cgi?id=1668763 and https://bugzilla.redhat.com/show_bug.cgi?id=1638447. It says oc rsh requires a pseudo-terminal and with option --no-tty=true it will not append CRLF. Meanwhile oc exec will not append CRLF. It seems to come from docker term which both depends on.

This would most likely have to be a patch made against the docker term[1] package, which the rsh and exec commands depend on.

[1]. https://github.com/moby/moby/tree/master/pkg/term

So, the following two forms will require no stripping of \r afterwards, and at the same time introduce no extra indentation after execution(before xxxxxx@localhost:~):

oc exec broker-amq-1-15-snd64 -- du -shc / 2>/dev/null | awk '$2=="total" {print $1}'
3.6G
xxxxxx@localhost:~

or:

oc rsh --no-tty=true broker-amq-1-15-snd64 du -shc / 2>/dev/null | awk '$2=="total" {print $1}'
3.6G
xxxxxx@localhost:~

Note for exec you need --.

WesternGun
  • 11,303
  • 6
  • 88
  • 157
0

Perhaps a solution that neither involves forcing oc (or any other application) to alter it's line-ending behavior, nor having to fudge with either FS or RS ?

gawk -te 'NF=/\11total\15?$/'
19G

Tested and confirmed working on mawk 1.3.4, mawk 1.9.9.6, macos nawk, and gawk 5.1.1, including invocation flags -c/-P/-t/-S

I would've used \r instead of \15 but gawk -t traditional mode was complaining about \r not being available, so \15 would be the most portable solution

But if you don't need gawk -t compatibility and don't mind fudging FS, then

nawk NF=NF==2 FS='\ttotal\r?'
or
mawk -— —-NF FS='\ttotal\r?'

or even

mawk 'NF=!+FS<NF' FS='total$' RS='\r?\n'

-- The 4Chan Teller

RARE Kpop Manifesto
  • 2,453
  • 3
  • 11