0

I am using a python script that outputs many warnings/errors (going to stderr) based on perfectly normal bad test certificates. Based on several SO posts, I was able find a way to run the script and ignore select stdout, stderr lines but its cumbersome:

runThing 3>&1 1>&2 2>&3 3>&- | grep -r 's/Insecure/'
 OR 
runThing 3>&1 1>&2 2>&3 3>&- | sed 's/Insecure/g'

Both filter out lots of lines like: 

 /Users/xxx/.blah/lib/python2.7/site-   packages/requests/packages/urllib3/connectionpool.py:791: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html  InsecureRequestWarning)

The sed one has this error, though: sed: 1: "s/Insecure/g": unterminated substitute in regular expression

Its a lot to add at the end of a line that is used a lot (the real runThing has commands and parameters), so I am trying to make a command like:

runThingClean() { command runThing "$@" > /dev/null 3>&1 1>&2 2>&3 3>&- | sed 's/Insecure/g' & }

When I run this, it now fails to filter (and shows the error is sed is used):

 sed: 1: "s/Insecure/g": unterminated substitute in regular expression

Can someone help me fix this command?

Thanks in advance..

midori
  • 4,807
  • 5
  • 34
  • 62
chrismead
  • 2,163
  • 3
  • 24
  • 36
  • BTW, did you consider `-W once`, to print each warning at most once? If the problem is noise level, that gets you a fairly happy medium. – Charles Duffy Jan 27 '16 at 18:07

2 Answers2

2

Don't do that at all: Just tell Python not to print the warnings in the first place, and then you have no need to filter them out.

python -W ignore yourprogram ...

Alternately, consider modifying your code calling the relevant components to suppress the warning at the call site, as given in the Python documentation:

import warnings
with warnings.catch_warnings():
    warnings.simplefilter('ignore')
    urllib3.doTheThingThatCausedTheWarning(...)

Finally, if you really want to do it out-of-process as a filter, just use grep -v Insecure; sed is a larger hammer than is needed for this job.

sh -c 'printf "%s\n" "stderr: keep" "stderr: Insecure" >&2; echo "stdout"' \
  3>&1 1>&2 2>&3 3>&- | grep -v Insecure 3>&1 1>&2 2>&3 3>&-

What all this redirection is doing is swapping stdout and stderr twice -- once before the grep, and once after to restore them. Let's look at how this works:

  • FD 1 (original stdout) is copied to FD 3 (to serve as backup)
  • FD 2 (original stderr) is copied to FD 1 (so it can be filtered)
  • FD 3 (the backup of original stdout) is copied to FD 2 (now stored in stderr)
  • FD 3 is closed, as it's no longer needed with the swap complete.
Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
  • Thanks for the tips. I was hoping not suppress all the warnings, though, and am not able to modify the code. For some reason, this: clean() { command dirty "$@" > /dev/null 3>&1 1>&2 2>&3 3>&- | grep -v s/Insecure/ & } doesn't filter the lines with Insecure in them. I wonder if the pipe is working at all? – chrismead Jan 27 '16 at 17:53
  • 1
    You're redirecting FD 1 to `/dev/null`, then copying that handle on `/dev/null` to FD 3? The logic doesn't look quite right. – Charles Duffy Jan 27 '16 at 17:58
  • 1
    @chrismead, ...I extended the answer here to provide a tested, working example (that also moves stderr and stdout back to their original FDs after the grep). – Charles Duffy Jan 27 '16 at 18:02
  • Thanks, again @Charles Duffy. I now am trying this: clean() { command noisy "$@" 3>&1 1>&2 2>&3 3>&- | grep -v Insecure 3>&1 1>&2 2>&3 3>&- & } and it still doesn't work, but I wonder if "noisy" runs another program as part of its operation. Maybe stdout, stdin, stderr would be different in that case? – chrismead Jan 27 '16 at 18:56
  • @chrismead, depends on the details, unfortunately. If you've got something writing output straight to the TTY, you won't be able to filter it without using something like `script` to get it back to stdout/stderr. – Charles Duffy Jan 27 '16 at 19:04
  • @chrismead, ...in general, though, FDs are inherited by child processes, so redirections like this will work even if it's something further down the process tree doing the write. – Charles Duffy Jan 27 '16 at 19:05
  • Anyhow, I think this has been expanded/clarified enough that I'm no longer confident that it's a duplicate. If you could come up with a reproducer (ie. a `runThing` definition to amend into your question that behaves the same way your actual program does), I'd be happy to reopen the question. – Charles Duffy Jan 27 '16 at 19:06
  • @chrismead, ...also, are you sure that the warning is actually written to stderr rather than stdout? If it's stdout all the redirections are unnecessary and you can just pipe straight to `grep -v` without them. – Charles Duffy Jan 27 '16 at 19:12
1

The error means you didn't close the s in sed correctly. For s it should be s/regex/substitution/g

If you are trying to remove Insecure or substitute:

sed 's/Insecure/YOUR_SUBSTITUTION/g'

if you are trying to display only lines with Insecure:

sed -n '/Insecure/p'
midori
  • 4,807
  • 5
  • 34
  • 62
  • Thanks for your help. If I do this: clean() { command dirty "$@" > /dev/null 3>&1 1>&2 2>&3 3>&- | sed 's/Insecure//g' & } I get no errors but also no removal of lines containing Insecure. – chrismead Jan 27 '16 at 17:53