Why are Python 3 Line breaks \n not working with print and subprocess Popen stdout?

Question

Python 3.7.1: Calling grep with subprocess.Popen to pull errors from a log file. When printing to screen, line breaks \n are not processed.

Example

a = subprocess.Popen(["grep", "ERR", "errors.log"], stdout=subprocess.PIPE)
print(a.stdout.read())

Output

ERR 1 ret: 31113\nERR 2 ret: 35523\nERR 3 ret: 3810 (etc.)

Can't imagine why the line breaks are not processed. I would expect this:

ERR 1 ret: 31113
ERR 2 ret: 35523
ERR 3 ret: 3810
(etc.)

Been combing the 'net for answers, but no luck. Thx :^)

References:

How would I specify a new line in Python?
Python Popen grep

I think it is due to the newlines, in my case, changing `subprocess.Popen(["grep", "ERR", "errors.log"], stdout=subprocess.PIPE)` to `subprocess.Popen(["grep", "ERR", "errors.log"], stdout=subprocess.PIPE, bufsize=1, universal_newlines=True)` worked. — Wiktor Stribiżew, May 22 '23 at 09:07

score 3 · Answer 1 · answered Nov 07 '18 at 19:53

3

Specify an encoding to decode the subprocess output with. I can't tell you what your errors.log file is encoded with, but give "utf-8" a try:

a = Popen(["grep", "ERR", "errors.log"], stdout=subprocess.PIPE, encoding='utf-8')

answered Nov 07 '18 at 19:53

wim

338,267
99
616
750

score 2 · Answer 2 · answered Nov 07 '18 at 20:03

Popen standard output and errors are bytes per default. Printing bytes doesn't try to print the linefeeds as such, but prints their representation instead.

Prior to 3.5, you cannot use encoding, but you can certainly decode the output to string using bytes.decode:

a = subprocess.Popen(["grep", "ERR", "errors.log"], stdout=subprocess.PIPE)
print(a.stdout.read().decode())

(decode can need an argument depending on the contents of the file)

Why are Python 3 Line breaks \n not working with print and subprocess Popen stdout?

2 Answers2