129

Can someone explain why the result I want, "hi", is preceded with a letter 'b' and followed with a newline?

I am using Python 3.3

>>> import subprocess
>>> print(subprocess.Popen("echo hi", shell=True,
                           stdout=subprocess.PIPE).communicate()[0])
b'hi\n'

This extra 'b' does not appear if I run it with python 2.7

Eric Leschinski
  • 146,994
  • 96
  • 417
  • 335
imagineerThat
  • 5,293
  • 7
  • 42
  • 78
  • 1
    What version of Python are you using? – Necrolyte2 Mar 12 '13 at 23:27
  • 2
    Not sure about the 'b', but the newline is because `echo hi` prints `hi\r\n`. To avoid that, you could add .strip() at the end, or similar fix. – azhrei Mar 12 '13 at 23:34
  • 8
    you could use `check_output()` instead of `.communicate()` here: `print(subprocess.check_output("echo hi", shell=True, universal_newlines=True), end="")` – jfs Mar 13 '13 at 00:00

4 Answers4

119

The b indicates that what you have is bytes, which is a binary sequence of bytes rather than a string of Unicode characters. Subprocesses output bytes, not characters, so that's what communicate() is returning.

The bytes type is not directly print()able, so you're being shown the repr of the bytes you have. If you know the encoding of the bytes you received from the subprocess, you can use decode() to convert them into a printable str:

>>> print(b'hi\n'.decode('ascii'))
hi

Of course, this specific example only works if you actually are receiving ASCII from the subprocess. If it's not ASCII, you'll get an exception:

>>> print(b'\xff'.decode('ascii'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0…

The newline is part of what echo hi has output. echo's job is to output the parameters you pass it, followed by a newline. If you're not interested in whitespace surrounding the process output, you can use strip() like so:

>>> b'hi\n'.strip()
b'hi'
Mattie
  • 20,280
  • 7
  • 36
  • 54
  • 1
    How do you get the print() function to print a byte string without a preceeding 'b'? Or do you need to convert it to a unicode string first? – imagineerThat Mar 12 '13 at 23:53
  • I'm curious, when `os.popen` returns text strings, whether there's a way to make `subprocess.Popen` also return them, instead of the byte strings. – Pavel Šimerda Dec 13 '14 at 20:58
  • 13
    I'll answer myself, there's an option with cryptic name called `universal_newlines` that causes the `Popen` object to accept and return text strings. – Pavel Šimerda Dec 13 '14 at 21:24
  • 3
    @PavelŠimerda While os.popen returns text strings, they are apparently being decoded incorrectly for non-ascii characters, at least on Windows. E.g. running `check_output("dir")`, extracting a file name from the output and then trying to access it with `open` will fail if the filename contains German umlauts. Might be a bug. – kdb May 21 '15 at 17:44
97

As mentioned before, echo hi actually does return hi\n, which it is an expected behavior.

But you probably want to just get the data in a "right" format and not deal with encoding. All you need to do is pass universal_newlines=True option to subprocess.Popen() like so:

>>> import subprocess
>>> print(subprocess.Popen("echo hi",
                           shell=True,
                           stdout=subprocess.PIPE,
                           universal_newlines=True).communicate()[0])
hi

This way Popen() will replace these unwanted symbols by itself.

Danil
  • 3,348
  • 2
  • 20
  • 16
  • 20
    `universal_newlines=True` worked like a charm. This should be the accepted answer, in my humble opinion... – Ethan Strider Aug 17 '18 at 23:55
  • 5
    It produces extra empty lines. – LoMaPh Mar 13 '19 at 23:09
  • 7
    You may need *both* `universal_newlines=True` in `Popen` (to get rid of the `b''`) and a `strip()` on the resulting string, if you want to chop the terminating newline. – arielf Feb 05 '20 at 05:22
  • 1
    FYI, [the documentation](https://docs.python.org/3/library/subprocess.html#subprocess.run) says `universal_newlines` is now just a backwards-compatible alias for the `text` parameter, which is clearer but only in Python 3.7 and above. – Harry Cutts Feb 07 '20 at 00:17
  • It produces extra empty lines because it doesn't work. universal_newlines does not remove \n – kol23 Oct 15 '20 at 12:55
  • From Python3.6+ correct way is to open the streams in text mode. This can be done by specifying `encoding` in `Popen`. As per [docs](https://docs.python.org/3.6/library/subprocess.html#subprocess.Popen) `If encoding or errors are specified, the file objects stdin, stdout and stderr are opened in text mode with the specified encoding and errors, as described above in Frequently Used Arguments. If universal_newlines is True, they are opened in text mode with default encoding. Otherwise, they are opened as binary streams` – RedBaron Oct 22 '20 at 06:19
  • @HarryCutts Seems like universal_newlines is present in 3.6, so it can't only be in 3.7 and above. – Lord of Grok Jan 03 '22 at 19:07
  • @teal'c Ah, yes, my comment isn't clearly phrased. `universal_newlines` is present before 3.7; `text` is a more readable alias for it that was added in 3.7. – Harry Cutts Jan 05 '22 at 00:42
27

The echo command by default returns a newline character

Compare with this:

print(subprocess.Popen("echo -n hi", \
    shell=True, stdout=subprocess.PIPE).communicate()[0])

As for the b preceding the string it indicates that it is a byte sequence which is equivalent to a normal string in Python 2.6+

http://docs.python.org/3/reference/lexical_analysis.html#literals

Telemachus
  • 19,459
  • 7
  • 57
  • 79
Necrolyte2
  • 738
  • 5
  • 13
12

b is the byte representation and \n is the result of echo output.

Following will print only the result data

import subprocess
print(subprocess.Popen("echo hi", shell=True,stdout=subprocess.PIPE).communicate()[0].decode('utf-8').strip())
Jenish
  • 579
  • 7
  • 15