Get output colored text with Popen

Question

I'm using popen to make a plugin, this plugin used a external program who shows some colored text in the output.

The output is something like this:

avr-g++ -o .pioenvs\uno\FrameworkArduino\HardwareSerial.o -c -std=gnu++11 -fno-exceptions -fno-threadsafe-statics -g -Os -Wall -ffunction-sections -fdata-sections -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10607 -I.pioenvs\uno\FrameworkArduino -I.pioenvs\uno\FrameworkArduinoVariant .pioenvs\uno\FrameworkArduino\HardwareSerial.cpp    
avr-g++ -o .pioenvs\uno\FrameworkArduino\HardwareSerial0.o -c -std=gnu++11 -fno-exceptions -fno-threadsafe-statics -g -Os -Wall -ffunction-sections -fdata-sections -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10607 -I.pioenvs\uno\FrameworkArduino -I.pioenvs\uno\FrameworkArduinoVariant .pioenvs\uno\FrameworkArduino\HardwareSerial0.cpp
=============================================
Path\file.cpp: in function 'void loop()':
Path\file.cpp:23:2 error: expected ';' before '}' token
}
^
=============================================

All inside "=" is in red and yellow.

When I run the command in the command console, I can see the full output, but when I use Popen I only can get the uncolored text

This is how I'm using Popen

process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, cwd=self.cwd, universal_newlines=True, shell=True)
output = process.communicate()

stdout = output[0]
stderr = output[1]

print(stdout)
print(stderr)

I would like to get the text even if it's not colored, the important is to get the full log.

Any suggestion will be appreciated

Kindly vote up my answer if it helped to fix your problem. :) — ‌‌R‌‌‌., Jan 06 '16 at 04:08

score 4 · Accepted Answer · answered Jan 06 '16 at 03:41

You are not able to get all that message because a part of your command output is not regular output, it is considered as Error (or log or debug messages)

Now you can either add stderr=subprocess.PIPE to your Popen's parameters, which will put all the errors in your stderr variable:

process = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=self.cwd, universal_newlines=True, shell=True)
output = process.communicate()

stdout = output[0]
stderr = output[1]

print(stdout)
print(stderr)

OR if you want to have all the errors and output just like what you see in the console, add 2>&1 at the end of your command. something like:

avr-g++ -o .pioenvs\uno\FrameworkArduino\HardwareSerial.o -c -std=gnu++11 -fno-exceptions -fno-threadsafe-statics -g -Os -Wall -ffunction-sections -fdata-sections -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO_AVR_UNO -DARDUINO=10607 -I.pioenvs\uno\FrameworkArduino -I.pioenvs\uno\FrameworkArduinoVariant .pioenvs\uno\FrameworkArduino\HardwareSerial.cpp 2>&1

Thanks @Rahman, the last suggestion solved my problem. Do you have a link to read something about that solution, or can you explain me the meaning of `2>&1`? — GEPD, Jan 06 '16 at 04:10
@GEPD Here's the link that I read before answering you http://stackoverflow.com/a/818284/811922 — ‌‌R‌‌‌., Jan 06 '16 at 04:12
@GEPD: And to see how to preserve the color codes take a look at `-fdiagnostics-color[=WHEN]` in the [avr-g++ man page](https://manned.org/avr-g++). — PM 2Ring, Jan 06 '16 at 06:40
Thanks @PM2Ring I'll look now, I'm not directly working with avr-g++ but it is good to know this extra information. — GEPD, Jan 06 '16 at 13:19

score 3 · Answer 2 · edited May 23 '17 at 12:16

When I run the command in the command console, I can see the full output, but when I use Popen I only can get the uncolored text

There could be two issues:

the colored text happens to be produced on stderr, not stdout. To capture stderr, set stderr=PIPE, as @Rahman suggested
avr-g++ command may disable colors if it detects that the output is not a tty e.g., if it is redirected to a pipe like in your case. To enable colors, pass the command-line option -fdiagnostics-color=always as @PM 2Ring suggested.

Or (less portable but it may work with more commands) provide a pseudo-tty to hoodwink the avr-g++ command into thinking that it runs interactively (and therefore it should enable colors). You could do it using pty.spawn(), pexpect module:
```
import pexpect # $ pip install pexpect

output, exitstatus = pexpect.runu(command, withexitstatus=1)
```
or (more low-level) using pty.openpty() + subprocess module.

Get output colored text with Popen

2 Answers2