reading a os.popen(command) into a string

Question

I'm not to sure if my title is right. What I'm doing is writing a python script to automate some of my code writing. So I'm parsing through a .h file. but I want to expand all macros before I start. so I want to do a call to the shell to:

gcc -E myHeader.h

Which should out put the post preprocessed version of myHeader.h to stdout. Now I want to read all that output straight into a string for further processing. I've read that i can do this with popen, but I've never used pipe objects.

how do i do this?

Duplicates: http://stackoverflow.com/search?q=%5Bpython%5D+subprocess+output, http://stackoverflow.com/questions/1180606/using-subprocess-popen-for-process-with-large-output Even so, +1 for being clear and well-asked, and for "post preprocessed". :) — , Feb 26 '10 at 04:22

score 37 · Accepted Answer · edited Mar 09 '22 at 01:56

37

The os.popen function just returns a file-like object. You can use it like so:

import os

process = os.popen('gcc -E myHeader.h')
preprocessed = process.read()
process.close()

As others have said, you should be using subprocess.Popen. It's designed to be a safer version of os.popen. The Python docs have a section describing how to switch over.

edited Mar 09 '22 at 01:56

That Brazilian Guy

3,328
5
31
49

answered Feb 26 '10 at 04:26

Brian McKenna

45,528
6
61
60

2

I don't think that the suggestion to replace `os.popen` by `subprocess.Popen` is still valid nowadays. I think this dates back from the transition from Python 2 to Python 3 when `popen`, `popen2`, `popen3` and `popen4` were deprecated. As of Dec. 2021, [the Python documentation for os.popen](https://docs.python.org/3/library/os.html#os.popen) states that "This is implemented using subprocess.Popen; see that class’s documentation for more powerful ways to manage and communicate with subprocesses." – Hemerson Tacon Dec 02 '21 at 16:23

score 17 · Answer 2 · answered Feb 26 '10 at 06:03

17

import subprocess

p = subprocess.popen('gcc -E myHeader.h'.split(),
                     stdout=subprocess.PIPE)
preprocessed, _ = p.communicate()

String preprocessed now has the preprocessed source you require -- and you've used the "right" (modern) way to shell to a subprocess, rather than old not-so-liked-anymore os.popen.

answered Feb 26 '10 at 06:03

Alex Martelli

854,459
170
1,222
1,395

Very nice example. Is there a way to separate the stdout and stderr streams? – smith324 Apr 05 '11 at 06:19
Now `.split()` is one of the things that happen in a shell and make it unsafe to use. (Consider `subprocess.open(cmd.split())`, and the case that somebody passes in `'gcc "My Program.c"'` for `cmd`...) – toolforger Jul 02 '19 at 00:37

score 5 · Answer 3 · edited May 23 '17 at 12:00

5

you should use subprocess.Popen() there are numerous examples on SO

How to get output from subprocess.Popen()

edited May 23 '17 at 12:00

Community

1
1

answered Feb 26 '10 at 04:27

John La Rooy

295,403
53
369
502

Please include an excerpt for the database to protect against link-rot. – SDsolar Mar 29 '18 at 05:44

Paolo Rovelli · Answer 4 · 2013-04-28T19:14:36.003

The os.popen() has been deprecated since Python 2.6. You should now use the subprocess module instead: http://docs.python.org/2/library/subprocess.html#subprocess.Popen

import subprocess

command = "gcc -E myHeader.h"  # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)

#Launch the shell command:
output = process.communicate()

print output[0]

In the Popen constructor, if shell is True, you should pass the command as a string rather than as a sequence. Otherwise, just split the command into a list:

command = ["gcc", "-E", "myHeader.h"]  # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None)

If you need to read also the standard error, into the Popen initialization, you can set stderr to subprocess.PIPE or to subprocess.STDOUT:

import subprocess

command = "gcc -E myHeader.h"  # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

#Launch the shell command:
output, error = process.communicate()

SDsolar · Answer 5 · 2018-03-29T05:45:53.380

Here is another approach which captures both regular output plus error output:

com_str = 'uname -a'
command = subprocess.Popen([com_str], stdout=subprocess.PIPE, shell=True)
(output, error) = command.communicate()
print output

Linux 3.11.0-20-generic  Fri May 2 21:32:55 UTC 2014 GNU/Linux

and

com_str = 'id'
command = subprocess.Popen([com_str], stdout=subprocess.PIPE, shell=True)
(output, error) = command.communicate()
print output

uid=1000(myname) gid=1000(myGID) groups=1000(mygrp),0(root)

reading a os.popen(command) into a string

5 Answers5

Linked