how to use popen() to grab stdout from child programs in Linux to a
main C++ program
Are you familiar with the UNIX philosophy? You might read the beginning of the Wikipedia article "UNIX Philosophy". Here is a snippet:
The UNIX philosophy is documented by Doug McIlroy[1] in the Bell
System Technical Journal from 1978:[2]
1) Make each program do one thing well. To do a new job, build afresh
rather than complicate old programs by adding new "features".
2) Expect the output of every program to become the input to another, as
yet unknown, program. Don't clutter output with extraneous
information. Avoid stringently columnar or binary input formats. Don't
insist on interactive input.
... and more ...
I prefer popen to access many of the Linux tools I find useful. Examples include sort, unique, and sha256sum, and several others. But you may notice the many equivalents are moving into the C++ libraries (std::sort etc) and, with effort, I have found, for example, the code of sha256sum, suitable for compiling within my c++ app.
1) When launched, popen is given a string, which can be in one part identifying an app to run, or in multiple parts, the first is an app, and additions strings are parameters to hand to that app just as if a command line input. Just as in the shell (hint)
2) When launched in 'r' mode, the stdout pipe of sha256sum is directed into the FILE* handle opened by your invocation of popen. Thus the output is 'fed back' to your program.
But I can't understand how this stuff works.
Not really a useful comment, and not a question. Your snippet of code looks straight forward to me (though substantially c code). Perhaps you should pick a line of code and ask something specific about it?
But, before you work too hard on your code, let me perform some initial investigation where we will manually run the code we wish to use with popen, and capture results for review against your app's output.
Investigation 1 -
In your snippet, you are trying the command, "which driftnet". An unfortunate choice when the program driftnet does not exist on your PATH.
Results on my system:
$ which driftnet
$
Because the command does not exist,, the command 'which' simply returns an empty result.
Hard to tell if we did something wrong or not. Without driftnet, I can't help you much either.
Investigation 2 - a somewhat more interesting trial.
sha256sum requires a file for input ... Here I provide my 'C++ Hello World' program, which you might capture to file "HelloWorld.cc" (or any where you want), or use any of your own files.
// If your C++ 'Hello World' has no class ... why bother?
#include <iostream>
class Hello_t {
public:
Hello_t() { std::cout << "\n Hello" << std::flush; }
~Hello_t() { std::cout << "World!" << std::endl; }
void operator() () { std::cout << " C++ "; }
};
int main(int, char**) { Hello_t()(); }
A total of 10 lines of code, starts with a comment, has 1 blank line after the single include. I have my editor set to auto-magically remove trailing spaces. If yours leaves a space in, your result may be different.
So, what happens when you calculate the sha256sum of the file "HelloWorld.cc" from the command line? No need to speculate ... just TRY IT!
On the command line, invoke the command and see its response on the next line:
$ sha256sum HelloWorld.cc
f91c6c288325fb4f72546482a9b4f1fa560f05071a12aa2a1268474ea3eeea3e HelloWorld.cc
Perhaps you might try your popen code with sha256sum and a known file?
Now you can see what the output (of your popen code) might contain. For this investigation, **the command sha256sum result is 1 long string with a space in it.
FYI - I broke my C++ popen code a few months back during an overhaul of my library. When I get it fixed, I plan to revisit your effort to see how you are progressing.
If your code starts working ... I encourage you to submit your own answer!
If you can identify more more specific question, refine the question.
I recommend you move to C++ code ... if you are going to learn something, and you want to learn C++, no use studying what you've found.
My code is working again. (update 2017/8/2)
But, first let me share a couple of consequences of UNIX Philosophy, in no particular order:
Consequence 1 - pipes and filters
Filters are programs that do one thing well. They accept an input stream, and produce an output stream, typically 'removing' something.
In the field of code metrics, management can review line counts to get a sense of progress (line count growth slowing), and problems (not slowing).
Every one has an opinion on code metrics ... not every line in a file should be counted. For example, I think white space is important, but a line of white space should not be counted as code. Similarly, a full line of comment is not code. IMHO.
In the following command
f1 < pfn | f2 | f3 | f4
f1..fn are filters, all submitted as one command to the shell. In my code metrics tool, I created
f1: ebl erase blank lines
f2: ecppc erase cpp comments
f3: ecc erase c comments
and several more.
Thus
ebl < HelloWorld.cc | ecppc | ecc | ebl | wc
is a single string submittal to the shell that ultimately cout's 3 numbers.
The UNIX like command 'wc' probably exists on your system. On Ubuntu 15.10:
$ wc < HelloWorld.cc
10 52 309
Indicating HelloWorld.cc has 10 lines (without filtering).
Consequence 2 - command lists
A command list is a list of shell commands that each do one thing well. The list identifies a time sequence of execution. A more general idea than pipe and filter.
As an example, I have set up my editor (emacs) with an assigned key (F8) to provide the following compile command:
USER_FLAGS='-O0 ' ; export USER_FLAGS ; time make CC='g++-5 -m64 ' dumy514 ; ./dumy514
The specific file pfn's are perhaps entered once, but kept in a history buffer inside of emacs and thus no need to reenter them.
Note the 3 semi-colons. They delimit the multiple commands in one string submitted to the Linux command shell. A 3 element command list: set an environment variable to override the default option; export the new flag to make it available to the compiler; and launch a make.
So? popen() accepts command lists. A 'command list' like shell might accept.
I finally read the "man popen" response with sufficient care.
From the DESCRIPTION:
The popen() function opens a process by creating a pipe, forking, and
invoking the shell.
Your command given to popen, runs in a shell.
But I can't understand how this stuff works.
This 'stuff' should start to make sense, now. We are accessing the shell to process a command / command list / pipe filters. This starts the 'other' process, and interacts with the first process using the PIPE.
So how does the output of a command processed in the shell delivered to my code? some clues to understanding follow. Some unit test demo's are included, where you can easily change a popen cmd, and then compile and run the new cmd.
The following provides code to use popen in one of two modes. Yes they are in 3 class, a base and two derived. The hope is that this allows you to have more than one of these running at a time. (Not terribly useful on my 2 core machine.) Feel free to refactor the code, keeping just what you need. The rest will be here for your continued guidance.
#include <iostream>
#include <sstream>
#include <string>
#include <cstring> // sterror
#include <cstdio> // popen, FILE, fgets, fputs
#include <cassert>
class POpen_t // access to ::popen
{
protected:
FILE* m_FILE;
std::string m_cmd;
public:
POpen_t(void) : m_FILE(nullptr)
{ }
virtual ~POpen_t(void) {
if (m_FILE) (void)close();
m_FILE = 0;
}
// on success: 0 == return.size(), else returns error msg
std::string close()
{
std::stringstream errSS;
do // poor man's try block
{
// pclose() returns the term status of the shell cmd
// otherwise -1 and sets errno.
assert(nullptr != m_FILE); // tbr - some sort of logic error
// tbr if(0 == m_FILE) break; // success?
int32_t pcloseStat = ::pclose(m_FILE);
int myErrno = errno;
if (0 != pcloseStat)
{
errSS << "\n POpen_t::close() errno " << myErrno
<< " " << std::strerror(myErrno) << std::endl;
break;
}
m_FILE = 0;
}while(0);
return(errSS.str());
} // std::string close(void)
}; // class POpen_t
I think it will be easier for copy/paste to split these into 3 sections. But remember, I have all the code in one file. Not a single header file, but it works for me. You can refactor into .h and .cc files, if you wish.
class POpenRead_t : public POpen_t // access to ::popen read-mode
{
public:
POpenRead_t(void) { }
// ::popen(aCmd): opens a process (fork), invokes shell,
// and creates a pipe
// returns NULL if the fork or pipe calls fail,
// or if it cannot allocate memory.
// on success: 0 == return.size(), else returns error msg
std::string open (std::string aCmd)
{
std::stringstream errSS; // 0 == errSS.str().size() is success
assert (aCmd.size() > 0);
assert (0 == m_FILE); // can only use serially
m_cmd = aCmd; // capture
do // poor man's try block
{
if(true) // diagnosis only
std::cout << "\n POpenRead_t::open(cmd): cmd: '"
<< m_cmd << "'\n" << std::endl;
// ::popen(aCmd): opens a process by creating a pipe, forking,
// and invoking the shell.
// returns NULL if the fork or pipe calls fail,
// or if it cannot allocate memory.
m_FILE = ::popen (m_cmd.c_str(), "r"); // create 'c-stream' (FILE*)
// ^^ function is not in namespace std::
int myErrno = errno;
if(0 == m_FILE)
{
errSS << "\n POpenRead_t::open(" << m_cmd
<< ") popen() errno " << myErrno
<< " " << std::strerror(myErrno) << std::endl;
break;
}
} while(0);
return (errSS.str());
} // std::string POpenRead_t::open(std::string aCmd)
// success when 0 == errStr.size()
// all outputs (of each command) captured into captureSS
std::string spinCaptureAll(std::stringstream& captureSS)
{
const int BUFF_SIZE = 2*1024;
std::stringstream errSS; // capture or error
do
{
if(0 == m_FILE)
{
errSS << "\n ERR: POpenRead_t::spinCaptureAll(captureSS) - m_FILE closed";
break;
}
size_t discardedBlankLineCount = 0;
do
{
// allocate working buff in auto var, fill with nulls
char buff[BUFF_SIZE] = { 0 };
if(true) { for (int i=0; i<BUFF_SIZE; ++i) assert(0 == buff[i]); }
// char * fgets ( char * str, int num, FILE * c-stream );
// Reads characters from c-stream and stores them as a C string
// into buff until
// a) (num-1) characters have been read
// b) a newline or
// c) the end-of-file is reached
// whichever happens first.
// A newline character makes fgets stop reading, but it is considered
// a valid character by the function and included in the string copied
// to str
// A terminating null character is automatically appended after the
// characters copied to str.
// Notice that fgets is quite different from gets: not only fgets
// accepts a c-stream argument, but also allows to specify the maximum
// size of str and includes in the string any ending newline character.
// fgets() returns buff or null when feof()
char* stat = std::fgets(buff, // char*
BUFF_SIZE, // count - 1024
m_FILE); // c-stream
assert((stat == buff) || (stat == 0));
int myErrno = errno; // capture
if( feof(m_FILE) ) { // c-stream eof detected
break;
}
// when stat is null (and ! feof(m_FILE) ),
// even a blank line contains "any ending newline char"
// TBD:
// if (0 == stat) {
// errSS << "0 == fgets(buff, BUFF_SIZE_1024, m_FILE) " << myErrno
// << " " << std::strerror(myErrno) << std::endl;
// break;
// }
if(ferror(m_FILE)) { // file problem
errSS << "Err: fgets() with ferror: " << std::strerror(myErrno);
break;
}
if(strlen(buff)) captureSS << buff; // additional output
else discardedBlankLineCount += 1;
}while(1);
if(discardedBlankLineCount)
captureSS << "\n" << "discarded blank lines: " << discardedBlankLineCount << std::endl;
} while(0);
return (errSS.str());
} // std::string POpenRead_t::spinCaptureAll(std::stringstream& ss)
}; // class POpenRead_t
And now POpenWrite_t:
class POpenWrite_t : public POpen_t // access to ::popen
{
public:
POpenWrite_t(void) { }
// ::popen(aCmd): opens a process (fork), invokes the non-interactive shell,
// and creates a pipe
// returns NULL if the fork or pipe calls fail,
// or if it cannot allocate memory.
// on success: 0 == return.size(), else returns error msg
std::string open (std::string aCmd)
{
std::stringstream errSS; // 0 == errSS.str().size() is success
assert (aCmd.size() > 0);
assert (0 == m_FILE);
m_cmd = aCmd; // capture
do // poor man's try block
{
if(true) // diagnosis only
std::cout << "\n POpenWrite_t::open(cmd): cmd: \n '"
<< "'" << m_cmd << std::endl;
m_FILE = ::popen (m_cmd.c_str(), "w"); // use m_FILE to write to sub-task std::in
int myErrno = errno;
if(0 == m_FILE)
{
errSS << "\n POpenWrite_t::open(" << m_cmd
<< ") popen() errno " << myErrno
<< " " << std::strerror(myErrno) << std::endl;
break;
}
} while(0);
return (errSS.str());
} // std::string POpenWrite_t::open(std::string aCmd)
// TBR - POpenWrite_t::write(const std::string& s)
// work in progress - see demo write mode
}; // class POpenWrite_t
Next I have what I call T514_t, a class which defines my unit test or demo efforts. Entry is in 'exec()'.
It turns out that the shell used in popen might be different than the shell you experience in a terminal. During the start up of a session, part of shell decides if it is in an interactive (terminal) or non-interactive (no terminal) mode. On my system, the terminal / interactive shell is 'bash', and the popen / non-interactive shell is 'sh'. Remember this when developing shell commands for popen.
An interesting use of POpenRead_t is in the method "std::string shellCheck(std::stringstream& rsltSS)".
class T514_t
{
const std::string line = "\n------------------------------------------------------";
const char escape = 27;
public:
T514_t() = default;
~T514_t() = default;
std::string exec (int argc, char* argv[],
std::stringstream& resultSS )
{
std::string errStr;
errStr = shellCheck(resultSS); // when not bash, make a choice
if (0 == errStr.size()) // shell is expected
{
// bash reported, continue
switch (argc)
{
case 2 : { errStr = exec (argv, resultSS); } break;
default: { errStr = " User error"; usage(argv); } break;
} // switch (argc)
}
return (errStr); // when no err, errStr.size() is 0
} // int exec()
private:
std::string exec(char* argv[], std::stringstream& resultSS)
{
std::string errStr;
std::cout << clrscr() << std::flush;
switch (std::atoi(argv[1]))
{
case 1:
{
std::cout << clrscr() << boldOff() << line;
errStr = demoReadMode(resultSS);
std::cout << boldOff() << std::endl;
} break;
case 2:
{
std::cout << clrscr() << boldOn() << line;
errStr = demoWriteMode();
std::cout << boldOff() << std::endl;
} break;
default: { usage(argv); } break;
} // switch (std::atoi(argv[1]))
return(errStr);
} // int exec(char* argv[], std::stringstream& resultSS)
// Ubuntu has 5 (or more) shells available
// 1) Bourne shell (' sh'), 2) C shell (' csh'), 3) TC shell (' tcsh'),
// 4) Korn shell (' ksh'), 5) Bourne Again shell (' bash')
//
// bash is my interactive shell
// sh is my non-interactive shell
// which (mildly) influences the cmds of the demo's
//
// when not bash, what do you want to do?
std::string shellCheck(std::stringstream& rsltSS)
{
std::stringstream errValSS;
std::string cmd;
cmd += "ps -p \"$$\" ";
{
POpenRead_t popenR;
errno = 0;
std::string rStat = popenR.open(cmd);
int myErrno = errno;
if(0 != rStat.size()) {
errValSS << "\n Err: " << cmd << " failed. rStat: "
<< std::strerror(myErrno) << std::endl;
return(errValSS.str());
}
do
{
errno = 0;
std::string errSS = popenR.spinCaptureAll (rsltSS);
myErrno = errno;
if (false) { // dianosis only
std::cout << "\n demoReadMode() ss/myErrno/s:\n"
<< rsltSS.str() << " "
<< myErrno << " '"
<< errSS << "'" << std::endl;
}
break;
if(0 != myErrno)
{
errValSS << "\n Err: popenR.spinCaputureAll(): cmd / strerror(myErrno): "
<< cmd << " / " << std::strerror(myErrno) << std::endl;
return(errValSS.str());
}
if (0 == rsltSS.str().size()) { // TBR
std::cout << "\n demoReadMode: ss.str().size() is 0, indicating completed" << std::endl;
} break;
}while(1);
// NOTE: pclose() returns the termination status of the shell command
// otherwise -1 and sets errno.
// errno = 0;
(void)popenR.close(); // return value is term status of the shell command
if(errno != 0)
{
errValSS << "\n Err: POpen_t::close() - cmd: " << cmd
<< " err:" << std::strerror(errno) << std::endl;
}
}
std::string s = rsltSS.str();
std::string nishell (" sh"); // non-interactive shell expected is: " sh"
size_t indx = s.find(nishell); // my interactive shell is bash
// clear
rsltSS.str(std::string()); rsltSS.clear(); // too much info
if (std::string::npos != indx)
{
// normally I would not include a 'success' message (not the 'UNIX pholosopy'),
// but here I have added to the (success) results:
if(true) //<-- feel free to comment out or delete this success action
rsltSS << "\n the reported non-interactive shell is the required '"
<< nishell << "' ... continuing.\n" << std::endl;
}
else
{
// TBR - when non-interactive shell is unexpectedly different that nishell,
// this demo code aborts ... (with no results) and the error msg:
errValSS << "\n the reported non-interactive shell is not " << nishell
<< " \n" << s << "\n ... aborting\n";
// alternative actions are _always_ possible
// untested examples:
// the shell can be temporarily changed as part of cmd (untested)
// the user could change the non-interactive shell
// your use of popen (POpen_t classes above)
// can change 'cmd's based on the discovered shell
}
return(errValSS.str());
} // std::string shellCheck(std::stringstream& rsltSS)
std::string demoReadMode(std::stringstream& rsltSS)
{
std::stringstream errValSS;
std::string cmd;
int i = 1;
cmd += "./HelloWorld ; ";
cmd += "echo " + std::to_string(i++) + " ; ";
cmd += "./HelloWorld ; ";
cmd += "sha256sum HelloWorld.cc ; ";
cmd += "echo " + std::to_string(i++) + " ; ";
cmd += "./HelloWorld ; ";
cmd += "echo TEST WORKS ; ";
cmd += "./HelloWorld";
{
POpenRead_t popenR;
errno = 0;
std::string rStat = popenR.open(cmd);
int myErrno = errno;
if(0 != rStat.size()) {
errValSS << "\n Err: " << cmd << " failed. rStat: "
<< std::strerror(myErrno) << std::endl;
return(errValSS.str());
}
do
{
errno = 0;
std::string errSS = popenR.spinCaptureAll (rsltSS);
myErrno = errno;
if (false) { // dianosis only
std::cout << "\n demoReadMode() ss/myErrno/s:\n"
<< rsltSS.str() << " "
<< myErrno << " '"
<< errSS << "'" << std::endl;
}
break;
if(0 != myErrno)
{
errValSS << "\n Err: popenR.spinCaputureAll(): cmd / strerror(myErrno): "
<< cmd << " / " << std::strerror(myErrno) << std::endl;
return(errValSS.str());
}
if (0 == rsltSS.str().size()) { // TBR
std::cout << "\n demoReadMode: ss.str().size() is 0, indicating completed" << std::endl;
} break;
}while(1);
// NOTE: pclose() returns the termination status of the shell command
// otherwise -1 and sets errno.
// errno = 0;
(void)popenR.close(); // return value is term status of the shell command
if(errno != 0)
{
errValSS << "\n Err: POpen_t::close() - cmd: " << cmd
<< " err:" << std::strerror(errno) << std::endl;
}
}
return(errValSS.str());
} // std::string demoReadMode(std::stringstream& rsltSS)
std::string demoWriteMode()
{
std::stringstream errValSS;
std::string cmd;
int i = 1;
cmd += "./HelloWorld ; ";
cmd += "echo " + std::to_string(i++) + " ; ";
cmd += "./HelloWorld ; ";
cmd += "sha256sum HelloWorld.cc ; ";
cmd += "echo " + std::to_string(i++) + " ; ";
cmd += "./HelloWorld ; ";
{
POpenWrite_t popenW; // popen in write mode
// errno = 0;
std::string wStat = popenW.open(cmd);
int myErrno = errno;
if (0 != wStat.size()) {
errValSS << "\n Err: " << cmd << "\n failed. wStat: "
<< std::strerror(myErrno) << std::endl;
return(errValSS.str());
}
// tbd - Work in Progress - what command to receive what data from here
// login - needs root, tbr - can cmd contain sudo? probably
//
//
// NOTE: pclose() returns the termination status of the shell command
// otherwise -1 and sets errno.
// errno = 0;
(void)popenW.close(); // return value is term status of the shell command
if(errno != 0)
{
errValSS << "\n Err: POpen_t::close() - cmd: " << cmd
<< " err:" << std::strerror(errno) << std::endl;
}
}
return(errValSS.str());
} // std::string demoWriteMode()
void usage(char* argv[])
{
std::cout << " executable: "<< argv[0]
<< "\n USAGE - user must select test mode"
<< "\n test"
<< "\n 1 - demoReadMode"
<< "\n 2 - demoWriteMode"
<< std::endl;
}
std::string boldOff(){std::string rV; rV.push_back(escape); rV += "[21m"; return(rV); } // bold off
std::string boldOn() {std::string rV; rV.push_back(escape); rV += "[1m"; return(rV); } // bold on
inline std::string clrscr(void) {
std::stringstream ss;
ss << static_cast<char>(escape) << "[H" // home
<< static_cast<char>(escape) << "[2J"; // clrbos
return(ss.str());
}
}; // class T514_t
And finally, my main ... There are only a few things I care to do here.
int main(int argc, char* argv[])
{
std::string errStr;
std::stringstream resultSS;
{
T514_t t514;
errStr = t514.exec(argc, argv, resultSS);
}
// display result
if (resultSS.str().size())
std::cout << "\n\n "<< resultSS.str() << std::endl;
if(errStr.size())
std::cerr << errStr << std::endl;
return(static_cast<int>(errStr.size()));
}
So, build, and play with the std::string cmd; in the mode you are interested in. Good luck.
Questions welcome.
