How to use running output from the console in another program

Question

I'm running a program from the command line. This program prints out some of my CPU's statistics (link to program if anyone's interested: https://github.com/opcm/pcm). The program doesn't exit. It'll print updated stats periodically in the same terminal.

I'm trying to use these stats that the program is retrieving in another program that I'm writing myself. The program I'm writing is just supposed to graph some of these CPU statistics vs. time in real time in python.

I'm not sure how to retrieve and use the stats being printed to the console in my program in real time. I know I can direct output to a text file instead of to the console by typing ./program.x > output.txt.

My first thought is to push the outputted statistics to a text file, and simultaneously be reading from that text file in my program. But this feels like a brute force method, and I'm not sure it'll work.

Does anyone know how I can use the real-time output from one program in another program?

Answer 1

You can achieve this result in more than one way.

On a unix system the easiest one i believe would be the use of pipes Lets say you have program1 (p1) that outputs some data and program2(p2) that reads and use this data.

You could pipe the output of p1 to p2 as suggested by @Charles Landau using | which creates a pipe between the stdout and stdin of teh 2 programs

./p1 | p2

If you want to also visualize the output of p1 you could use tee which take the stdin and forks it to the stdout and to a program

./p1 | tee p2

Now This would work only if you intend to launch p1 and p2 at the same time (at which point a believe it would be better to make it a single program). A more generic use of pipe would be for p1 to create a named pipe (FIFO) and for p2 to read from this pipe:

p1.py (server)

import os
from random import randint
import time

FIFO="/tmp/test_fifo"

if not os.path.exists(FIFO):
    os.mkfifo(FIFO)

pipe = os.open( FIFO, os.O_RDWR) #I use this to avoid the program hanging on this step.
                                 #When you open a FIFO, the behavior is blocking, meaning that the program will hang until both end (r/w) of the pipe are being used
f = os.fdopen( pipe, "w")        #Then i want a file object so i can flush it

while 1:
    # Do operation
    output = randint(0,999999)
    # Print output
    print(output)

    # Send output into pipe
    f.flush()
    f.write( str(output) + '\n')
    time.sleep(0.5)

f.close()
pipe.close()

os.unlink(FIFO)

p2.py (client)

import os
import sys

FIFO="/tmp/test_fifo"

if not os.path.exists(FIFO):
    exit()

f = open( FIFO, "r")

while 1:
    read_fifo = f.readline()
    print( read_fifo, end='')
    sys.stdout.flush()

You can do some simple testing on FIFO's in linux also using the shell

mkfifo 1
shell1: tail -f 1
shell2: echo "Hello!" > 1

In a similar manner you could use the TCP protocol for the 2 program to communicate See python https://docs.python.org/3/library/socket.html for more info on this method

Answer 2

It sounds like you want pipes. Pipes take the standard output of a program and pass it directly to the next. The absolute simplest example I can think of:

ls | grep ".bin"

It's easy to remember because the pipe operator kinda looks like a pipe |

More about pipes: http://web.cse.ohio-state.edu/~mamrak.1/CIS762/pipes_lab_notes.html

Edit: using sys.stdin to read pipes forever is explained well here

The important pattern to take from this is right there:

import sys
while True:
    line = sys.stdin.readline()
    # do something with line

Answer 3

You can make something like tail in python. This is sample of code but can't handle if the file is deleted.

def tail_F(some_file):
    while True:
        try:
            for line in sh.tail("-f", some_file, _iter=True):
                yield line
                # do whatever with new line
        except sh.ErrorReturnCode_1:
            yield None