How, can I pipe the output from the first process (function) to the next process (function) in python?

Question

In the following program

1) I want to write the output to a file 2) And, also want to access the output file downstream in another process by piping the final output of the file, rather than reading from it.

With the following python code:

global product
product = ""
product_file = open('product.txt', 'w')   

def read_file():   
    file1 = open ('file01.txt', 'r')
    data1 = read.file1().rstrip('\n')
    data1_lines = data1.split('\n)
    for lines in data1_lines:
        columns = lines.split('\t')
        x = int(columns[0])
        y = int(columns[1])
        prod = x*y
        output = open('product.txt', 'a')
        output.write(str(x) + '\t' + str(y) + '\t' + str(prod))
        output.close()
        product += str(x +'\t'+ y +'\t'+ prod +'\n')

def manipulate_file():
    global product;
    for lines in product:
        line = product.split('\t')
        do..something.....

I want to access the final output file from def read_file() to be used in the downstream process (function i.e def mainpulate_file()) rather than opening it again.

I want to use either subprocess.call and/or stdin-out and/or tempfile and finally clear the memory when done.

I read several examples but could not find anything clear to work out the process.

I would appreciate any explanation.

Kritzefitz · Answer 1 · 2016-12-11T23:13:48.090

0

If you want to be able to pipe the output you currently write to product.txt to some other process, you have to write it to stdout. This can be done like this:

line = str(x) + '\t' + str(y) + '\t' + str(prod)
output.write(line)
print(line, end='')

The end='' part is to prevent printing line breaks after each entry, as your current code does not write them to the file either. If you actually want line breaks you could get them like this:

output.write(line + '\n')
print(line)

If you actually just want to work with the output later in your script, you can just create list before entering the loop and append each line to that list. After the loop you can concatenate all the lines by doing '\n'.join(your_list).

edited Dec 11 '16 at 23:13

answered Dec 11 '16 at 23:02

Kritzefitz

2,644
1
20
35

I just updated the question. Also, can you show my how you would use `stdout` to be able to use it in next/downstream function. – everestial007 Dec 11 '16 at 23:06
@everestial007 If you want to use the string later in your script, stdout is not helpful at all. I updated my answer to show hoe you can keep your produced data. See [this question](http://stackoverflow.com/q/3385201/2211273) to learn about stdout. – Kritzefitz Dec 11 '16 at 23:12
I want to store the file (information) generated from function #1 and pipe it as `stdin` for next function. `print` writes the ouput to the screen for each `for-loop` session. But, I want to pipein the final output (not the last line) to the next function. I think I am not understanding your answer. But, thanks ! – everestial007 Dec 11 '16 at 23:22
1

@everestial007 As I said in my previous comment, if you want to use your produced string in the same script neither `stdin` nor `stdout` are gonna help you. As you already noticed, everything you write to `stdout` (this is what `print` does) goes to your terminal. What you want to do is actually to just keep your produced string around in a variable and use that variable later on. That's what I tried to express with the last paragraph about list concatenation. – Kritzefitz Dec 12 '16 at 10:06

everestial007 · Accepted Answer · 2016-12-24T00:25:03.700

We don't even need global argument.

product = ""
product_file = open('product.txt', 'w')   

def read_file():   
    file1 = open ('file01.txt', 'r')
    data1 = read.file1().rstrip('\n')
    data1_lines = data1.split('\n)
    for lines in data1_lines:
        columns = lines.split('\t')
        x = int(columns[0])
        y = int(columns[1])
        prod = x*y
        output = open('product.txt', 'a')
        output.write(str(x) + '\t' + str(y) + '\t' + str(prod))
        output.close()
        product += str(x +'\t'+ y +'\t'+ prod +'\n')
    return manipulate_file(product)

def manipulate_file(product):
    data = StringIO(product)
    data_lines = data.read().rstrip('\n).split('\n')
    for lines in data_lines:
        do..something.....

So:

Just simply create a variable that can be updated in the loop.
Return this variable with the defined function
Read the file with the function it was returned too, but this needs to be read using StringIO since the data is on the console.

How, can I pipe the output from the first process (function) to the next process (function) in python?

2 Answers2