fifo - reading in a loop

Question

I want to use os.mkfifo for simple communication between programs. I have a problem with reading from the fifo in a loop.

Consider this toy example, where I have a reader and a writer working with the fifo. I want to be able to run the reader in a loop to read everything that enters the fifo.

# reader.py
import os
import atexit

FIFO = 'json.fifo'

@atexit.register
def cleanup():
    try:
        os.unlink(FIFO)
    except:
        pass

def main():
    os.mkfifo(FIFO)
    with open(FIFO) as fifo:
#        for line in fifo:              # closes after single reading
#        for line in fifo.readlines():  # closes after single reading
        while True:
            line = fifo.read()          # will return empty lines (non-blocking)
            print repr(line)

main()

And the writer:

# writer.py
import sys

FIFO = 'json.fifo'


def main():
    with open(FIFO, 'a') as fifo:
        fifo.write(sys.argv[1])

main()

If I run python reader.py and later python writer.py foo, "foo" will be printed but the fifo will be closed and the reader will exit (or spin inside the while loop). I want reader to stay in the loop, so I can execute the writer many times.

Edit

I use this snippet to handle the issue:

def read_fifo(filename):
    while True:
        with open(filename) as fifo:
            yield fifo.read()

but maybe there is some neater way to handle it, instead of repetitively opening the file...

Related

Getting readline to block on a FIFO

score 8 · Answer 1 · answered Aug 17 '17 at 23:58

8

You do not need to reopen the file repeatedly. You can use select to block until data is available.

with open(FIFO_PATH) as fifo:
    while True:
        select.select([fifo],[],[fifo])
        data = fifo.read()
        do_work(data)

In this example you won't read EOF.

answered Aug 17 '17 at 23:58

steveayre

1,075
11
8

I believe this is the better answer as it doesn't consume CPU cycles with the while loop. – huggie Mar 07 '19 at 09:24
1

It seems if the writer closes, this will loop continuously, in which case this helps: https://stackoverflow.com/questions/39089776/python-read-named-pipe, or use `for line in fifo` instead of `fifo.read()` – huggie Mar 08 '19 at 13:57

glglgl · Accepted Answer · 2013-07-03T14:55:00.947

5

A FIFO works (on the reader side) exactly this way: it can be read from, until all writers are gone. Then it signals EOF to the reader.

If you want the reader to continue reading, you'll have to open again and read from there. So your snippet is exactly the way to go.

If you have mutliple writers, you'll have to ensure that each data portion written by them is smaller than PIPE_BUF on order not to mix up the messages.

edited Jul 03 '13 at 14:55

answered Jul 03 '13 at 13:46

glglgl

89,107
13
149
217

I suppose that for multiple writers I should use something more fancy, because here the writers would close each others FIFOs? And that data from one writer might be mixed with data from the other writer – Jakub M. Jul 03 '13 at 13:52
They don't close each other, and if the data written is small enough, they as well don't interfere. There is a constant `PIPE_BUF` which tells you how large the data packes may become while not interfering each other. – glglgl Jul 03 '13 at 14:54

score 0 · Answer 3 · answered Jan 14 '23 at 06:32

The following methods on the standard library's pathlib.Path class are helpful here:

Here is a demo:

# reader.py
import os
from pathlib import Path

fifo_path = Path("fifo")
os.mkfifo(fifo_path)

while True:
    print(fifo_path.read_text())  # blocks until data becomes available

# writer.py
import sys
from pathlib import Path

fifo_path = Path("fifo")
assert fifo_path.is_fifo()

fifo_path.write_text(sys.argv[1])

fifo - reading in a loop

3 Answers3

Linked