When working with a named pipe is there a way to do something like readlines()

Question

Overall Goal: I am trying to read some progress data from a python exe to update the progress of the exe in another application

I have a python exe that is going to do some stuff, I want to be able to communicate the progress to another program. Based on several other Q&A here I have been able to have my running application send progress data to a named pipe using the following code

import win32pipe
import win32file
import glob
test_files = glob.glob('J:\\someDirectory\\*.htm')
# test_files has two items a.htm and b.htm
p = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
                          win32pipe.PIPE_ACCESS_DUPLEX,
                          win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
                          1,65536,65536,300,None)
# the following line is the server-side function for accepting a connection
# see the following SO question and answer
""" http://stackoverflow.com/questions/1749001/named-pipes-between-c-sharp-and-python
"""
win32pipe.ConnectNamedPipe(p, None)
for each in testFiles:
    win32file.WriteFile(p,each + '\n')
#send final message
win32file.WriteFile(p,'Process Complete')
# close the connection
p.close()

In short the example code writes the path of the each file that was globbed to the NamedPipe - this is useful and can be easily extended to more logging type events. However, the problem is trying to figure out how to read the content of the named pipe without knowing the size of each possible message. For example the first file could be named J:\someDirectory\a.htm, but the second could have 300 characters in the name.

So far the code I am using to read the contents of the pipe requires that I specify a buffer size

First establish the connection

file_handle = win32file.CreateFile("\\\\.\\pipe\\wfsr_pipe",
                          win32file.GENERIC_READ | win32file.GENERIC_WRITE,
                          0, None,
                          win32file.OPEN_EXISTING,
                          0, None)

and then I have been playing around with reading from the file

data = win32file.ReadFile(file_handle,128)

This generally works but I really want to read until I hit a newline character, do something with the content between when I started reading and the newline character and then repeat the process until I get to a line that has Process Complete in the line

I have been struggling with how to read only until I find a newline character (\n). I basically want to read the file by lines and based on the content of the line do something (either display the line or shift the application focus).

Based on the suggestion provided by @meuh I am updating this because I think there is a dearth of examples, guidance in how to use pipes

My server code

import win32pipe
import win32file
import glob
import os

p = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
                          win32pipe.PIPE_ACCESS_DUPLEX,
                          win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
                          1,65536,65536,300,None)
# the following line is the server-side function for accepting a connection
# see the following SO question and answer
""" http://stackoverflow.com/questions/1749001/named-pipes-between-c-sharp-and-python
"""
win32pipe.ConnectNamedPipe(p, None)
for file_id in glob.glob('J:\\level1\\level2\\level3\\*'):
    for filer_id in glob.glob(file_id + os.sep + '*'):
        win32file.WriteFile(p,filer_id)    
#send final message
win32file.WriteFile(p,'Process Complete')

# close the connection
p.close()  #still not sure if this should be here, I need more testing
# I think the client can close p

The Client code

import win32pipe
import win32file

file_handle = win32file.CreateFile("\\\\.\\pipe\\wfsr_pipe",
                               win32file.GENERIC_READ |
                               win32file.GENERIC_WRITE,
                               0, None,win32file.OPEN_EXISTING,0, None)
# this is the key, setting readmode to MESSAGE
win32pipe.SetNamedPipeHandleState(file_handle,
                              win32pipe.PIPE_READMODE_MESSAGE, None, None)
# for testing purposes I am just going to write the messages to a file
out_ref = open('e:\\testpipe.txt','w')

dstring = ''  # need some way to know that the messages are complete
while dstring != 'Process Complete':
    # setting the blocksize at 4096 to make sure it can handle any message I
    # might anticipate
    data = win32file.ReadFile(file_handle,4096)
# data is a tuple, the first position seems to always be 0 but need to find 
# the docs to help understand what determines the value, the second is the 
# message
    dstring = data[1]
    out_ref.write(dstring + '\n')

out_ref.close() # got here so close my testfile
file_handle.close() # close the file_handle

Maybe you could establish a convention where the sending process sends a fixed-length integer indicating the line length, followed by the line? — augurar, Aug 18 '15 at 06:53

score 2 · Accepted Answer · answered Aug 18 '15 at 07:15

2

I don't have windows but looking through the api it seems you should convert your client to message mode by adding after the CreateFile() the call:

win32pipe.SetNamedPipeHandleState(file_handle,
    win32pipe.PIPE_READMODE_MESSAGE, None, None)

then each sufficiently long read will return a single message, ie what the other wrote in a single write. You already set PIPE_TYPE_MESSAGE when you created the pipe.

answered Aug 18 '15 at 07:15

meuh

11,500
2
29
45

Very nice and simple - thank you for this. the key for me to make this work was doing a read of as you said some generic length, long enough to get the message as the pipe is capable of emitting messages – PyNEwbie Aug 18 '15 at 15:42

score 1 · Answer 2 · answered Aug 18 '15 at 08:41

You could simply use an implementation of io.IOBase that would wrap the NamedPipe.

class PipeIO(io.RawIOBase):
    def __init__(self, handle):
        self.handle = handle
    def read(self, n):
        if (n == 0): return ""
        elif n == -1: return self.readall()
        data = win32file.ReadFile(self.file_handle,n)
        return data
    def readinto(self, b):
        data = self.read(len(b))
        for i in range(len(data)):
            b[i] = data[i]
        return len(data)
    def readall(self):
        data = ""
        while True:
            chunk = win32file.ReadFile(self.file_handle,10240)
            if (len(chunk) == 0): return data
            data += chunk

BEWARE : untested, but it should work after fixing the eventual typos.

You could then do:

with PipeIO(file_handle) as fd:
    for line in fd:
        # process a line

Dunes · Answer 3 · 2015-08-18T14:40:27.433

You could use the msvcrt module and open to turn the pipe into a file object.

Sending code

import win32pipe
import os
import msvcrt
from io import open

pipe = win32pipe.CreateNamedPipe(r'\\.\pipe\wfsr_pipe',
                          win32pipe.PIPE_ACCESS_OUTBOUND,
                          win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_WAIT,
                          1,65536,65536,300,None)
# wait for another process to connect
win32pipe.ConnectNamedPipe(pipe, None)
# get a file descriptor to write to
write_fd = msvcrt.open_osfhandle(pipe, os.O_WRONLY)
with open(write_fd, "w") as writer:
    # now we have a file object that we can write to in a standard way
    for i in range(0, 10):
        # create "a\n" in the first iteration, "bb\n" in the second and so on
        text = chr(ord("a") + i) * (i + 1) + "\n"
        writer.write(text)

Receiving code

import win32file
import os
import msvcrt
from io import open

handle = win32file.CreateFile(r"\\.\pipe\wfsr_pipe",
                          win32file.GENERIC_READ,
                          0, None,
                          win32file.OPEN_EXISTING,
                          0, None)
read_fd = msvcrt.open_osfhandle(handle, os.O_RDONLY)

with open(read_fd, "r") as reader:
    # now we have a file object with the readlines and other file api methods
    lines = reader.readlines()

print(lines)

Some notes.

I've only tested this with python 3.4, but I believe you may be using python 2.x.
Python seems to get weird if you try to close both the file object and the pipe..., so I've only used the file object (by using the with block)
I've only created the file objects to read on one end and write on the other. You can of course make the file objects duplex by
- Creating the file descriptors (read_fd and write_fd) with the os.O_RDWR flag
- Creating the file objects in in "r+" mode rather than "r" or "w"
- Going back to creating the pipe with the win32pipe.PIPE_ACCESS_DUPLEX flag
- Going back to creating the file handle object with the win32file.GENERIC_READ | win32file.GENERIC_WRITE flags.

I appreciate your effort - but I receive an _TypeError_ in the writing code and so when I test it 'write_fd = msvcrt.open_osfhandle(pipe, os.O_WRONLY)' **write_fd** is an integer, in this first case it has the value 3 — PyNEwbie, Aug 18 '15 at 14:33
My mistake. I haven't used Python 2 in a while. To clarify, write_fd should be an int. To open it in python 2 use [`os.fdopen`](https://docs.python.org/2/library/os.html#os.fdopen) or [`io.open`](https://docs.python.org/2/library/io.html#io.open). — Dunes, Aug 18 '15 at 14:39
I am right now working though **help(msvcrt.open_osfhandle)** Thanks, this comment crossed with your edit — PyNEwbie, Aug 18 '15 at 14:40
I got this to work, it was very helpful as I learned a lot, I marked up but gave the solution to the first solution as it is simpler — PyNEwbie, Aug 18 '15 at 15:43

When working with a named pipe is there a way to do something like readlines()

3 Answers3

Sending code

Receiving code