Stacking filters around a file object

Question

I would like to stack filters around an open() function. These filters are supposed, for example, to change every encountered a characters into b in the stream read from the file.

For example, here is a code sample:

def filter (stream):
    for line in stream:
        yield line.replace('a', 'b')

def add_filter(filter, file):
    return io.TextIOWrapper(filter(file))

def processing_file(f):
    import sys
    for line in f:
        sys.stdout.write("aa: " + line)

f = open('./example.txt', 'r')
f = add_filter(filter, f)
processing_file(f)

I guess that the filter_a() function should return a TextIOWrapper to mimic the result of an open() function. But, I keep having the following error message:

AttributeError: 'generator' object has no attribute 'readable'

In fact, I understand the error, but I do not know how to work around and make it work properly.

https://docs.python.org/2/library/io.html#io.TextIOWrapper shows that you need to pass buffer but you are passing a generator. — noorul, Jan 31 '16 at 09:26
Yes, but how do I turn a generator in a buffer ? I guess that this is my question once you removed anything else. — perror, Jan 31 '16 at 09:31

score 3 · Answer 1 · answered Jan 30 '16 at 23:18

3

You can iterate directly over the filter generator:

with open('./example.txt', 'r') as f:
    for line in filter(f):
        sys.stdout.write("aa: " + line)

answered Jan 30 '16 at 23:18

mhlester

22,781
10
52
75

1

In fact, this does not really fit with my need. I would like to be able to add the filter **inside** the `TextIOWrapper` and use it normally as the filter did not exist. I will modify my question to make it a bit more clearer. – perror Jan 31 '16 at 09:07

score 3 · Accepted Answer · edited May 23 '17 at 12:07

I came with a solution to my own question... I, first, have to admit that my question was not totally well formed and may have lacked of precision. So, I do not blame anybody to have discarded it.

My original intention was to come out with a stackable framework of filters over a stream (open()). Trying to make it easy to use, also.

I mainly found inspiration in this answer on StackOverflow which was solving about 90% of my problem.

So, imagine we have two filters (which are coded as generators):

def tab_filter(stream):
    for line in stream:
        yield line.replace ('\t', ' ' * 8)

def a_filter(stream):
    for line in stream:
        yield line.replace ('a', 'z')

Then, we have this class allowing to wrap a generator inside a stream:

class IterStream(object):
    "File-like streaming iterator."

    def __init__(self, generator):
        self.generator = generator
        self.iterator = iter(generator)
        self.leftover = ''

    def __len__(self):
        return self.generator.__len__()

    def __iter__(self):
        return self.iterator

    def next(self):
        return self.iterator.next()

    def read(self, size):
        data = self.leftover
        count = len(self.leftover)
        try:
            while count < size:
                chunk = self.next()
                data += chunk
                count += len(chunk)
        except StopIteration:
            self.leftover = ''
            return data

        if count > size:
            self.leftover = data[size:]

        return data[:size]

Using it in the code will be as follow:

import sys
f = IterStream(a_filter(IterStream(tab_filter(open('Example.txt', 'r')))))
for line in f:
    sys.stdout.write("aa: " + line)

But, this is not yet totally satisfactory because we need a lot of useless function stacking. So, I decided to wrap it inside a decorator:

def streamfilter(filter):
    def stream(iostream):
        return IterStream(filter(iostream))
    return stream

@streamfilter
def tab_filter(stream):
    for line in stream:
        yield line.replace ('\t', ' ' * 8)

@streamfilter
def a_filter(stream):
    for line in stream:
        yield line.replace ('a', 'z')

Then, using the code is much easier now:

import sys
f = a_filter(tab_filter(open('Example.txt', 'r')))
for line in f:
    sys.stdout.write("aa: " + line)

I hope that some of you will find this few lines useful.

Stacking filters around a file object

2 Answers2