Python - read from multiple big files in paralell and yield them individually

Question

I have multiple big files and need to yield them round-robin style line by line. Something like this pseudo code:

    def get(self):
        with open(file_list, "r") as files:
            for file in files:
                yield file.readline()

How would I do this?

Found a possible duplicate: https://stackoverflow.com/questions/4617034/how-can-i-open-multiple-files-using-with-open-in-python?rq=1 — Harald Thomson, Feb 07 '20 at 10:04

juanpa.arrivillaga · Accepted Answer · 2020-02-07T10:31:37.273

The itertools documentation has several recipes, among them a very neat round-robin recipe. I would also use ExitStack to work with multiple file context-managers:

from itertools import cycle, islice
from contextlib import ExitStack

# https://docs.python.org/3.8/library/itertools.html#itertools-recipes
def roundrobin(*iterables):
    "roundrobin('ABC', 'D', 'EF') --> A D E B F C"
    # Recipe credited to George Sakkis
    num_active = len(iterables)
    nexts = cycle(iter(it).__next__ for it in iterables)
    while num_active:
        try:
            for next in nexts:
                yield next()
        except StopIteration:
            # Remove the iterator we just exhausted from the cycle.
            num_active -= 1
            nexts = cycle(islice(nexts, num_active))

...

def get(self):
    with open(files_list) as fl:
        filenames = [x.strip() for x in fl]
    with ExitStack() as stack:
        files = [stack.enter_context(open(fname)) for fname in filenames]
        yield from roundrobin(*files)

Although, perhaps the best design is to use inversion of control, and provide the sequence of file-objects as an argument to .get, so the calling code should take care of using an exit-stack:

class Foo:
    ...
    def get(self, files):
        yield from roundrobin(*files)

# calling code:
foo = Foo() # or however it is initialized

with open(files_list) as fl:
    filenames = [x.strip() for x in fl]
with ExitStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]
    for line in foo.get(files):
        do_something_with_line(line)

The `file_list` is meant to be a python list of file name strings. — Harald Thomson, Feb 07 '20 at 10:49
`open` doesn't accept a list as input though, does it? it takes `str`, `bytes`, `os.path`-like object. Am I missing something? — Z4-tier, Feb 07 '20 at 11:03
No, you are right. My code however is meant to be only pseudocode. I assumed the hypothetical existence of such an 'open' command. — Harald Thomson, Feb 07 '20 at 11:34

Z4-tier · Answer 2 · 2020-02-07T11:57:21.277

This would be tricky (or require some additional libraries) to do using a context manager, but it shouldn't be very difficult without. open() takes the name of a single file, so assuming file_list is a list of strings naming the input files, this should work:

def get(files_list):
  file_handles = [open(f, 'r') for f in files_list]
  while file_handles:
    for fd in file_handles:
      line = fd.readline()
      if line:
        yield line
      else:
        file_handles.remove(fd)

I'm assuming you want to keep going until every line is read from every file, with shorter files dropping off as they hit EOF.

Python - read from multiple big files in paralell and yield them individually

2 Answers2