1

I'm trying to multi-core read data from 2 files, but after executing this code list1 and list2 is empty.

from multiprocessing import Process

def getfile(fn, out):
    print("start reading file {}".format(fn))
    with open(fn) as file:
        for line in file:
            out.append(line)

if __name__ == '__main__':
    file1 = []
    file2 = []
    p1 = Process(target=getfile, args=("100.txt", file1))
    p2 = Process(target=getfile, args=("98.txt", file2))
    p1.start()
    p2.start()

    p1.join()
    p2.join()
    print(file1)
    print(file2)

How to get write data from files to list or something iterable using multiprocessing?

Entro
  • 76
  • 1
  • 5
  • look at [this](https://stackoverflow.com/a/29050564/6111440) – Adrián Kálazi Jul 29 '17 at 23:07
  • If you're trying to do this for performance purposes, you need to be aware that if both files reside on the same physical spinning disk(s) reading both of them at the same time can't make the overall read time any faster and in fact might make it *slower* as the disk heads might wind up doing extra seeks. – Andrew Henle Jul 29 '17 at 23:43

1 Answers1

3

When using multiple processes, use Queues or Pipes to exchange data between your parent process and the processes you spawn off.

Pipes intuitively allow you to pass data between the parent process and a child process.

Queues allow your child process to store some data on the queue allowing your parent process to retrieve it.

In your case, a queue makes the most sense since it seems your parent process does not need to pass any data to the child process after its spawned.

Here's an example for one file:

from multiprocessing import Process, Queue

def getfile(fn, out):
    with open(fn) as file:
        for line in file:
            out.put(line)

if __name__ == '__main__':
    file1 = Queue()
    p1 = Process(target=getfile, args=("100.txt", file1))
    p1.start()
    file1.get() //returns lines of the file
    p1.join()

You can pass the same queue to two processes, just note the order of messages may be mixed between the two files in case you're trying to separate the contents of each file.

Josh Hamet
  • 957
  • 8
  • 10