Run function in parallel but inherite dynamic data from the previous function

Question

I'm trying to run multiple functions simultaneously:

-or so called functions because they belong to a class:

from sh import tail
data = {}
class my_Class():
    def __init__(self):
        """Nothing to declare for initializing"""
    def get_data(self, filepath):
        """I'm trying to import the data from several files"""
        for line in tail("-f", "-n 1", filepath, _iter=True):
            data[filepath] = line
            print(data)
my_Class().get_data("path/to/file") #call 1
my_Class().get_data("path/to/another/file") #call 2
# ... 14 similar calls

I want each call to append it's data to the dictionary. And so, when I call:

my_Class().get_data("path/to/file") #call 1
my_Class().get_data("path/to/another/file") #call 2
# ... 14 similar calls

The result should print:

#1 {'filepath1' : line}
#2 {'filepath1' : line, 'filepath2' : line}
#3 {'filepath1' : line, 'filepath2' : line, 'filepath3' : line}
# ... 13 more

At the same time I want the content of dictionary data{...} to keep changing dynamically; because of the data in the files is flexible. For example:

#1 {'filepath1' : line}
#2 {'filepath1' : last_line_in_the_file}, 'filepath2' : line}
#3 {'filepath1' : last_line_in_the_file, 'filepath2' : last_line_in_the_file, 'filepath3' : line}
# ... 13 more

I've already checked these posts: but it doesn't do what I ask; Python: How can I run python functions in parallel?, How to do parallel programming in Python Thank you! Please tell me if something sounds obscure

Answer 1

It sounds like you're asking for two things here:

1. How to run tasks in parallel, and
2. How to share (mutable) values between those tasks.

---

For the first one, the answer is, as you suspected, threads. For some programs, that isn't the answer, because they're spending most of their time doing heavy CPU computation in Python, or because you need thousands of tasks rather than a handful. But here, just run each one as a thread.

Instead of this:

my_Class().get_data("path/to/file") #call 1
my_Class().get_data("path/to/another/file") #call 2

… you create your threads:

t1 = threading.Thread(target=my_Class().get_data, args=

("path/to/file",)) t2 = threading.Thread(target=my_Class().get_data, args= ("path/to/another/file",))

… then start them:

t1.start()
t2.start()

… then wait for them all to finish (which, in this case, will obviously take forever, so you could simplify things here…):

t1.join()
t2.join()

---

Now, how do you share mutable data between threads?

To start with, you can just access and mutate the same values from the different threads. But in general, you'll want to put a lock around each value, unless you know that you don't need one.

If you only care about CPython on Windows, macOS, Linux, and BSD, inserting a string value with a string key into a dict object is one of those things that doesn't need a lock. And printing to stdout is another one. And those are the only things you're sharing. So, you actually don't need any locks here; things will just work.

But, since you probably didn't know that dicts were safe in this way, let's see how you'd use a lock.

data = {}
data_lock = threading.Lock()

# etc.

def get_data(self, filepath):
    """I'm trying to import the data from several files"""
    for line in tail("-f", "-n 1", filepath, _iter=True):
        with data_lock:
            data[filepath] = line
            print(data)

That's all there is to it.

Things can get a bit more complicated. For example, you don't really need to hold the lock for as long as you're doing here. If you had 30 threads, there'd be a good chance that one of them is trying to grab the lock to add a new value, while another thread had already made a string out of data, and was taking its time printing that string (printing to stdout is pretty slow), but hadn't released the lock yet. If so, you could get a bit more parallelism by breaking things down:

def get_data(self, filepath):
    """I'm trying to import the data from several files"""
    for line in tail("-f", "-n 1", filepath, _iter=True):
        with data_lock:
            data[filepath] = line
        with data_lock:
            datastr = str(data)
        print(datastr)

But that's really as complicated as it gets. The hard part about threading is when you have to compose separate locks because you have separate data that's getting passed around between threads and so on. For simple cases like this, it's actually pretty simple.