2

Set Up

I have a complex set of data that I need to compare to various sets of constraints concurrently, but I'm running into multiple issues. The first issue is getting results out of my multiple processes, and the second issue is making anything beyond an extremely simple function to run concurrently

Part two of this is a bit more complicated
Link to part Two: Click Here for Part Two

Example:

I have a simple function that completes an action but I can't access the results outside of the processes that are spawned in.

Functions

Because I'm using a Jupyter notebook on windows I have the functions saved in a separate file (multi_process_functions.py) and I'm importing that file as mpf

def f2(name):
    return('hello ' + name)

def f3(name):
    return('Bye ' + name)

Jupyter Notebook

Then in my jupyter notebook I have the following code:

data_manager = mp.Manager()
ns = data_manager.Namespace()
ns.names = ['Bob', 'Joe', 'Brian']
ns.results = []

if __name__ == '__main__':
    with mp.Pool() as p:
        res = p.map(mpf.f2, ns.names)
        ns.results.append(res)
        print(res)
    with mp.Pool() as x:
        res = x.map(mpf.f3, ns.names)
        ns.results.append(res)
        print(res)
    
    print(ns.results)

The resulting output:

['hello Bob', 'hello Joe', 'hello Brian']
['Bye Bob', 'Bye Joe', 'Bye Brian']
[]

I was hoping to have the output look like this:

['hello Bob', 'hello Joe', 'hello Brian']
['Bye Bob', 'Bye Joe', 'Bye Brian']
[['hello Bob', 'hello Joe', 'hello Brian'], ['Bye Bob', 'Bye Joe', 'Bye Brian'] ]

Question:

How can I aggregate and access the results once all of my processes have finished?

Branden-Pincince
  • 53
  • 6
  • Removing the manager and namespace and just using standard lists for name and results should work . You don't seem to be actually sharing data between your processes? – Iain Shelvington Apr 14 '21 at 01:18
  • @IainShelvington I don't need to share the data between them as they're processing, but I need to output their results to a shared memory of some kind. I'll be posting part two in about 45 minutes which might better illustrate my issue regarding sharing memory. (I would post it now, but StackOverflow is requiring me to wait 90 mins between posts) – Branden-Pincince Apr 14 '21 at 01:50
  • @IainShelvington I've updated my question with a link to part two which may help show what I'm trying to do regarding sharing data. – Branden-Pincince Apr 14 '21 at 02:53
  • Your code is actually correct, if you replace `ns.results` by a simple un-managed list. So I suspect it has something to do with the Namespace-Manager. Do you require this? – RaJa Apr 14 '21 at 13:58

0 Answers0