Can't mutate a variable local to a function in another function

Question

I am trying to manipulate the lists inside the dictionary clean_txt in another function, but its not working and I end up with empty lists inside the dict.

My understading is that both lists and dicts are mutable objects so what is the problem here?

def process_questions(i, question_list, questions, question_list_name):
    ''' Transform questions and display progress '''
    print('processing {}: process {}'.format(question_list_name, i))
    for question in questions:
        question_list.append(text_to_wordlist(str(question)))

@timeit
def multi(n_cores, tq, qln):
    procs = []
    clean_txt = {}
    for i in range(n_cores):
        clean_txt[i] = []

    for index in range(n_cores):
        tq_indexed = tq[index*len(tq)//n_cores:(index+1)*len(tq)//n_cores]
        proc = Process(target=process_questions, args=(index, clean_txt[index], tq_indexed, qln, ))
        procs.append(proc)
        proc.start()

    for proc in procs:
        proc.join()

    print('{} records processed from {}'.format(sum([len(x) for x in clean_txt.values()]), qln))
    print('-'*100)

score 1 · Answer 1 · edited May 23 '17 at 11:46

Your are using Processes not threads.

When the process is created the memory of your program is copied and each process work on its own set, therefore it is NOT shared.

Here's a question that can help you understand: Multiprocessing vs Threading Python

If you want to share memory between processes you should look into semaphores or use Threads instead. There are also other solutions to share data, like queues or database etc.

Shreyash S Sarnayak · Answer 2 · 2017-05-01T11:44:11.930

1

You are appending to clean_txt[index] from a different Process. clean_txt[index] belongs to the main python process who created it. Since a process can't access or modify another process's memory, you can't append to it. (Not really. See edit below)

You will need to create shared memory.

You can use Manager to create shared memory, something like this

from multiprocessing import Manager
manager = Manager()
...
    clean_txt[i] = manager.list()

Now you can append to this list in the other process.

Edit -

My explanation about clean_txt wasn't clear. Thanks to @Maresh.

When a new Process is created the whole memory is copied. So modifying the list in the new process won't affect the copy in the main process. So you need a shared memory.

edited May 01 '17 at 11:44

answered May 01 '17 at 11:19

Shreyash S Sarnayak

2,309
19
23

It's a bit misleading to say that it belongs to the parent process :) – Maresh May 01 '17 at 11:21
@Maresh He is doing `clean_txt[i] = []` in the main process, so that list will belong to the main process right? – Shreyash S Sarnayak May 01 '17 at 11:28
Well each process will have its own memory space, copied from the parent process' one, it's important to understand that when playing with Multiprocessing. In the end you are right, this exact memory address belongs to the parent, but it should be clearer in the explanation IMO. – Maresh May 01 '17 at 11:31
@Maresh Ok. Now I get it. Thanks for the clarification. :) – Shreyash S Sarnayak May 01 '17 at 11:34
@shreyash-s-sarnayak it worked but doubled my runtime – user3804483 May 01 '17 at 11:37
@user3804483 For a small size of input `Multiprocessing` is not really efficient. What is the size of input you tried (tq, qln, ..)? – Shreyash S Sarnayak May 01 '17 at 11:42
2.5 million records. – user3804483 May 01 '17 at 11:43
Looks like shared memory is creating a lot of overhead. – Shreyash S Sarnayak May 01 '17 at 11:48
@user3804483 If you think this answer helped please click the tick mark and accept this answer. – Shreyash S Sarnayak May 03 '17 at 19:10

Can't mutate a variable local to a function in another function

2 Answers2