Stack Overflow
Stack Overflow

Questions tagged [python-multiprocessing]

multiprocessing is a package that supports spawning processes using an API similar to the threading module in python programming language. When asking a question about this topic, please mention the operating system you are running it on, or add it to the tags.

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using Operating System processes instead of threads.

The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. It runs on both Unix and Windows.

4758 questions
866
votes
24 answers

How to use multiprocessing pool.map with multiple arguments

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments? import multiprocessing text = "test" def harvester(text, case): X = case[0] text + str(X) if __name__ == '__main__': pool =…
user642897
  • 9,091
  • 3
  • 21
  • 22
368
votes
10 answers

Python multiprocessing PicklingError: Can't pickle

I am sorry that I can't reproduce the error with a simpler example, and my code is too complicated to post. If I run the program in IPython shell instead of the regular Python, things work out well. I looked up some previous notes on this problem.…
CuriousMind
  • 15,168
  • 20
  • 82
  • 120
345
votes
13 answers

How to get the return value of a function passed to multiprocessing.Process?

In the example code below, I'd like to get the return value of the function worker. How can I go about doing this? Where is this value stored? Example Code: import multiprocessing def worker(procnum): '''worker function''' print…
Louis Thibault
  • 20,240
  • 25
  • 83
  • 152
265
votes
2 answers

multiprocessing.Pool: What's the difference between map_async and imap?

I'm trying to learn how to use Python's multiprocessing package, but I don't understand the difference between map_async and imap. I noticed that both map_async and imap are executed asynchronously. So when should I use one over the other? And how…
spacegoing
  • 5,056
  • 5
  • 25
  • 42
135
votes
2 answers

When should we call multiprocessing.Pool.join?

I am using 'multiprocess.Pool.imap_unordered' as following from multiprocessing import Pool pool = Pool() for mapped_result in pool.imap_unordered(mapping_func, args_iter): do some additional processing on mapped_result Do I need to call…
hch
  • 6,030
  • 8
  • 20
  • 24
133
votes
3 answers

multiprocessing: Understanding logic behind `chunksize`

What factors determine an optimal chunksize argument to methods like multiprocessing.Pool.map()? The .map() method seems to use an arbitrary heuristic for its default chunksize (explained below); what motivates that choice and is there a more…
Brad Solomon
  • 38,521
  • 31
  • 149
  • 235
122
votes
2 answers

Sharing a result queue among several processes

The documentation for the multiprocessing module shows how to pass a queue to a process started with multiprocessing.Process. But how can I share a queue with asynchronous worker processes started with apply_async? I don't need dynamic joining or…
alexis
  • 48,685
  • 16
  • 101
  • 161
104
votes
2 answers

What's the difference between ThreadPool vs Pool in the multiprocessing module?

Whats the difference between ThreadPool and Pool in multiprocessing module. When I try my code out, this is the main difference I see: from multiprocessing import Pool import os, time print("hi outside of main()") def hello(x): print("inside…
ozn
  • 1,990
  • 3
  • 26
  • 37
102
votes
4 answers

Is it possible to run function in a subprocess without threading or writing a separate file/script.

import subprocess def my_function(x): return x + 100 output = subprocess.Popen(my_function, 1) #I would like to pass the function object and its arguments print output #desired output: 101 I have only found documentation on opening…
wroscoe
  • 1,914
  • 3
  • 19
  • 18
101
votes
2 answers

Python multiprocessing.Pool: AttributeError

I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores. I wrote the following code, which works if I use normal map(), but with pool.map() returns an error. import…
Amit
  • 5,924
  • 7
  • 46
  • 94
80
votes
1 answer

multiprocessing: map vs map_async

What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes? So is it wrong to presume both are running asynchronous and parallel? def f(x): return…
aman
  • 1,875
  • 4
  • 18
  • 27
69
votes
2 answers

Python multiprocessing.Queue vs multiprocessing.manager().Queue()

I have a simple task like that: def worker(queue): while True: try: _ = queue.get_nowait() except Queue.Empty: break if __name__ == '__main__': manager = multiprocessing.Manager() # queue =…
novicef
  • 803
  • 1
  • 8
  • 9
59
votes
4 answers

What's the point of multithreading in Python if the GIL exists?

From what I understand, the GIL makes it impossible to have threads that harness a core each individually. This is a basic question, but, what is then the point of the threading library? It seems useless if the threaded code has equivalent speed to…
coolster1278
  • 701
  • 1
  • 5
  • 8
54
votes
1 answer

Shared variable in python's multiprocessing

First question is what is the difference between Value and Manager().Value? Second, is it possible to share integer variable without using Value? Below is my sample code. What I want is getting a dict with a value of integer, not Value. What I did…
user2435611
  • 1,093
  • 1
  • 12
  • 18
51
votes
2 answers

multiprocessing fork() vs spawn()

I was reading the description of the two from the python doc: spawn The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process objects run() method. In particular,…
Crystina
  • 990
  • 1
  • 5
  • 16
1
2 3
…
99 100