Questions tagged [multiprocessing]

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. The term also refers to the ability of a system to support more than one processor and/or the ability to allocate tasks between them.

There are many variations on this basic theme, and the definition of multiprocessing can vary with context, mostly as a function of how CPUs are defined (multiple cores on one die, multiple dies in one package, multiple packages in one system unit, etc.).

Multiprocessing sometimes refers to the execution of multiple concurrent software processes in a system as opposed to a single process at any one instant. However, the terms multitasking or multiprogramming are more appropriate to describe this concept which is implemented mostly in software, whereas multiprocessing is more appropriate to describe the use of multiple hardware CPUs.

http://en.wikipedia.org/wiki/Multiprocessing

Multiprocessing may also refer to the Python multiprocessing module, which is a package that supports spawning processes using an API similar to Python's threading module. It offers both local and remote concurrency, effectively side-stepping the limitations of the Global Interpreter Lock by using sub-processes instead of threads.

14352 questions
1019
votes
12 answers

Multiprocessing vs Threading Python

I am trying to understand the advantages of multiprocessing over threading. I know that multiprocessing gets around the Global Interpreter Lock, but what other advantages are there, and can threading not do the same thing?
John
  • 10,989
  • 4
  • 19
  • 7
866
votes
24 answers

How to use multiprocessing pool.map with multiple arguments

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments? import multiprocessing text = "test" def harvester(text, case): X = case[0] text + str(X) if __name__ == '__main__': pool =…
user642897
  • 9,091
  • 3
  • 21
  • 22
396
votes
3 answers

multiprocessing.Pool: When to use apply, apply_async or map?

I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map; what are the advantages of others?
Phyo Arkar Lwin
  • 6,673
  • 12
  • 41
  • 55
368
votes
10 answers

Python multiprocessing PicklingError: Can't pickle

I am sorry that I can't reproduce the error with a simpler example, and my code is too complicated to post. If I run the program in IPython shell instead of the regular Python, things work out well. I looked up some previous notes on this problem.…
CuriousMind
  • 15,168
  • 20
  • 82
  • 120
345
votes
13 answers

How to get the return value of a function passed to multiprocessing.Process?

In the example code below, I'd like to get the return value of the function worker. How can I go about doing this? Where is this value stored? Example Code: import multiprocessing def worker(procnum): '''worker function''' print…
316
votes
23 answers

How should I log while using multiprocessing in Python?

Right now I have a central module in a framework that spawns multiple processes using the Python 2.6 multiprocessing module. Because it uses multiprocessing, there is module-level multiprocessing-aware log, LOG = multiprocessing.get_logger(). Per…
cdleary
  • 69,512
  • 53
  • 163
  • 191
311
votes
10 answers

multiprocessing vs multithreading vs asyncio

I found that in Python 3.4, there are few different libraries for multiprocessing/threading: multiprocessing vs threading vs asyncio. But I don't know which one to use or is the "recommended one". Do they do the same thing, or are different? If so,…
289
votes
10 answers

RuntimeError on windows trying python multiprocessing

I am trying my very first formal python program using Threading and Multiprocessing on a windows machine. I am unable to launch the processes though, with python giving the following message. The thing is, I am not launching my threads in the main…
NG Algo
  • 3,570
  • 2
  • 18
  • 27
265
votes
2 answers

multiprocessing.Pool: What's the difference between map_async and imap?

I'm trying to learn how to use Python's multiprocessing package, but I don't understand the difference between map_async and imap. I noticed that both map_async and imap are executed asynchronously. So when should I use one over the other? And how…
spacegoing
  • 5,056
  • 5
  • 25
  • 42
259
votes
6 answers

Concurrent.futures vs Multiprocessing in Python 3

Python 3.2 introduced Concurrent Futures, which appear to be some advanced combination of the older threading and multiprocessing modules. What are the advantages and disadvantages of using this for CPU bound tasks over the older multiprocessing…
GIS-Jonathan
  • 4,347
  • 11
  • 31
  • 45
234
votes
14 answers

Can't pickle when using multiprocessing Pool.map()

I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine: import multiprocessing def f(x): return x*x def go(): pool = multiprocessing.Pool(processes=4) …
ventolin
  • 2,971
  • 3
  • 21
  • 25
218
votes
10 answers

Multiprocessing : use tqdm to display a progress bar

To make my code more "pythonic" and faster, I use multiprocessing and a map function to send it a) the function and b) the range of iterations. The implanted solution (i.e., calling tqdm directly on the range tqdm.tqdm(range(0, 30))) does not work…
SciPy
  • 5,412
  • 4
  • 18
  • 18
208
votes
20 answers

Multiprocessing: How to use Pool.map on a function defined in a class?

When I run something like: from multiprocessing import Pool p = Pool(5) def f(x): return x*x p.map(f, [1,2,3]) it works fine. However, putting this as a function of a class: class calculate(object): def run(self): def f(x): …
Mermoz
  • 14,898
  • 17
  • 60
  • 85
208
votes
6 answers

What are the differences between the threading and multiprocessing modules?

I am learning how to use the threading and the multiprocessing modules in Python to run certain operations in parallel and speed up my code. I am finding this hard (maybe because I don't have any theoretical background about it) to understand what…
200
votes
4 answers

Multiprocessing - Pipe vs Queue

What are the fundamental differences between queues and pipes in Python's multiprocessing package? In what scenarios should one choose one over the other? When is it advantageous to use Pipe()? When is it advantageous to use Queue()?
Jonathan Livni
  • 101,334
  • 104
  • 266
  • 359
1
2 3
99 100