Concerning running parallel code in separate processes (unlike multithreading) and/or related to package `multiprocess` from PyPI.
Questions tagged [multiprocess]
462 questions
47
votes
14 answers
how to write a process-pool bash shell
I have more than 10 tasks to execute, and the system restrict that there at most 4 tasks can run at the same time.
My task can be started like:
myprog taskname
How can I write a bash shell script to run these task. The most important thing is that…
Sili
- 927
- 1
- 12
- 17
42
votes
6 answers
How can Google Chrome isolate tabs into separate processes while looking like a single application?
We have been told that Google Chrome runs each tab in a separate process. Therefore a crash in one tab would not cause problems in the other tabs.
AFAIK, multi-processes are mostly used in programs without a GUI. I have never read any technique…
faceclean
- 3,781
- 8
- 39
- 58
39
votes
4 answers
Mutex lock threads
Am new to multi threaded/processs programming.
So here's what I need to clarify.
Process A code
pthread_mutex_lock()
pthread_create(fooAPI(sharedResource)) //fooAPI creates another thread with shared resource that shares across…
resting
- 16,287
- 16
- 59
- 90
32
votes
8 answers
multiprocess or threading in python?
I have a python application that grabs a collection of data and for each piece of data in that collection it performs a task. The task takes some time to complete as there is a delay involved. Because of this delay, I don't want each piece of data…
Ryan
- 11,743
- 16
- 37
- 37
27
votes
3 answers
How to use printf() in multiple threads
I am implementing a multithreaded program that uses different cores, and many threads are executed simultaneously. Each thread makes a printf() call, and the result is not readable.
How can I make printf() atomic, so that a printf() call in one…
user3242743
- 1,751
- 5
- 22
- 32
26
votes
4 answers
Python: concurrent.futures How to make it cancelable?
Python concurrent.futures and ProcessPoolExecutor provide a neat interface to schedule and monitor tasks. Futures even provide a .cancel() method:
cancel(): Attempt to cancel the call. If the call is currently being executed and cannot be…
Ketzu
- 660
- 1
- 7
- 12
16
votes
5 answers
How can I make my Perl script use multiple cores for child processes?
I'm working on a mathematical model that uses data generated from XFOIL, a popular aerospace tool used to find the lift and drag coefficients on airfoils.
I have a Perl script that calls XFOIL repeatedly with different input parameters to generate…
Dang Khoa
- 5,693
- 8
- 51
- 80
15
votes
1 answer
Fill up a dictionary in parallel with multiprocessing
Yesterday i asked a question: Reading data in parallel with multiprocess
I got very good answers, and i implemented the solution mentioned in the answer i marked as correct.
def read_energies(motif):
os.chdir("blabla/working_directory")
…
Gábor Erdős
- 3,599
- 4
- 24
- 56
15
votes
2 answers
How to register "atexit" function in python's multiprocessing subprocess?
I have some subprocesses (using multiprocessing) and when they stop, each of them need do some final work. Something like the following, which did not work though...
import multiprocessing
import atexit
def final():
print "final work"
def…
rongdong.bai
- 471
- 1
- 6
- 16
14
votes
4 answers
Python: How can I check the number of pending tasks in a multiprocessing.Pool?
I have a small pool of workers (4) and a very large list of tasks (5000~). I'm using a pool and sending the tasks with map_async(). Because the task I'm running is fairly long, I'm forcing a chunksize of 1 so that one long process can't hold up…
jkeating
- 1,154
- 1
- 9
- 12
14
votes
3 answers
Parallelize these nested for loops in python
I have a multidimensional array (result) that should be filled by some nested loops. Function fun() is a complex and time-consuming function. I want to fill my array elements in a parallel manner, so I can use all my system's processing…
Amir Masoud
- 341
- 1
- 6
- 17
13
votes
2 answers
Python multicore programming
Please consider a class as follow:
class Foo:
def __init__(self, data):
self.data = data
def do_task(self):
#do something with data
In my application I've a list containing several instances of Foo class. The aim is to…
Larry
- 131
- 1
- 1
- 4
12
votes
3 answers
Why multiprocess python grpc server do not work?
I instantiate one grpc server per subprocess by multiprocess pool. When I use multiple clients to access the server, I found the following two problems：
all clients access same server subprocess
client raise MaybeEncodingError
By the way, my…
Long.zhao
- 1,085
- 2
- 11
- 16
12
votes
2 answers
How much memory should 'managed_shared_memory' allocate? (boost)
I'm looking for a definitive answer (if indeed one exists) on how much memory should be allocated when creating a static chunks of shared memory via boost::interprocess's managed_shared_memory. Even official examples seem to allocate arbitrarily…
Courtney Christensen
- 9,165
- 5
- 47
- 56
11
votes
1 answer
progress bar slows down code by factor of 5 using tqdm and multiprocess
I added a progress bar to my 2.7 python code using tqdm but it has slowed down my code significantly. Without the progress bar for one example it takes 12 seconds while with the progress bar it takes 57 seconds.
The code without the progress bar…
0_o
- 333
- 3
- 9