Stack Overflow
Stack Overflow

Questions tagged [pool]

A pool describes a cache of resources that an application can draw from to save instantiation time.

Pools typically refer to database connections but the term can be applied to graphics handles, files and other run-time processes. When the use of such a resource is complete but the application is still running, it will typically return the resource to the pool rather than release it.

1504 questions
256
votes
4 answers

How is the java memory pool divided?

I’m currently monitoring a Java application with jconsole. The memory tab lets you choose between: Heap Memory Usage Non-Heap Memory Usage Memory Pool “Eden Space” Memory Pool “Survivor Space” Memory Pool “Tenured Gen” Memory Pool “Code…
Dani Cricco
  • 8,829
  • 8
  • 31
  • 35
234
votes
14 answers

Can't pickle when using multiprocessing Pool.map()

I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine: import multiprocessing def f(x): return x*x def go(): pool = multiprocessing.Pool(processes=4) …
ventolin
  • 2,971
  • 3
  • 21
  • 25
165
votes
11 answers

Keyboard Interrupts with python's multiprocessing Pool

How can I handle KeyboardInterrupt events with python's multiprocessing Pools? Here is a simple example: from multiprocessing import Pool from time import sleep from sys import exit def slowly_square(i): sleep(1) return i*i def go(): …
Fragsworth
  • 33,919
  • 27
  • 84
  • 97
145
votes
10 answers

Python Process Pool non-daemonic?

Would it be possible to create a python Pool that is non-daemonic? I want a pool to be able to call a function that has another pool inside. I want this because deamon processes cannot create process. Specifically, it will cause the…
Max
  • 8,671
  • 4
  • 33
  • 46
91
votes
4 answers

How to troubleshoot an "AttributeError: __exit__" in multiproccesing in Python?

I tried to rewrite some csv-reading code to be able to run it on multiple cores in Python 3.2.2. I tried to use the Pool object of multiprocessing, which I adapted from working examples (and already worked for me for another part of my project). I…
László
  • 3,914
  • 8
  • 34
  • 49
87
votes
3 answers

Passing multiple parameters to pool.map() function in Python

I need some way to use a function within pool.map() that accepts more than one parameter. As per my understanding, the target function of pool.map() can only have one iterable as a parameter but is there a way that I can pass other parameters in as…
DJMcCarthy12
  • 3,819
  • 8
  • 28
  • 34
80
votes
5 answers

How to let Pool.map take a lambda function

I have the following function: def copy_file(source_file, target_dir): pass Now I would like to use multiprocessing to execute this function at once: p = Pool(12) p.map(lambda x: copy_file(x,target_dir), file_list) The problem is, lambda's…
Peter Smit
  • 27,696
  • 33
  • 111
  • 170
69
votes
4 answers

Combine Pool.map with shared memory Array in Python multiprocessing

I have a very large (read only) array of data that I want to be processed by multiple processes in parallel. I like the Pool.map function and would like to use it to calculate functions on that data in parallel. I saw that one can use the Value or…
Jeroen Dirks
  • 7,705
  • 12
  • 50
  • 70
60
votes
2 answers

urllib3 connectionpool - Connection pool is full, discarding connection

Does seeing the urllib3.connectionpool WARNING - Connection pool is full, discarding connection mean that I am effectively loosing data (because of lost connection) OR Does it mean that connection is dropped (because pool is full); however, the…
JavaFan
  • 1,295
  • 3
  • 19
  • 28
58
votes
4 answers

multiprocessing.Pool() slower than just using ordinary functions

(This question is about how to make multiprocessing.Pool() run code faster. I finally solved it, and the final solution can be found at the bottom of the post.) Original Question: I'm trying to use Python to compare a word with many other words in a…
Karim Bahgat
  • 2,781
  • 3
  • 21
  • 27
51
votes
11 answers

what is java.io.EOFException, Message: Can not read response from server. Expected to read 4 bytes, read 0 bytes

This question has been asked a couple of times in SO and many times in other sites. But I didn't get any satisfiable answer. My problem: I have a java web application which uses simple JDBC to connect to mysql database through Glassfish…
mukund
  • 2,866
  • 5
  • 31
  • 41
50
votes
6 answers

Memory usage keep growing with Python's multiprocessing.pool

Here's the program: #!/usr/bin/python import multiprocessing def dummy_func(r): pass def worker(): pass if __name__ == '__main__': pool = multiprocessing.Pool(processes=16) for index in range(0,100000): …
C.B.
  • 766
  • 1
  • 6
  • 7
43
votes
2 answers

How do you pass a Queue reference to a function managed by pool.map_async()?

I want a long-running process to return its progress over a Queue (or something similar) which I will feed to a progress bar dialog. I also need the result when the process is completed. A test example here fails with a RuntimeError: Queue objects…
David
  • 1,423
  • 1
  • 12
  • 9
40
votes
4 answers

java.lang.IllegalMonitorStateException: (m=null) Failed to get monitor for

Why may this happen? The thing is that monitor object is not null for sure, but still we get this exception quite often: java.lang.IllegalMonitorStateException: (m=null) Failed to get monitor for (tIdx=60) at…
Andrey Adamovich
  • 20,285
  • 14
  • 94
  • 132
35
votes
2 answers

Can I use a multiprocessing Queue in a function called by Pool.imap?

I'm using python 2.7, and trying to run some CPU heavy tasks in their own processes. I would like to be able to send messages back to the parent process to keep it informed of the current status of the process. The multiprocessing Queue seems…
Olson
  • 1,884
  • 2
  • 17
  • 18
1
2 3
…
99 100