7

Is there a way to control the thread pools which handle the functions which get sent to agents? As I understand things, if I send-off, underneath the hood I'm using an unbounded thread pool. I would like to, say, run some functions on one thread pool and other functions on another. The reason for this is say I have a some functions which do IO and which are also less important. I'd throw these on some bounded thread pool and wouldn't worry if there was excessive blocking and they stacked up since they're, well, less important. The main thing is that I wouldn't want their crappy IO blocking to say have an effect on some more important functions which are running on another thread pool.

I'm basing the question off of something similar I did with thread pools in Akka and I'm just wondering I can accomplish the same thing with Clojure.

pondermatic
  • 6,453
  • 10
  • 48
  • 63
  • I would suggest using async IO to solve such problem rather than managing a thread pool for IO operations – Ankur Jul 04 '12 at 04:12

3 Answers3

14

For Clojure versions up to 1.4:

You cannot replace the built-in agent send and send-off thread pools. They are hard-coded in Agent.java.

The send pool (used for computation) is fixed size = 2 + Runtime.getRuntime().availableProcessors().

The send-off pool (also used for futures) is a cached thread pool and will grow without bound. This allows an arbitrary number of background tasks to wait for I/O. The cached threads will be reused and discarded if they've been idle for one minute.

If you want to manage work on your own thread pool, you'll need to dip into java.util.concurrent or use a Clojure wrapper lib.

For Clojure 1.5 (upcoming):

You can supply your own ExecutorService using (send-via executor a f), the default threadpools are not hard-wired anymore. See Agent.java in Clojure 1.5+ for details.

laczoka
  • 407
  • 5
  • 15
Alex Miller
  • 69,183
  • 25
  • 122
  • 167
  • Any way to do the same defining of custom ThreadPools for Futures in 1.5+ ? – Didier A. Oct 23 '15 at 22:07
  • @DidierA. based on the information in the answer (at the time of this reading) it looks like `future` uses the same Executor as `send-off` so I’d think we could set the executor used by `future` via `set-agent-send-off-executor!` – Avi Flax Apr 28 '20 at 18:50
2

The Clojure library Claypoole is designed for exactly this. It lets you define threadpools and use (and reuse) them for futures, pmaps, and so on.

Leon Barrett
  • 121
  • 1
  • 3
  • 1
    Correct me if I'm wrong, but I don't think Claypool gives you control over agent thread pools, only thread pools used for _it's_ custom `future`, `pmap` and so on implementations. – metasoarous Aug 26 '14 at 18:47
1

Amit Rathore (of Runa inc), has published a library (called medusa) for managing thread pools. It looks like a fairly close match for what you are looking for.

http://s-expressions.com/2010/06/08/medusa-0-1-a-supervised-thread-pool-for-clojure-futures-2/

Arthur Ulfeldt
  • 90,827
  • 27
  • 201
  • 284
  • is that integrated with agents, or is it separate? in other words - can you use it to replace the built-in thread pools? or is there some way to create agents with a different thread pool? i can't see anything that explains this... when i was worried about this i asked this q - http://stackoverflow.com/questions/10969708/parallel-doseq-for-clojure - and i don't think any answer gave a seamless way to replace the system functionality. – andrew cooke Jul 04 '12 at 01:46
  • medusa is an alternate library for running asynchronous tasks, it is an alternative to the built in agents. – Arthur Ulfeldt Jul 04 '12 at 05:24