Clojure - Defining Number of Thread Used in Pmap

Question

I'm trying to use concurrency for my maps using pmap in Clojure, and I need to do some analysis based on the efficiency of the program under different thread counts.

Is the number of threads defined in Clojure within the pmap function, or somewhere in the project file? Looking at the pmap documentation there are no additional parameters compared to the map function.

For example, I need to run the program under 2, 32, 64 etc... threads.

Answer 1

Your question seems be to closely relative enough to: How many threads does Clojure's pmap function spawn for URL-fetching operations?

From the answer of Alex Miller, you can deduce that the number of threads used by pmap is <your number of core> + 2. I don't why there is a + 2 but even with the current release of Clojure, 1.10.0, the source code of the pmap function is still the same.

As I have 4 cores on my machine, pmap should use 6 threads.

-- EDIT

To really answer to your question, you can define a custom pmap function, custom-pmap, which allow you to specify the number of thread you would like to use:

(defn custom-pmap
  ([f coll nb-thread]
   (let [n nb-thread
         rets (map #(future (f %)) coll)
         step (fn step [[x & xs :as vs] fs]
                (lazy-seq
                 (if-let [s (seq fs)]
                   (cons (deref x) (step xs (rest s)))
                   (map deref vs))))]
     (step rets (drop n rets)))))

(custom-pmap inc (range 1000) 8)
;; => (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ....999 1000)

Answer 2

You can use claypoole's pmap that takes a certain sized threadpool as a first argument.

# project.clj
[com.climate/claypoole "1.1.4"]

# or deps.edn
com.climate/claypoole {:mvn/version "1.1.4"}

Now let's specify some pool sizes and map an operation that takes one second over a collection of size 64.

(ns demo
  (:refer-clojure :exclude-clojure [pmap])
  (:require [com.climate.claypoole :refer [threadpool pmap]]))

(def pool-sizes 
  [2 32 64]) 

(doseq [pool-size pool-sizes]
  (time (doall (pmap (threadpool pool-size) (fn [n] (Thread/sleep 1000)) (range 64)))))

"Elapsed time: 32113.704013 msecs"
"Elapsed time: 2013.242638 msecs"
"Elapsed time: 1011.616369 msecs"

So some overhead and 32 seconds for a threadpool with size 2, 2 seconds for size 32 en 1 second for size 64.