Microbenchmark Clojure functions

Question

Question

How fast are small Clojure functions like assoc? I suspect that assoc operates in the 100ns to 3us range, which makes it difficult to time.

Using `time`

user=> (def d {1 1, 2 2})
#'user/d
user=> (time (assoc d 3 3))
"Elapsed time: 0.04989 msecs"
{1 1, 2 2, 3 3}

There is clearly a lot of overhead there so I don't trust this benchmark. Friends pointed me to Criterium which handles a lot of the pain of benchmarking (multiple evaluations, warming up the JVM, GC see How to benchmark functions in Clojure?).

Using `Criterium`

Sadly, on such a small benchmark even Criterium seems to fail

user=> (use 'criterium.core)
nil
user=> (def d {1 1 2 2})
#'user/d
user=> (bench (assoc d 3 3))
WARNING: JVM argument TieredStopAtLevel=1 is active, and may lead to unexpected results as JIT C2 compiler may not be active. See http://www.slideshare.net/CharlesNutter/javaone-2012-jvm-jit-for-dummies.
WARNING: Final GC required 1.694448681330372 % of runtime
Evaluation count : 218293620 in 60 samples of 3638227 calls.
             Execution time mean : -15.677491 ns
    Execution time std-deviation : 6.093770 ns
   Execution time lower quantile : -20.504699 ns ( 2.5%)
   Execution time upper quantile : 1.430632 ns (97.5%)
                   Overhead used : 123.496848 ns

Just in case you missed it, this operation takes -15ns on average. I know that Clojure is pretty magical, but negative runtimes seem a bit too good to be true.

Repeat the Question

So really, how long does an assoc take? How can I benchmark micro operations in Clojure?

Just curious, what part of application do you want to improve the performance of? — albusshin, Apr 04 '14 at 15:46
I don't think there is an direct answer. You may want to watch Zach Tellmans talk "Predictably fast Clojure" from this years Clojure talk https://www.youtube.com/watch?v=iQwQXVM6oiY — Leon Grapenthin, Apr 04 '14 at 15:46
@AlbusShin at the moment I'm actually genuinely interested in the performance of `assoc`. I'm interested in hashmap performance under structural sharing vs copy-and-edit. — MRocklin, Apr 04 '14 at 15:49

score 5 · Answer 1 · answered Apr 04 '14 at 16:31

Criterium tries to net out the overhead due to its own measurement. This can result in negative results for fast functions. See the Measurement Overhead Estimation section of the readme. Your overhead is suspiciously high. You might run (estimatated-overhead!) [sic] a few times to sample for a more accurate overhead figure.

score 5 · Accepted Answer · answered Apr 06 '14 at 02:55

5

Why not just wrap it in a loop and renormalize?

On my hardware,

(bench (dotimes [_ 1000] (assoc d 3 3)))

yields a mean execution time of roughly 1000x that of

(bench (assoc d 3 3))

namely, about 100 µs in the first case, and 100 ns in the second. If your single-assoc is "in the noise" for Criterium, you could try wrapping it in the same way and get pretty close to the "intrinsic" value. ((dotimes [_ 1000] 1) clocks in at .59 µs, so the extra overhead imposed by the loop itself is comparatively small.)

answered Apr 06 '14 at 02:55

JohnJ

4,753
2
28
40

Any danger that this takes advantage of cached results? Or is each `assoc` call doing equivalent work to the first? – MRocklin Apr 06 '14 at 17:29
From the Clojure standpoint, you'd have to memoize to get caching, I think. Of course the hardware's use of its cache is another story, as are JVM particulars. But the above test tells a pretty simple tale. – JohnJ Apr 06 '14 at 18:26

score 4 · Answer 3 · answered Apr 04 '14 at 16:26

Andy Fingerhut has maintained some of these for a while.

ClojureScript:

Project - https://github.com/netguy204/cljs-bench
Results - http://www.50ply.com/cljs-bench/

Microbenchmark Clojure functions

Question

Using time

Using Criterium

Repeat the Question

3 Answers3

Using `time`

Using `Criterium`