Elapsed time in lazy sequence evaluation

Question

Given this code:

(reduce my-fun my-lazy-seq)

To measure the elapsed time of the entire operation:

(time (reduce my-fun my-lazy-seq))  ;;Elapsed time: 1000.1234 msecs

How do I measure the elapsed time of this loop at various stages before completion? For example:

Elapsed time to process next 1000 samples in my-lazy-seq: 100.1234 msecs  
Elapsed time to process next 1000 samples in my-lazy-seq: 99.1234 msecs  
Elapsed time to process next 1000 samples in my-lazy-seq: 101.1234 msecs  
...

I think a solution would be to convert the lazy sequence into a lazy sequence of batches of size 1000, and put the time around processing of each batch... but I am not sure how to do the first step in a clean way, given the origin code is already so simple and elegant. — CJLam, Apr 03 '15 at 15:20

score 2 · Accepted Answer · answered Apr 04 '15 at 11:45

2

(doseq [thousand (partition 1000 my-lazy-seq)]
  (time (reduce my-fun thousand)))

answered Apr 04 '15 at 11:45

Leon Grapenthin

9,246
24
37

score 1 · Answer 2 · edited May 23 '17 at 10:24

How about this:

(defn seq-counter [n coll]
  (let [t0 (System/currentTimeMillis)
        f (fn [i x]
            (let [i (inc i)]
              (if (= 0 (rem i n))
                (println i "items processed in" (- (System/currentTimeMillis) t0) "ms.")) 
              x))]
    (map-indexed f coll)))

map-indexed used to check the progress. The above function will print the count and processing time in every n elements.

user=> (reduce + (seq-counter 10 (range 100)))
10 items processed in 0 ms.
20 items processed in 0 ms.
...
100 items processed in 1 ms.
4950

Refer to Idiomatic clojure for progress reporting?

A very minor adjustment, since `if` has only one branch, you can put a `when` — T.Gounelle, Apr 03 '15 at 09:16

Elapsed time in lazy sequence evaluation

2 Answers2