caching parsed data in clojure

Question

Say I do some parsing to generate a dataset. The job can be pretty slow, so is the loading of my webpage that is displaying the dataset.

Is there a way to tell the server to parse say every minute, to cache this and to rely on the cache to display results.

I've taken a look at core.cache and its docs but I can't figure out a simple way to just do that:

(require '[clojure.core.cache :as cache])

(def Dataset 
  (atom 
    (-> (parse-fn "http://nytimes.com")
        (cache/ttl-cache-factory :ttl 20000))) 

How can the cache be refreshed do I can rely on it?

(build-html-table @Dataset)

Thank you!

Answer 1

You could start up an extra thread that handles the data updates (see this question for ways to periodically run a function), e.g.:

(defonce data (atom nil))
(defonce updater
  (future
    (while true
      (reset! data (parse-fn "http://nytimes.com"))
      (Thread/sleep 60000))))

To access the data:

@data

To cancel the updater:

(future-cancel updater)

Make sure to handle exceptions within parse-fn. Also, you might want to wait for the initial data to be available which could be achieved using a promise that gets delivered from within updater.

---

core.cache won't really work for your case unless you can accept that requests coming in after the TTL has expired will have to wait until the data is back in the cache. Also, when caching function results core.memoize already does most of the work for you:

(require '[clojure.core.memoize :as memo])
(defn fetch-dataset!
  (memo/ttl
    #(parse-fn "http://nytimes.com")
    :ttl/threshold 60000))

(fetch-dataset!) ;; => <your data> (takes a while)
(fetch-dataset!) ;; => <your data> (immediately)
... 60s ...
(fetch-dataset!) ;; => <your data> (takes a while)