2

I'm using HTTP Kit to make requests, and I want them to be asynchronous, but I also want to cache the responses. The reason I want the requests to be asynchronous is that I am making several concurrently and I want them to operate in parallel.

Here's my function that makes several requests in parallel.

(defn parallel-requests-1 [urls]
  (let [; Dispatch all requests to run concurrently.
        responses (doall (map #(http/get %) urls))
        ; Realise all promises.
        realised (doall (map deref responses))
        ; Extract response body.
        bodies (map :body realised)]
    bodies))

(parallel-requests-1 ["http://example.com", "http://example.net"])

This is purely for illustrative purposes to demonstrate that I don't want to just deref the promise and memoize that.

Now I want to add caching using memoize. I tried this:

(def memoized-get (memoize http/get))

(defn parallel-requests-2 [urls]
  (let [; Dispatch all requests to run concurrently.
        responses (doall (map #(memoized-get %) urls))
        ; Realise all promises.
        realised (doall (map deref responses))
        ; Extract response body.
        bodies (map :body realised)]
    bodies))

All the signs show that this works well.

Is this a sensible solution? My concern is that caching a promise might constitute some kind of resource leak.

Joe
  • 46,419
  • 33
  • 155
  • 245
  • 1
    It is a lot more sensible to memoize promises than to memoize the value because that way you don't have a race condition in case someone fires the operation multiple times since the memoization will happen when the operation starts and not when it ends. – Benjamin Gruenbaum Jul 28 '15 at 13:37
  • 1
    Also, a cached promise will contain more information that its result. It will, if rejected, give you access to the reason for rejection, which may be useful under some circumstances. For example, you may choose to retry if the remote service was not available, but not to retry if the reason was a null result. Or you may simply want to log/display the reason (again). – Roamer-1888 Jul 28 '15 at 15:55
  • Promises do already memoize their results -- it's not clear to me what you hope to achieve with the call to memoize. – schaueho Jul 28 '15 at 20:01
  • Promises memoize their own results, but the call to `http/get` returns a different promise each time, and thus results in a new request. I want to cache the result of the `http/get`, i.e. store each resultant promise. – Joe Jul 28 '15 at 20:30
  • related: [Memoization of promise-based function](http://stackoverflow.com/q/28763057/1048572) – Bergi Jul 30 '15 at 18:06
  • Thanks @Bergi. I did see that but my question was about the mechanics of the clojure implementation wrt to possible resource leaks etc rather than the principle of promise caching. – Joe Jul 30 '15 at 20:06

1 Answers1

0

Yes, it is sensible to memoize a promise. In fact, it's probably the most sensible solution in this situation. After all, that's what http/get returns, and that is the function you want to memoize.

As far as resource leakage, that shouldn't be a problem any more than usual whether you memoize the promise or its value. A promise is basically just some state that references value once it's deliver!ed. So you have a slight overhead for this reference, but it's minuscule in comparison to the whole response.

Nathan Davis
  • 5,636
  • 27
  • 39
  • Thanks very much. In the past I've had a finite supply of `chan`s (the thread has just blocked when I try to create too many), so I wanted to be sure. I've not had any problems caching promises so far (though it's only been a day). – Joe Jul 30 '15 at 17:00