Clojure: memory leaks using eval

Question

My application evaluates quoted expressions received from remote clients. Overtime, my system's memory increases and eventually it crashes. What I've found out is that:

When I execute the following code from Clojure's nrepl in a docker container:

(dotimes [x 1000000] ; or some arbitrary large number
 (eval '(+ 1 1)))

the container's memory usage keeps rising until it hits the limit, at which point the system will crash.

How do I get around this problem?
There's another thread mentioning this behavior. One of the answers mentions the use of tools.reader, which still uses eval if I need code execution, leading to the same problem.

score 4 · Accepted Answer · answered Mar 12 '22 at 07:48

4

There's no easy way to get around this as each call to eval creates a new class, even though the form you're evaluating is exactly the same. By itself, JVM will not get rid of new classes.

There are two ways to circumvent this:

Stop using eval altogether (by e.g. creating your own DSL or your own version of eval with limited functionality) or at least use it less frequently, e.g. by batching the forms you need to evaluate
Unload already loaded classes - I haven't done it myself and it probably requires a lot of work, but you can follow answers in this topic: Unloading classes in java?

answered Mar 12 '22 at 07:48

Eugene Pakhomov

9,309
3
27
53

I don't think this is correct: "By itself, JVM will not get rid of new classes" -> see my answer. – Juraj Martinka Mar 17 '22 at 08:01
1

Thanks. I delved a bit deeper and seems like the truth is somewhere in between. My statement is correct. However, `eval` creates a new classloader every time, and those do get eventually GC'ed along with all the classes that were loaded by it. Why it can sometimes lead to OOM, I do not know. – Eugene Pakhomov Mar 17 '22 at 14:43
Right, every top-level form creates a new classloader and GC doesn't seem to (typically) have problems getting rid of those. I remember there might be something specific to nREPL although I do not know if that's applicable here: "In a nREPL client, instances of DynamicClassLoader keep piling up." (see https://danielsz.github.io/blog/2021-05-12T13_24.html) – Juraj Martinka Mar 17 '22 at 16:39
1

A third option: replace your usage of eval with https://github.com/babashka/sci, it does not suffer from the above problem. – Michiel Borkent Apr 20 '23 at 15:19

score 1 · Answer 2 · answered Mar 17 '22 at 08:00

I don't know how eval exactly works internally, but based on my observations I don't think your conclusions are correct and also Eugene's remark "By itself, JVM will not get rid of new classes" seems to be false.

I run your sample with -Xmx256m and it went fine.

  (time (dotimes [x 1000000] ; or some arbitrary large number
          (eval '(+ 1 1))))
  ;; "Elapsed time: 529079.032449 msecs"

I checked the question you linked and they say it's Metaspace that is growing not heap. So I observed the Metaspace and it's usage is growing but also shrinking.

You can find my experiment here together with some graphs from JMC console: https://github.com/jumarko/clojure-experiments/commit/824f3a69019840940eaa88c3427515bcba33c4d2

Note: To run this experiment, I've used JDK 17.0.2 on macOS

Clojure: memory leaks using eval

2 Answers2