Use of eval inside macro (global vars unexpectedly resolved)

Question

Recently I came across a use for eval within a macro, which I understand is a bit of a faux pas but let's ignore that for now. What I found surprising, was that eval was able to resolve global vars at macroexpansion time. Below is a contrived example, just to illustrate the situation I'm referring to:

(def list-of-things (range 10))

(defmacro force-eval [args]
  (apply + (eval args)))

(macroexpand-1 '(force-eval list-of-things))

; => 45

I would have expected args to resolve to the symbol list-of-things inside force-eval, and then list-of-things to be evaluated resulting in an error due to it being unbound:

"unable to resolve symbol list-of-things in this context"

However, instead list-of-things is resolved to (range 10) and no error is thrown - the macroexpansion succeeds.

Contrast this with attempting to perform the same macroexpansion, but within a local binding context:

(defmacro force-eval [args]
  (apply + (eval args)))

(let [list-of-things (range 10)]
  (macroexpand-1 '(force-eval list-of-things)))

; => Unable to resolve symbol: list-of-thingss in this context

Note in the above examples I'm assuming list-of-things is not previously bound, e.g. a fresh REPL. One final example illustrates why this is important:

(defmacro force-eval [args]
  (apply + (eval args)))

(def list-of-things (range 10 20))

(let [list-of-thing (range 10)]
  (macroexpand-1 '(force-eval list-of-things)))

; => 145

The above example shows that the locals are ignored, which is expected behavior for eval, but is a bit confusing when you are expecting the global to not be available at macroexpansion time either.

I seem to have a misunderstanding about what exactly is available at macroexpansion time. I had previously thought that essentially any binding, be it global or local, would not be available until runtime. Apparently this is an incorrect assumption. Is the answer to my confusion simply that global vars are available at macroexpansion time? Or am I missing some further nuance here?

Note: this related post closely describes a similar problem, but the focus there is more on how to avoid inappropriate use of eval. I'm mainly interested in understanding why eval works in the first example and by extension what's available to eval at macroexpansion time.

Answer 1

Of course, vars must be visible at compile time. That's where functions like first and + are stored. Without them, you couldn't do anything.

But keep in mind that you have to make sure to refer to them correctly. In the repl, *ns* will be bound, and so a reference to a symbol will look in the current namespace. If you are running a program through -main instead of the repl, *ns* will not be bound, and only properly qualified vars will be found. You can ensure that you qualify them correctly by using

`(force-eval list-of-things)

instead of

'(force-eval list-of-things)

Note I do not distinguish between global vars and non-global vars. All vars in Clojure are global. Local bindings are not called vars. They're called locals, or bindings, or variables, or some combination of those words.

Answer 2

Clojure is designed with an incremental compilation model. This is poorly documented.

In C and other traditional languages, source code must be compiled, then linked with pre-compiled libraries before the final result can be executed. Once execution begins, no changes to the code can occur until the program is terminated, when new source code can be compiled, linked, then executed. Java is normally used in this manner just like C.

With the Clojure REPL, you can start with zero source code in a live executing environment. You can call existing functions like (+ 2 3), or you can define new functions and variables on the fly (both global & local), and redefine existing functions. This is only possible because core Clojure is already available (i.e. clojure.core/+ etc is already "installed"), so you can combine these functions to define your own new functions.

The Clojure "compiler" works just like a giant REPL session. It reads and evaluates forms from your source code files one at a time, incrementally adding them the the global environment. Indeed, it is a design goal/requirement that the result of compiling and executing source code is identical to what would occur if you just pasted each entire source code file into the REPL (in proper dependency order).

Indeed, the simplest mental model for code execution in Clojure is to pretend it is an interpreter instead of a traditional compiler.

Answer 3

And eval in a macro makes no sense. Because:

1. a macro already implicitely contains an eval at the very final step.

   If you use macroexpand-1, you make visible how the code was manipulated in the macro before the evocation of the implicite eval inside the macro.

   An eval in a macro is an anti-pattern which might indicate that you should use a function instead of a macro - and in your examle this is exactly the case.

2. So your aim is to dynamically (in run-time) evoke sth in a macro. This you can only do through an eval applied over a macro call OR you should rather use a function.

(defmacro force-eval [args]
  (apply + (eval args)))

;; What you actually mean is:
(defn force-eval [args]
  (apply + args))
;; because a function in lisp evaluates its arguments 
;; - before applying the function body.
;; That means: args in the function body is exactly
;; `(eval args)`!

(def list-of-things (range 10))
(let [lit-of-things (range 10 13)]
  (force-eval list-of-things))
;; => 45

;; so this is exactly the behavior you wanted!

The point is, your construct is a "bad" example for a macro. Because apply is a special function which allows you to dynamically rearrange function call structures - so it has some magic of macros inside it - but in run-time. With apply you can do quite some meta programming in some cases when you just quote some of your input arguments. (Try (force-eval '(1 2 3)) it returns 6. Because the (1 2 3) is put together with + at its front by apply and then evaluated.)

The second point - I am thinking of this answer I once gave and this to a dynamic macro call problem in Common Lisp.

In short: When you have to control two levels of evaluations inside a macro (often when you want a macro inject some code in runtime into some code), you need too use eval when calling the macro and evaluate those parts in the macro call which then should be processed in the macro.