Why aren't lisp macros eagerly expanded by default?

Question

Say I have macros foo and bar. If I write (foo (bar)) my understanding is that in most (all?) lisps foo is going to be given '(bar), not whatever bar would have expanded to had it been expanded first. Some lisps have something like local-expand where the implementation of foo can explicitly request the expansion of its argument before continuing, but why isn't that the default? It seems more natural to me. Is this an accident of history or is there a strong reason to do it the way most lisps do it?

I've been noticing in Rust that I want the macros to work this way. I'd like to be able to wrap a bunch of declarations inside a macro call so that the macro can then crawl the declarations and generate reflection information. But if I use a macro to generate the definitions that I want crawled, my macro wrapping the declarations sees the macro invocations that would generate the declarations rather than the actual declarations.

it looks more complex to me to process first the inner forms, then the parent macro, then again all the possible subterms that were introduced by the macro; usually if you need to, you can call macroexpand yourself in a macro, or a more implementation-specific function like SBCL's code walker. I am not sure I can find archives where this is discussed in details w.r.t. different lisp versions — coredump, Sep 29 '21 at 13:56
with the existing approach the macro has more control over all its subforms, it can also macroexpands code itself, e.g. it is possible to reuse symbols to mean something else, like lamda or defun, in order to interpret code differently (e.g. static anlysis), and recursive macroexpansion can terminate — coredump, Sep 29 '21 at 14:00
@coredump hmm.. one motivation to do it the way I describe is it would make macros less special -- they get their arguments evaluated before being passed in just like functions, it's just that their argument is only "evaluated" at the syntax/macro-expansion level, not at the runtime expression level. They become more like runtime functions that just happen to take compile time known syntax, and where as an optimization we can choose to run them at compile time. But if it's the default you need an opt-out for the cases you describe.... — Joseph Garvin, Sep 29 '21 at 15:23
@coredump like declaring as part of the macro definition a list of forms you don't want expanded before the syntax is passed in because you plan to treat them specially — Joseph Garvin, Sep 29 '21 at 15:24

Rainer Joswig · Answer 1 · 2021-09-29T21:39:57.423

If I write (foo (bar)) my understanding is that in most (all?) lisps foo is going to be given '(bar), not whatever bar would have expanded to had it been expanded first.

That would restrict Lisp such that (bar) would need to be something that can be expanded -> probably something which is written in the Lisp language.

Lisp developers would like to see macros where the inner stuff can be a completely new language with different syntax rules. For example something, where FOO does not expand it's subforms, but transpiles/compiles a fully/partially different language to Lisp. Something which does not have the usual prefix expression syntax:

Examples

(postfix (a b +) sin)

  -> (sin (+ a b))

Here the + in the macro form is not the infix +.

or

(query-all (person name)
   where (person = "foo") in database DB)

Lisp macros don't work on language parse trees, but arbitrary, possibly nested, s-expressions. Those don't need to be valid Lisp code outside of that macro -> don't need to follow the usual syntax / semantics.

Common Lisp has the function MACROEXPAND and MACROEXPAND-1, such that the outer macro can expand inner code if it wants to do it during its own macro expansion:

CL-USER 26 > (defmacro bar (a) `(* ,a ,a))
BAR

CL-USER 27 > (bar 10)
100

CL-USER 28 > (defmacro foo (a &environment e)
               (let ((f (macroexpand a e)))
                 (print (list a '-> f))
                 `(+ ,(second f) ,(third f))))
FOO

CL-USER 29 > (foo (bar 10))

((bar 10) -> (* 10 10))
20

In above macro, if FOO would see only an expanded form, it could not print both the source and the expansion.

This works also with scoped macros. Here the BAR macro gets locally redefined and the MACROEXPAND generates different code inside FOO for the same form:

CL-USER 30 > (macrolet ((bar (a)
                          `(expt ,a ,a)))
               (foo (bar 10)))

((bar 10) -> (EXPT 10 10))
20

Kaz · Answer 2 · 2022-02-07T22:57:53.983

If foo is a macro then (foo (bar)) must pass the raw syntax (bar) to the foo macro expander. This is absolutely essential.

This is because foo can give any meaning whatsoever to bar.

Consider the defmacro macro itself:

(defmacro foo (bar) body)

Here, the argument (bar) is a parameter list ("macro lambda list") and not a form (Common Lisp jargon for to-be-evaluated expression). It says that the macro shall have a single parameter called bar. Therefore it is nonsensically wrong to try to expand (bar) before handing it to defmacro's expander.

Only if we know that an expression is going to be evaluated is it legitimate to expand it as a macro. But we don't know that about an expression which is the argument to a macro.

Other counterexamples are easy to come up with. (defstruct point (x 0) (y 0)): (x 0) isn't a call to operator x, but a slot x whose default value is 0. (dolist (x list) ...): x is a variable to be stepped over list.

That said, there are implementation choices regarding the timing of macro expansion.

A Lisp implementation can macro-expand an entire top-level form before evaluating or compiling any of it. Or it can expand incrementally, so that for instance when (+ x y) is being processed, x is macro-expanded and evaluated or compiled into some intermediate form already before y is even looked at.

A pure syntax tree interpreter for Lisp which always keeps the code in the original form and always expands (and re-expands) the code as it is evaluating has certain interactivity advantages. Any macro that you rewrite goes instantly "live" in all the existing code that you have input into the REPL, like existing function definitions. It is obviously quite inefficient in terms of execution speed, but any code that you call uses the latest definition of your macros without any hassle of telling the system to reload that code to have it expanded again. That also eliminates the risk that you're testing something that is still based on the old, buggy version of some macro that you fixed. If you're ever writing a Lisp, the range of timing choices for expansion is good to keep in mind, so that you consciously reject the choices you don't go with.

In turn, that said, there are some constraints on the timing of macro expansion. Conceivably, a Lisp interpreter or compiler, when processing an entire file, could go through all the top level forms and expand all of them at once before processing any of them. The implementor will quickly learn that this is bad, because some of the later forms depend on the side effects of the earlier forms. Such as, oh, macros being defined! If the first form defines a macro, which the second one uses, then we cannot expand the second form without evaluating the effect of the first.

It makes sense, in a Lisp, to split up physical top-level forms into logical ones. Suppose that someone writes (or uses a macro to generate) codde like (progn (defmacro foo ...) (foo)). This entire progn cannot be macro expanded up-front before evaluation; it won't work! There has to be a rule such as "whenever a top-level form is based on the progn operator, then the children of the progn operator are considered top-level forms by all the processing which treats top-level forms specially, and this rule is recursively applied." The top-level entry point into the macro-expanding code walker then has to contain special case hacks to do this recognition of logical top-level forms, breaking them up and recursing into a lower level expander which doesn't do those checks any more.

score 2 · Answer 3 · answered Sep 29 '21 at 14:11

I've been noticing in Rust that I want the macros to work this way. I'd like to be able to wrap a bunch of declarations inside a macro call so that the macro can then crawl the declarations and generate reflection information.

It does sound like local-expand is the right tool for that job.

However, an alternative approach would be something like this:

Suppose that wrapper is our outer macro, and that the intended syntax is:

(wrapper decl1 decl2 ...)

where decl is a declaration that potenteally uses some standard form declare.

We can let (wrapper decl1 decl2 ...) expand to

(let-syntax ([declare our-declare])
   decl1 decl2 ... 
   (post-process-reflection-information))

where our-declare is a helper macro that expands both to the standard declaration as well as some form that stores the reflection information, also post-process-reflection-information is another macro that does any needed post processing.

That's a nice approach. Unfortunately specifically in Rust the hygiene rules prevent this, the wrapped macro calls either have to refer to a macro that already exists, or the outer macro needs to swap them in for something different itself. If the outer macro just expands to provide it's own definition of the wrapped macro, the user's invocations won't refer to it. — Joseph Garvin, Sep 29 '21 at 15:19

coredump · Answer 4 · 2021-10-04T07:53:52.717

I think you are trying to use macros for something they are not designed to solve. Macros are primarily a text/code substitution mechanism, and in the case of Lisp this looks a lot like a simplified term-rewriting system (see also How does term-rewriting based evaluation work?). There are different strategies possible for how to substitute a pattern of code, and in which order, but in C/C++ preprocessor macros, in LaTeX, and in Lisp, the process is typically done by computing the expansion until the form is no longer expandable, starting from the topmost terms. This order is quite natural and because it is distinct from normal evaluation rules, it can be used to implement things the normal evaluation rules cannot.

In your case, you are interested in getting access to all the declarations of some object/type, something which falls under the introspection/reflection category (as you said yourself). But implementing reflection/introspection with macros doesn't look totally doable, since macros work on abstract syntax trees and this might be a poor way to access the metadata you want.

Typically the compiler is going to parse/analyze the struct definitions and build the definitive, canonical representation of the struct, even if there are different way to express that syntactically; it may even use prior information not available directly as source code to compute more interesting metadata (e.g. if you had inheritance, there could be a set of properties inherited from a type defined in another module (I don't think this applies to Rust)).

I think currently Rust does not offer compile-time or runtime introspection facilities, which explains why are you going with the macro route. In Common Lisp macros are definitely not used for introspection, the actual values obtained after evaluation (at different times) is used to gain information about an object. For example, defclass expands as a set of instructions that register a class in the language, but in order to get all the slots of a class, you ask the language to give it to you, e.g:

(defclass foo () (x))       ;; define class foo with slot X
(defclass bar () (y))       ;; define class bar with slot Y
(defclass zot (foo bar) ()) ;; define class zot with foo and bar as superclasses

USER> (c2mop:class-slots (find-class 'zot))                                                                                                                                                   
(#<SB-MOP:STANDARD-EFFECTIVE-SLOT-DEFINITION X>                                                                                                                                               
 #<SB-MOP:STANDARD-EFFECTIVE-SLOT-DEFINITION Y>)

I don't know what the solution for your problem is, but in addition to the other answers, I think it is not specifically a fault of the macro system. If a macro is defined as done usually as only a term rewriting system, it will always have difficulties to perform some tasks on the semantic level. But Rust is still evolving so there might be better ways to do things in the future.

Why aren't lisp macros eagerly expanded by default?

4 Answers4