`apply` or `funcall` for macros instead of functions

Question

In Lisp, a function's arguments are evaluated first before entering the function body. Macro arguments stay not evaluated.

But sometimes, one wants to inject code pieces stored in variables into a macro. This means evaluating the argument for the macro first, and then apply the macro-of-choice on this evaluated result.

One has to resort to

(eval `(macro ,arg))

To achieve this - but eval does not behave correctly in different environments.

The best thing would be, if one could do:

(apply macro (list arg))

or

(funcall macro arg)

But since the macro is not a function this doesn't work.

Is it possible to achieve something like this? - To circumvent that problem oder to make the macro available in the functions namespace?

Or am I missing some other ways to solve such problems?

I came to this question while trying to answer How to produce HTML from a list. but also in Generate TYPECASE with macro in common lisp, Evaluate arguments passed to a macro that generates functions in lisp, and How to convert a list to code/lambda in scheme?. But I always thought while answering them it would be good to have an apply or funcall-like function which can take macros.

You can funcall the macro function of a macro, which is a function. However what you are describing is almost certainly a design problem or a misunderstanding of what macros are for. — ignis volens, Nov 15 '21 at 10:57
@ignisvolens in as much misunderstanding or design problem? Can you give some examples? - I would say macros are for manipulating code. But sometimes one wants to manipulate code dynamically in runtime. Sometimes - one doesn't know which code one wants to manipulate before runtime. And then, one has to run macro on the piece of code generated in runtime. — Gwang-Jin Kim, Nov 15 '21 at 11:00
@Gwang-JinKim: please can you supply a minimum reproducible example of what you are trying to achieve — Francis King, Nov 15 '21 at 12:06
@FrancisKing I believe this question is related to the problem in [How to produce HTML from a list](https://stackoverflow.com/questions/69969577/how-to-produce-html-from-a-list). — Flux, Nov 15 '21 at 12:56
@Flux yes - it is related to that - but more general. I we would have a general way to do sth like `apply` on a macro, then many other problems will be solved. — Gwang-Jin Kim, Nov 15 '21 at 14:52
I don't understand what you mean by "but eval does not behave correctly in different environments." — coredump, Nov 15 '21 at 15:14
@coredump If I have a macro call - where I want to give the macro an argument after its first evaluation (because its content is actually what I want to give the macro) - how one can do that? the point is that let's say the variable `q` contains the actual argument. And I do `(mymacro q)` - then `q` does not evaluated before entering the macro body. How to evaluate it once and then call the macro on it ... I could only think of `(eval `(mymacro ,q))` ... but is there a way wihout `eval`? — Gwang-Jin Kim, Nov 15 '21 at 15:24
or probably calling `eval` inside a macro definition is the better way - since then at least the `eval` call is evaluated exaclty in that environment where the macro-call was invoked? — Gwang-Jin Kim, Nov 15 '21 at 15:29

ignis volens · Answer 1 · 2021-11-16T13:02:07.910

It is not clear what you are trying to do, although it is almost certain that you are confused about something. In particular if you are calling eval inside macroexpansions then in almost all cases you are doing something both seriously wrong and seriously dangerous. I can't ever think of a case where I've wanted macros which expand to things including eval and I have written Lisp for a very very long time.

That being said, here is how you call the function associated with a macro, and why it is very seldom what you want to do.

Macros are simply functions whose domain and range is source code: they are compilers from a language to another language. It is perfectly possible to call the function associated with a macro, but what that function will return is source code, and what you will then need to do with that source code is evaluate it. If you want a function which deals with run-time data which is not source code, then you need that function, and you can't turn a macro into that function by some magic trick which seems to be what you want to do: that magic trick does not, and can not, exist.

So for instance if I have a macro

(defmacro with-x (&body forms)
  `(let ((x 1))
     ,@forms))

Then I can call its macro function on a bit of source code:

> (funcall (macro-function 'with-x)
                     '(with-x (print "foo")) nil)
(let ((x 1)) (print "foo"))

But the result of this is another bit of source code: I need to compile or evaluate it, and nothing I can do will get around this.

Indeed in (almost?) all cases this is just the same as macroexpand-1):

> (macroexpand-1 '(with-x (print "foo")))
(let ((x 1)) (print "foo"))
t

And you can probably write macroexpand-1 in terms of macro-function:

(defun macroexpand-1/equivalent (form &optional (env nil))
  (if (and (consp form)
           (symbolp (first form))
           (macro-function (first form)))
      (values (funcall (macro-function (first form)) form env)
              t)
    (values form nil)))

So, if the result of calling a macro is source code, what do you do with that source code to get a result which is not source code? Well, you must evaluate it. And then, well, since the evaluator expands macros for you anyway, you might as well just write something like

(defun evaluate-with-x (code)
  (funcall (compile nil `(lambda ()
                           (with-x ,@code)))))

So you didn't need to call the macro's function in any case. And this is not the magic trick which turns macros into functions dealing with data which is not source code: it is a terrible horror which is entirely made of exploding parts.

A concrete example: CL-WHO

It looks like this question might have its origins in this one and the underlying problem there is that that's not what CL-WHO does. In particular it is a confusion to think that something like CL-WHO is a tool for taking some kind of list and turning it into HTML. It's not: it's a tool for taking the source code of a language which is built on CL but includes a way of expressing HTML output mingled with CL code, and compiles it into CL code which will do the same thing. It happens to be the case that CL source code is expressed as lists & symbols, but CL-WHO isn't really about that: it's a compiler from, if you like, 'the CL-WHO language' to CL.

So, let's try the trick we tried above and see why it's a disaster:

(defun form->html/insane (form)
  (funcall 
   (compile nil `(lambda () 
                   (with-html-output-to-string (,(make-symbol "O"))
                     ,@form)))))

And you might, if you did not look at this too closely, think that this function does in fact do the magic trick:

> (form->html/insane '(:p ((:a :href "foo") "the foo")))
"<p></p><a href='foo'>the foo</a>"

But it doesn't. What happens if we call form->html/insane on this perfectly innocuous list:

(:p (uiop/run-program:run-program "rm -rf $HOME" :output t))

Hint: don't call form->html/insane on this list if you don't have very good backups.

CL-WHO is an implementation of a programming language which is a strict superset of CL: if you try to turn it into a function to turn lists into HTML you end up with something involving the same nuclear weapon you tinker with every time you call eval, except that nuclear weapon is hidden inside a locked cupboard where you can't see it. But it doesn't care about that: if you set it off it will still reduce everything within a few miles to radioactive ash and rubble.

So if you want a tool which will turn lists – lists which aren't source code – into HTML then write that tool. CL-WHO might have the guts of such a tool in its implemenentation, but you can't use it as it is.

And this is the same problem you face whenever you are trying to abuse macros this way: the result of calling a macro's function is Lisp source code, and to evaluate that source code you need eval or an equivalent of eval. And eval is not only not a terrible solution to almost any problem: it's also a nuclear weapon. There are, perhaps problems for which nuclear weapons are good solutions, but they are few and far between.

Thank you for the answer. I understand what a macro is. And I know that a macro which evaluates its arguments should be rather a function. But now and then (see the links provided) you want to "misuse" a macro and use it like a function. We know that `with-html-output-to-string` is a macro for building html code. But as the person who was asking was asking - what if one already has the expression as a list and what when one wants to apply the macro on this pre-formed expression. It should be possible to achieve this - since lisp claims to be modable and foldable at will. — Gwang-Jin Kim, Nov 15 '21 at 16:11
Of course one could write a function version of `with-html-output-to-string` but this can become a lot of work. And one knows that the only difference betwen the macro and the function would be to evaluate the arguments once prior to giving them into the macro. I am just searching for this adapter. I don't see any reason why one should restrict running macros in that way. One could on the fly also call macros in runtime. — Gwang-Jin Kim, Nov 15 '21 at 16:14
@Gwang-JinKim: The fact that you can ask this question, I'm afraid, means that you really do not understand what macros are, because someone who did would not ask it. Lisp is indeed very mutable but even in Lisp magic is not possible. Unfortunately people who think they understand something but do not are usually the hardest people to explain it to: I hope my answer will help you but I suspect it may mostly be helpful for others. — ignis volens, Nov 16 '21 at 11:32

`apply` or `funcall` for macros instead of functions

1 Answers1

A concrete example: CL-WHO