Undefined function after macroexpansion

Question

I'm studying Common Lisp and want to play with lisp and web development. My current problem comes from a simple idea to iterate over all javascript files i want to include. I use SBCL and Quicklisp for fast startup. The problem could be related to the cl-who package I'm using.

So I've declared my package and started like this:

(defpackage :0xcb0
  (:use :cl :cl-who :hunchentoot :parenscript))
(in-package :0xcb0)

To keep it simple I reduced my problem function. So I have this page function:

(defun page (test)
  (with-html-output-to-string
    (*standard-output* nil :prologue nil :indent t)
    (:script
     (:script :type "text/javascript" :href test))))

This will produce the desired output

*(0xcb0::page "foo")

<script>
   <script type='text/javascript' href='foo'></script>
</script>

Now my I've created a macro which produces :script tags.

(defmacro js-source-file (filename)
  `(:script :type "text/javascript" :href ,filename)))

This works as expected:

*(macroexpand-1 '(0XCB0::js-source-file "foo"))

(:SCRIPT :TYPE "text/javascript" :HREF "foo")

However if I include this into my page function:

(defun page (test)
  (with-html-output-to-string
    (*standard-output* nil :prologue nil :indent t)
    (:script
     (js-source-file "foo"))))

...it will give me a style warning (undefined function: :SCRIPT) when defining the new page function. Also, the page function produces this error when executed:

*(0xcb0::page "foo")

The function :SCRIPT is undefined.
   [Condition of type UNDEFINED-FUNCTION]

Why does the embedded macro js-source-file works as expected, in that it produces the desired output, but fails when is called within another function?

P.S. I know the topic of macros in Lisp can be quite exhausting for a beginner like me. But currently I can't wrap my head around the fact that this should work but doesn't!

I'll have to test it later, but the problem is probably that `with-html-output-to-string` is a macro which turns the `(:script ...)` forms inside it into some other code. Since `js-source-file` hasn't been expanded yet, `with-html-output-to-string` doesn't process it. — jkiiski, Jan 30 '16 at 14:43

score 8 · Accepted Answer · answered Jan 30 '16 at 15:22

The problem is that macros are expanded a bit counter intuitively in order from outmost to inmost. For example:

(defmacro foobar (quux)
  (format t "Foo: ~s~%" quux))

(defmacro do-twice (form)
  `(progn
     ,form
     ,form))

(foobar (do-twice (format t "qwerty")))

The output will be

Foo: (DO-TWICE (FORMAT T "qwerty"))

foobar never sees the expansion of do-twice. You could avoid the problem by calling macroexpand yourself in foobar:

(defmacro foobar (quux)
  (format t "Foo: ~s~%" (macroexpand quux)))

(foobar (do-twice (format t "qwerty")))
; => Foo: (PROGN (FORMAT T "qwerty") (FORMAT T "qwerty"))

Since you're using a third party macro, that probably isn't a good solution. I think the best option is to generate the markup yourself in js-source-file. I'm not familiar with cl-who, but this seemed to work in my quick test:

(defun js-source-file (filename stream)
  (with-html-output (stream nil :prologue nil :indent t)
    (:script :type "text/javascript" :href filename))))

(defun page (test)
  (with-output-to-string (str)
    (with-html-output (str nil :prologue nil :indent t)
      (:script
       (js-source-file test str)))))

It indeed seems to be related to the internals of `with-output-to-string`. Your suggested solution works fine. I however hoped to get around the `with-html-output` helper function in every function. You are also right that third pary macro is not a good solution. For testing it seemed to be the right choice but as I see it can be done using cl alone with little extra work! Thanks, it made my knowledge of how cl works a bit deeper. — cb0, Jan 30 '16 at 15:39

score 4 · Answer 2 · edited May 23 '17 at 11:59

In addition to the other good answer, I'll cover the particular case of with-html-output. This is derived from the Syntax and Semantics section of the cl-who manual.

First, note that if you macroexpand the call yourself, you can see that with-html-output establishes macrolet bindings such as str, htm, ftm, esc... The htm macrolet takes no argument (except a body) and expands into a with-html-output form which has the same parameters has the lexically enclosing with-html-output macro. In order to fix your code, you can modify your macro as follows:

(defmacro js-source-file (filename)
  `(htm (:script :type "text/javascript" :href ,filename)))

Then:

with-html-output gets expanded into a tree which contains (js-source-file ...)
You macro is expanded and produces an (htm (:script ...)) form.
Then, the macrolet is expanded to produce an inner (with-html-output ...) form.
The inner with-html-output is expanded and processes (:script ...).

You have to choose if you prefer to use macros or functions here. Functions are generally not inlined and can easily be redefined at runtime. Macros can in theory be expanded at runtime too, but in most implementations (and default configurations) you have to recompile whichever function depends on a macro. You could also let the macro call an auxiliary function.

Thanks for the detailed explanation! For me I'll now stick to cl-who and use more functions. This should make my start more easy :) I tried to understand the cl-who documentation before, but it seems I need more practice with cl and the docs coming with it. — cb0, Jan 31 '16 at 17:06

Undefined function after macroexpansion

2 Answers2