Are lisp macros really useful in typical development?

Question

There are lots of delightful opinions about macros in Lisp, but I really cannot find a primer I can use in my every day workflow. I'm web developer, but use emacs for about 10 years. And I want to understand if macros are really useful for everyone, or it is just killer feature for very specific tasks.

Most of the primers using macros are tryings of making DSL on top of the lisp. Well it can be useful sometimes, but creating syntax extensions in commercial development is a bad practice as far as I understand. Most of languages has standard features like for, if, while, foreach operators, common conceptions like classes, namespaces, modules, closures(lambdas). So all programs are typically built from such pieces, and new programmer examining your code already knows that pieces and can operate them. And in case of DSL built upon macros, the programmer needs not only to explore business logic, but also learn your DSL. Commercial development gravities to typical solutions with easy support and high replaceability of programmers.

Next - creating template engines on top of the macros. Well, the industry standard now is Twig, Jinja2, Template Toolkit - you can create your own template system, but... It is 2015 now, not 2001 when Paul Graham's Viaweb was created. Modern templating systems are very similar to each other - they was developed for years and they have lots of conceptions like blocks, filters, inheritance, i18n. So in 2001 creating own templating engine had sense, but now there complex templating engines for all suddens of life.

Next - using macros to create algorithms independent on data passed to them. This conception makes sense, but does it really need macros? For example most of scripting languages support of passing functions by reference - that means you can create abstract function which will accept data and data-processing function as arguments. Of course you will need to create few data-processing functions to support all needed data types, but you will be able to pass all those ones to abstract function which will be short as comparable lisp macro.

Next - algorithms itself. Nowadays 90% of us, web-developers, are just using ready libraries and algorithms. So even if I'm totally wrong in previous paragraph, the role of macros is very small in our job.

So do you really use macros everyday, or you just arely shoot some problem with macros very elegantly and extrapolate that hapinees on next few monthes?

Answer 1

Macros are certainly useful in "typical development" because when done well, they greatly ease development tasks in many ways. For example, think of the ginormous number of websites out there written in Clojure using the wonderful defroutes macro from Compojure. Or within the Clojure.core library itself, all the useful macros, like when or if-let, or the very widely-used core.async library. They are all macros and dramatically empower the developer to do things that are otherwise not so easy without them.

As for building your own app with or without macros, in general they should be avoided until you have no choice but to use them. The two most common scenarios that come up for me when I turned to a macro:

1. I need to pass arguments to a "function" but not have them be evaluated at the call site. Macros are the only way to achieve this.
2. Boilerplate and tedious repetitious code setup is a perfect use-case for writing a macro to simplify things.

Functions, in general, are much more powerful. You can store a function as a value. You can pass a function as a an argument to another function. You can compose functions into new functions. You can't do these things with macros.

Answer 2

The quote

There is a quote I saw multiple times in the Clojure community :

data > functions > macros

Data

Why ? Well data is amazing : you can print it, transform it, serialize it to send it to the network... It is also language-agnostic. You can convert it into another representation and let another program written in another language use it. How powerful is that ? As a Clojurist, you can see that almost all Clojure functions are meant to manipulate data. It makes it a pleasure to manipulate data. To quote Fogus :

Well-formed Data at rest is as close to perfection in programming as it gets. All the crap that had to happen to put it there however..

Functions

Then come functions : in lisp we are lucky to have them mirror a data structure (a list), but it is still less composable than data. For instance, can you merge code ? Well let's assume that in this case merge means compose. Can you tell where a function comes from ? Can you diff it ? So in a sense, functions are a poor man's data.

Macros

But Clojure tries to alleviate the pain. And it has a few tools to do that. Some are simple (like meta, which really attaches data to a function), or macros. Lucky us ! We have macros to manipulate code. It is still less pleasant than to manipulate raw data, but it gets us there. Now for the usual argument of "but functions are data in Clojure" : how many of Clojure data functions work on functions ? There is a major inconvenient though. To quote Eric Normand :

Macros are opaque at runtime.

That is, they are not first-class citizens and you cannot pass macros as values. So how do we compose macros ? More macros ? It feels like a sliding path.

By now you can sense that we would prefer not to have to use them. They are very powerful of course (code that changes code ? That sounds pretty cool !).

By the way, I feel (although I haven't any reference) that Clojure tries to aim for readability and less parentheses than other lisps, at the expense of making macro writing slightly harder.

In short, macros are a very powerful tool, and I don't think I will want to program in a language that doesn't have them ever again. But you have to make good use of it.

Examples

The most classic use of a macro I can think of is to define the if construct. Could we do that with a function ? Let's try :

(defn if [condition if-true if-false]
   ...)

Can this work ? No, because when passing arguments to a function they will be evaluated. So the following :

(if true (println "true!") (println "false!)) ;; prints both "true!" and "false!"

Macros don't evaluate their arguments, and so are used in this case. You can find some in [clojure.core][3]. As for typical web development (define "typical" ?) let's have a look at some macros in the wild.

In [mori][3], there is a page in the wiki about benchmarking, where we can see that functions are defined with multiple arities. The macro to export those functions is defined here and used there.

From the same author, we can have a look at the [defui][6] macro of Om.

A lot of macros I have seen in the wild are also just to simplify declaration, like [defroute][6] in Compojure.

Also if you had time to read my answer, you should definitely watch "Zach Tellman - Always Be Composing".