Macros Outside of Lisp: Why I First Liked Elixir

There are lots of excellent reasons to learn Elixir, and most of these are due to Erlang.

The rest are mostly due to the Elixir core team's unfailingly good sense of design, and the best-in-class community they've built around the language.

But when you add things to your life, things that you enjoy coming back to -- the books, music, movies, tools, practices, places or people that you can live with, year after year, the ones that resonate with you -- sometimes it's something a bit more difficult to pin down, not the most obvious and weighty virtues, that resonate.

The real reason, the guilty pleasure reason that I first decided that I liked Elixir, is that it has Lisp-style macros in a productive language with syntax. Macros just make so much sense as a language feature (though Lisp's expressive power isn't always spoken of as a positive1).

To some, this will sound odd. Most programmers go through a Lisp infatuation phase at some point, but why single out a language feature that's rarely-used in your day-to-day programming? After all, the number one rule of macro writing is "don't write a macro." You don't need to write macros at all to use Elixir very productively.

Well, one way to think about macros is to demote them -- I often think of macros as "just compilation" -- little units of true compilation that you can use in your program, the same way you use functions. And this helps me feel less guilty about liking them, or pining for them in languages that don't have them. "I have this, and I want it to mean that, and both expressions parse." That's not a fit for every problem, of course, but it feels like such a reasonable thing to have available!

So when macros are a good fit for your problem, and you don't have them available, you really feel shackled. Throwing futile shovelfuls of closures across the steaming highway, under the watchful gaze of the Parser With No Eyes ...

I've often run into complexity that I knew compilation was a better fit for, but not having macros often made it a harder sell, especially to myself, so I'd do the busy-work instead.

But "Macros outside of Lisp" are something I've been following and experimenting with for years, and I appreciate Elixir more because of it.

Experiments In Macros Outside of Lisp

In the summer of 2011, at the tail end of 9 months in South America, I wrote a prototype of Lisp-style macros in CoffeeScript2 in under 100 lines of code.

Laptop and Hammock

The idea being that we implement macros without requiring the language to look like a mass of parentheses, by altering the AST but leaving tokenizing/parsing of a language's distinctive syntax alone — in Lisp terms, macros without read-macros.

I was pretty excited at the time, and did a lightning talk at a monthly programming language User Group. I presented3 the project, and demoed two examples:

  1. "list comprehensions in Spanish" — so that instead of for x in [1..10] when x > 3 -> x, you could write para x en [1..10] si (x > 3) hazte eso pues -> x
  2. cc() — a control-flow macro that turned synchronous code async, by reaching down its throat and turning it inside-out. (Just callbacks, not continuations).

There's something in the air.

I hasten to add: that idea wasn't new! Dylan probably doesn't count, as it was Lisp with block syntax. And there were doubtless efforts to use 'Lisp-style' macros in non-lisps earlier in the history of programming.

But for recent examples, I learned that MetaLua had done this in 2007, and I had known that Potion could do macros in 2009 or so. In late 2011 Rust added them in their first form, and in 2012 Julia appeared. Later in 2012 Scala added them as experimental.

Notice anything interesting in that progression? It's that macros moved from toy languages, or toy language forks, to "big" languages. Rust. Scala. Language designers have begun to realize that you can include macros in your language without your users burning everything down. Macros, without read-macros.

While obvious to language designers, I was shocked to realize that there wasn't usually a hard technical reason4 why macros couldn't be added to an existing language as an AST transform step.

Another year of volunteer work in South America intervened, but I finally cleaned up the project a bit, made the code readable with Docco so that it could serve as a reference to others, and presented it in a non-lightning-talk format.

Unsolicited Skin Graft

Although a fork of CoffeeScript (BlackCoffee) was initially based on my prototype, and garnered some attention from @jashenkas, it was clear that any closer integration was unlikely. As the creator of a little language with macros, 'GorillaScript,' mentioned to me, once you have macros it makes sense to write as much of the language as possible with them.

But who would want to rewrite a language to use macros retroactively? In theory, some form of true macros could be bolted onto our favorite languages as just a transform step5. In practice, it makes the most sense to build a language on top of macros.

Obviously, languages shouldn't try to change their nature willy-nilly. So why even consider adding macros to existing languages? Because there weren't then, to my knowledge, any non-Lisps that had macros and were good for general-purpose programming, and had an active code ecosystem/package manager. The languages I mentioned above were validating macros outside of Lisp, but none of them6 were good choices for general-purpose programming (things like UTF, localization, good protocol/networking support, code sharing, etc).

A block-structured language with flexible syntax, Lisp-style macros and an active community seemed like a perfect language for everyday programming. But my idea to bolt macros on top of existing languages sucked. And I lacked in-depth knowledge of language implementations, the free time, and (realistically) the skills to consider anything else.

Macros are the Hammer that Jose Used When He Freaking Nailed It

So when it began to dawn on me the extent to which Jose Valim had nailed it with Elixir, it was a little bit like living in a dream. Everything just made so much sense.

Of course, the things that are hardest to learn about Elixir to a non-Erlangist, the genuinely new ways of thinking, don't really have to do with the language. They have to do with Beam, and OTP, and the bits of Erlang that Elixir dispatches to — not language features like pattern matching, recursion, multiple functions heads, or even Lisp-style macros.

But Elixir makes a lot of bold decisions for a language trying to be accessible and productive. Processes, modules and functions only — no records, classes or objects! Macros make it work; they're such a useful compile-time means of abstraction and combination that stateful, runtime meta-programming can be merrily marginalized by language designers. They can say "you ain't gonna need it," secure in the knowledge that they have macros to fall back on.

It's paid off everywhere: code-generating UTF8 support; replacing 'macro methods' with actual macros; relational query languages that are languages; 'gettext' support. Protocols.

Elixir inherits the most from its Erlang heritage, and it's that heritage we credit for feeling that we've been gifted some kind of advanced alien technology. When I killed a supervised process tree in an OTP application, and for the first time I watched my system route around the damage and repair itself successfully, I didn't thank Lisp particularly.

But when I read Elixir code, I feel as though a lot of Lisp people have a reason to smile.

  1. The Lisp Curse, written April 2011, claims that Lisp's expressivity makes programmers less dependent on community, and posits a relative chilling effect on code sharing. The phrase is sort of a shorthand for Lisp's less-active library/package ecosystem. Personally, I think Lisp is doing fine (CL, Racket, Clojure, Joxa(!), and of course Emacs), but I think that Elixir will be as successful as the languages people compare Lisp to when they talk about the 'Lisp Curse.'

  2. The original code, which I put on github when I got back to the States, was dense and terrible. It only ran in the browser, as that was all I had access to while developing.

  3. A pretty cheesy-looking presentation

  4. ... because software is mostly social. However, there are 'soft technical reasons' it's easier in some languages: (1) an AST that can easily be serialized and deserialized makes 'quote' easier to implement, (2) for macro calls, a flexible syntax lets macros look good and blend in, and (3) for macro writers, pattern matching helps with the complex shape of non-Lisp quoted expressions.

  5. People have tried to retroactively add macros to already-popular languages, but empirically they seem to result in capable but little-used forks (MetaLua, Template Haskell). I'm not sure if Scala (whose macros were added in 2012 though still namespaced 'experimental') counts, as Scala in 2012 was still pretty malleable. For Rubyists, a project called RubyMacros made an effort in an earlier version of the language, and later Rubinius (the third-most-popular implementation of Ruby) supported parse transforms. But in Ruby's world, where run-time meta-programming reigns, it's never been clear why macros would be worth the disruption.

  6. One huge shoutout to Factor, a capable and brain-bending Forth-inspired language that can do macros (among many, many other things). However, there was no formalized code sharing in packages that would have supported a package manager at the time; last I heard they still didn't have one.