ParserMacros

Parser Macros

(This is a continuation of a discussion which started in the issue tracker but has outgrown that format and would benefit from a more structured design document.)

It would be useful in some cases for code to be able to emit tables which print thru fennel.view as literals but have some special behavior attached to them. For instance, if you wanted to implement a set, you can do this presently with a table that has a special metatable attached to it, but you would want it still to be able to round-trip thru fennel.view and the parser and get a set back on the other side. This requires parser macros.

This feature was previously discussed as "reader macros" but since Fennel does not use the term "reader" the way other lisps do, the term "parser macros" seems like a better fit.

This is a rough proposal in brainstorming stages; there is no concrete plan to implement it. An alternative proposal is require field notation.

There are three main questions to decide:

Activation

Parser macros need to be specified at the compiler's entry points. It only seems natural for macros to be specified in terms of modules, similar to how compiler plugins currently work. For instance:

table.insert(package.loaders,
             fennel.make_searcher({correlate=true,
                                   parserMacros={"foo.bar", "cljlib"}}))

In this case, the parser macro modules foo.bar and cljlib would be active during parsing. The CLI launcher would also support a way to activate them:

$ fennel --parser-macro foo.bar --parser-macro cljlib mycode.fnl

You could also pass them just to the opts table of fennel.parser if you're using the parser on its own.

Notation

Parser macros are a way to tag a piece of literal data which is already valid Fennel in order to indicate it's given special treatment somehow. We will put a prefix before the data; the prefix is used to look up which macro function will be passed the data at parse time.

We have currently reserved the @ character; since it is not allowed to be used as a symbol character, it would be a good fit for acting as a separator between the prefix and the data. It's also been suggested that we use # as a separator.

foo.bar@[:a :b :c]

foo.bar#[:a :b :c]

Early proposals had the notation start with a reserved character in addition to having a separator character between the prefix and the data. Another option is to begin the notation with a reserved character but omit the separator between the prefix and the data. This is tidier, but it means that the data cannot be a symbol or number.

@foo.bar[:a :b :c]

The big question here is whether the parser macro modules specified in the activation can contain multiple macro definitions, or if we enforce a 1:1 mapping between modules and macros themselves. If the module can contain several macros, then we need a way to refer to the parser macro module and the field inside it. If the module simply contains a single macro, then the module name itself is enough.

foo.bar@[:a :b :c]

foo.bar/set@[:a :b :c]

This is difficult since there are very few limitations on what constitutes a valid module name. I believe this is a decent argument for requiring parser modules to map 1:1 with the macros themselves and not have fields in them. (In practice this means the module would contain a function instead of a table, or maybe a table where the field name containing the macro function is fixed.)

Compilation

The goal here is to have metadata-enhanced tables which can be roundtripped thru fennel.view and the parser back and forth. The table with its metadata should be available to regular macros which are passed values coming from parser macros, but it also needs to compile to Lua output which preserves the metadata.

Maybe this will look like having the parser macros return a table which has its metatable set along with a new metamethod to allow the table to emit its own compilation into Lua?

TODO: need more detail here