Bug with $base

[wooorm]

Intro

Hi!

I’d like to discuss a bug.
The bug is not necessarily here. Nor in grammars, nor in tools that use linguist. It’s sort of in between everything, because how all the grammars combine.

Short aside, what is $base? Expand for info!

First, grammars include $self. That’s a reference to the grammar itself. When in source.js, this is an alias for source.js.

Grammars sometimes include $base. An alias for the top-level scope name.
source.c includes a pattern for functions which include $base. If source.c is used on its own, that means that functions can contain all the stuff you could put at the top-level of a C file.
source.c++ references source.c, and it has a pattern for namespaces (namespace { ... }).
Because $base in source.c now refers to source.cpp, that makes namespaces allowed in functions.

So it’s a way to allow other languages to extend your grammar. To include stuff inside your grammar.

Problem

A bug appears with $base being used in languages that embed code examples (markdown, restructuredtext) and languages like lua that support .cdef calls, julia that support icxx"myCppFunction();" calls, where they expect C/C++, not markdown/restructuredtext/lua/julia in there.

Here’s an illustration using markdown and C. This code block is marked as ```c for reference:

some_type function_name() {
*/ This is invalid in `source.c`
kAudioUnitSubType_3DMixer // This is invalid in to `source.c.platform` (included in `source.c`)
# Wait, is this *markdown*?!
}

This code block is marked as ````markdown, which includes C, which will incorrectly put markdown in functions:

```c
some_type function_name() {
*/ This is invalid in `source.c`
kAudioUnitSubType_3DMixer // This is invalid in to `source.c.platform` (included in `source.c`)
# Wait, is this *markdown*?!
}
```

Scope of problem

$base is used in the following grammars: source.apache-config, source.c, source.c++, source.cuda-c++, source.d, source.dm, source.fortran, source.fortran.modern, source.gap, source.hc, source.js.objl, source.lsl, source.mathematica, source.netlinx, source.nut, source.objc, source.pawn, source.stan, source.webidl, source.xc, text.html.mako, text.tex, text.tex.latex

Only the ones that are themselves used are affected. And of those, only when they can be used in an unexpected chain. These are the grammars that are used:

• source.c (used in objdump.x86asm, source.c++, source.c.ec, source.c.nwscript, source.crystal, source.gfm, source.lean.markdown, source.lua, source.nesc, source.objc, source.ruby, source.win32-messages, source.xc, text.html.markdown.source.gfm.apib, text.html.markdown.source.gfm.mson, text.restructuredtext)
• source.c++ (used in objdump.x86asm, source.crystal, source.cuda-c++, source.gfm, source.julia, source.lean.markdown, source.lex, source.logos, source.objc++, source.ruby, source.yacc, text.html.markdown.source.gfm.apib, text.html.markdown.source.gfm.mson, text.restructuredtext)
• source.netlinx (used in source.netlinx.erb)
• source.objc (used in source.gfm, source.lean.markdown, source.logos, source.objc++, text.html.markdown.source.gfm.apib, text.html.markdown.source.gfm.mson, text.restructuredtext)
• text.tex (used in source.pic, text.html.mediawiki, text.restructuredtext, text.tex.latex)
• text.tex.latex (used in text.muse, text.restructuredtext, text.tex.latex.haskell, text.texinfo)

Proposed solution

I think, to fix this automatically, you’d need to:

a) In grammars that aren’t used by others, replace $base with $self (but there’s no bug)
b) In grammars that are used by others, inline their dependencies, replacing $base with $self

It might also be good to report a warning when a grammar uses $base.
And/or to look for alternative grammars?

[lildude]

This is a long way outside my wheel house so I'm going to rope in @Alhadis to help with this.

That said, it looks like this is a very old known issue with TextMate and its grammars and an ex-GitHub employee and ex-maintainer of Linguist even submitted a fix for it in 2014: textmate/textmate#1276

This also affects VSCode: microsoft/vscode-textmate#117

As this is a known upstream issue, I'm not sure this is something we should be attempting to fix, though it's possibly something we could consider if we go ahead with @Alhadis's patching suggestion.

[Alhadis]

$base is essentially $self, but with different semantics when it comes to resolving injection targets (i.e., the recipient of an injection vs. the injection's grammar proper). However, there are serious portability hassles with both, as different implementations of TextMate's grammar system have disagreed on:

1. Whether an injection within an injection refers to the first or second injection's grammar,
2. Whether nested injections are supported at all, and
3. Whether $base is supported at all,

Because of this bedlam, I make a habit of using a single, named entry into a grammar (usually named #main), which simplifies recursive references to the grammar's root patterns-list. The top-level patterns: array will only contain patterns that shouldn't ever be matched more than once (e.g., hashbangs), and a reference to the #main rule (or whatever the hell I decided to name it at the time…)

Oh yeah, and GitHub imposes origin-based restrictions preventing one package's injections from affecting another's (which could otherwise lead to an orgy of messy highlighting issues that're impossible for GitHub staff to debug…)

[wooorm]

It looks like VSCode references a more up to date latex repo: https://github.com/jlelong/vscode-latex-basics. But it still uses base. but it might be easier to get changes into it?
From a quick search, VS Code references https://github.com/jeff-hykin/better-c-syntax, which does not include $base.
So maybe these people can be pinged/PRs would be accepted?

[Alhadis]

Yeah, when in doubt: avoid the $base and $self variables altogether in favour of a well-structured grammar with explicitly-named patterns (that is, don't stuff all the important patterns into the top-level patterns: array).