Implement parser for detecting Autometrics annotations

[arendjr]

Currently the Python implementation uses a regular expression to detect Autometrics annotations. In order to implement Rust support, this approach would not be sufficient, since Rust can annotate entire impl blocks with a single annotation, which would be too complex to parse with a regex.

To solve this, we need to implement a proper parsing solution in the extension. Something like Tree-sitter might be an option, since it supports all the languages we want to support as well, but other options may be fine too.

[arendjr]

I'm doing a little bit of research into how feasible Tree-sitter might be, but I do have two concerns.

• Using the incremental parsing seems to be off the table for us, because we don't know which parts of the document changed between one invocation of the hover provider and the next. This means we'll have to reparse the entire document.
• Parsing the document is already done by the language server, so doing it ourselves does seem to be doubling the work. Looking at our TypeScript implementation, it already hooks into the language server, so using a Tree-sitter-approach would be silly for that language.

As such, I'm thinking we should consider per language whether it's feasible to make use of the (dominant) language server for that language first. If that's not feasible somehow, we can always fall back to using Tree-sitter if necessary. In the case of Python, however, where the regex-based approach is probably fine, I would be hesitant spending effort rewriting that.

The main question then becomes what to do with Rust: A regex-based approach won't cut it there, but Rust Analyzer also doesn't seem to support plugins the way the TypeScript language server does. I'll first do some more research to see if we can hook into Rust Analyzer functionality somehow, before committing one way or another.

[emschwartz]

VS Code's docs provide an example of having one language server extension hijack requests of a certain type and forward some of those requests to the underlying language server. It seems like we should be able to use this approach to have an autometrics language server that intercepts requests, forwards them as necessary, and injects additional stuff into the response when it's appropriate https://code.visualstudio.com/api/language-extensions/embedded-languages#request-forwarding

[emschwartz]

Also, I really think we should try to use the LSP for as much of this as we can. The Language Server Protocol was designed to allow additional information to be added into the editor in a cross-editor way. If we can figure out how to leverage it, hack it, or stretch the bounds of what it was meant for, it seems like we'd be able to build something that would work across editors. This is important, as we've already gotten a request for Vim support and other people will want support for other editors.

[flenter]

I agree that an LSP approach might be the long term solution for several languages. Though not for all languages, for instance with typescript we can communicate with typescript server and get information about the current type through there. We should evaluate for each language which approach would work best.