Added Handler for Semantic Tokenization

[justinytchen]

Description
Fixes PowerShell/vscode-powershell#2813

Added a handler that utilizes the Parser in System.Management.Automation.Language to get PowerShell tokens and convert them to the semantic tokens that VS Code expects.

To use this, add "semanticHighlighting": true in the theme.json file.
Make sure a call is made to languageServerClient.registerProposedFeatures() in session.ts (vscode-powershell)
See PowerShell/vscode-powershell#2834

Before:

After:

Some issues this helps address:
PowerShell/EditorSyntax#143
PowerShell/EditorSyntax#157
PowerShell/EditorSyntax#167
PowerShell/EditorSyntax#191
PowerShell/EditorSyntax#198
PowerShell/EditorSyntax#199
PowerShell/EditorSyntax#200
PowerShell/EditorSyntax#201

[rjmholt]

Given that this is quite a nice, standalone service, we probably should add some e2e tests for this

[TylerLeonhardt]

Looking good - I had a few more comments. Don't forget that curly braces start on their own line:

public void foo()
{
}

instead of

public void foo() {
}

Also gave a bit of context in @rjmholt's comments

[rjmholt]

Suggested change

	//SemanticTokensHandler is labeled "Obsolete" because that is how Omnisharp marks proposed LSP features. Since we want this proposed feature, we disable this warning.
	// SemanticTokensHandler is labeled "Obsolete" because that is how Omnisharp marks proposed LSP features. Since we want this proposed feature, we disable this warning.

[rjmholt]

A question here: we already compile with warnings today, and while that's not a good thing, this particular warning should eventually disappear.

If we keep the pragma we suppress a warning that should eventually either go away, or which we should pay attention to since it will indicate that the proposed API hasn't stabilised. When it is stabilised, we would need to remove the pragma, but we won't know because of the pragma, meaning it may hang around in the code indefinitely (like so many fxcop suppressions).

Instead I think it probably makes sense for us to see the warning and not use a suppression pragma

[rjmholt]

Suggested change

	internal class PsesSemanticTokens : SemanticTokensHandler
	internal class PsesSemanticTokensHandler : SemanticTokensHandler

[TylerLeonhardt]

Tokens*

[rjmholt]

(Fixed in my suggestion)

[TylerLeonhardt]

Non-blocking but just wanted to say that if you get rid of this ctor, you can do:

new SemanticToken
{
    Text = "asdf",
    Line = 1,
    Column = 2,
   ...
}

and that will work the same way.

We can defer that for now.

[TylerLeonhardt]

What's this?

[david-driscoll]

🥳