Context sensitive tokens

[keyvan-m-sadeghi]

Following the discussions on #993 and #1023, I'm wondering if it'd make sense to have an option for stateful tokens.

const stringStartAndEnd = createToken({
  effect: ({ insideString }) => ({ insideString: !insideString }),
  pattern: /'/
});

const literal = createToken({
  gate: ({ insideString }) => insideString,
  pattern: /[^']*/
});

For this to work, the lexer would be initiated with a state variable, an empty object at first.
If present, the effect function of a token is invoked whenever encountered during lexing:

state = { ...state, ...token.effect(state)) };

The gate property of a token is called with the state variable before a match.

@bd82 does this make any sense?

[HoldYourWaffle]

I'm not sure I completely understand what you're suggesting, but this looks very similar to multi-mode lexers to me.

[bd82]

Sounds like multi mode Lexing too.
You can implement even more complex logic during lexing using custom token patterns.

• https://sap.github.io/chevrotain/docs/guide/custom_token_patterns.html

[keyvan-m-sadeghi]

Thanks @HoldYourWaffle and @bd82, that's right. Sorry for being a noob ;)

[HoldYourWaffle]

Fellow noobs should help each other out right 😉

@bd82 Maybe it's a good idea to make multi-mode lexing more prevalent in the documentation? There isn't really a tutorial page for it (although the linked example is plenty to understand how it works) and I didn't find the page until you linked to it from another issue. Perhaps we could add a section to the lexer tutorial page that mentions it?

[bd82]

Perhaps we could add a section to the lexer tutorial page that mentions it?

It seems like this feature is more popular than I thought...

Having a full guide e.g:

• https://sap.github.io/chevrotain/docs/guide/generating_syntax_diagrams.html#examples

Would be best but that it a fair bit of work to create...

Perhaps (as suggested) a small note could be added in the lexer tutorial to aid discover-ability.

[HoldYourWaffle]

I'd love to add one, but the 'Edit on github' link results in a 404 😅

[bd82]

The move to mono repo structure broke the links.
Add packages/chevrotain to the link, e.g:
https://github.com/SAP/chevrotain/edit/master/packages/chevrotain/docs/guide/generating_syntax_diagrams.md

I will sort it out later.

[bd82]

The edit links are now fixed.

• 33788dd

But you may want to edit them on a local dev env due to enforcement of prettier formatting.

[HoldYourWaffle]

I created a pull request with an added note.

The move to mono repo structure broke the links.

I saw something about a 'transitioning to a mono repo' before, and I've always wondered: what kinds of chevrotain were in a different repository? The packages folder has just 1 subdirectory, and I can't think of anything easily "splittable" in this library.

[bd82]

I saw something about a 'transitioning to a mono repo' before, and I've always wondered: what kinds of chevrotain were in a different repository? The packages folder has just 1 subdirectory, and I can't think of anything easily "splittable" in this library.

• The examples directory includes multiple packages (private ones).
• The Lexer could at one point be split out.
• The Diagrams generation could also be split out.

[HoldYourWaffle]

That makes sense, thanks!