Add references to lexer modes and custom token patterns to the lexer tutorial

[HoldYourWaffle]

As discussed in #1027.

I'm not completely satisfied with this text, but it serves its purpose I think.

[HoldYourWaffle]

I just noticed something: the tutorial page says the lexer is stateless, but how can it be stateless if it supports multi-mode lexing?

[bd82]

I just noticed something: the tutorial page says the lexer is stateless, but how can it be stateless if it supports multi-mode lexing?

Good catch 👍

Maybe it is more correct to say the tokenize method is a pure function.

[bd82]

2c52e88