Welcome to csly Discussions!

[b3b00]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[matthiasbannach]

Hey!

Is it possible to switch the Lexer during the parsing process. E.g. switch to a different set of tokens once a specific Parser element is reached? (e.g. Language 2 inside Language 1)

I also noted that using
var lexed = lexer.Tokenize(line);
lineNumber++;
if (lexed.IsError)
{
throw new AssertFailedException("Lexing failed.");
}
foreach (var token in lexed.Tokens)
{
Debug.WriteLine($"[{lineNumber}]: {token.TokenID}:{((int)token.TokenID)} = {token.Value} ( {token.Position} )");
}
The lexer processes always the entire line and throws an exception before it reaches a place that is cannot tokenize.

e.g. Line = "var value =1; note ---- this is not workint"
|---good ----|^^^^^^ BAD
I would expect to get all good tokens till the bad one is reached, rather than having an error.

But maybe I interpret it wrong.

[b3b00]

Hello @matthiasbannach ,

First :
It is not possible to change the lexerduring parse as you state it. Lexing and parsing are strictly sequential processes.
But you might find the Generic Lexer modes feature usefull. The idea is to switch the token set when a specific token is found in the token stream.

Second :
Indeed the lexer always try to tokenize all the source code. If it fails it stops with an error stating where it failed.
it could be possible to return the tokens already found along the error. I will look at it.

it the meantime you could workaround this with the following process :

1. tokenize the full source code : it returns an error with its position
2. extract the begining of the string up to the error position.
3. tokenize the new string, it should succeed.

I hope I 've help.

Olivier

[b3b00]

@matthiasbannach, I've published a new release 2.9.2.2. lexers now return found tokens as long as the error that stopped the lexing process.

[b3b00]

@matthiasbannach , are you ok with my answers ? do you need more explanations ? have you found another way ?
please answer.

Olivier

[matthiasbannach]

Olivier, Funny – as this email arrived I was just responding to your previous mail. Changing the token set is a solution to the problem. I am currently experimenting with the solution. I do have one suggestion – rather dramatic, but I think this would make the parser hugely more usable: # Typing The Parser should be require a single type to be provided by any production. ```csharp var builder = new ParserBuilder<IpTablesToken, Models.IModel>(); // ............................................^^^^^^^^^^^^^ Remove or equate to return type of 'root' var buildResult = builder.BuildParser(parserInstance, ParserType.EBNF_LL_RECURSIVE_DESCENT, "configuration"); // ..........................................................................................^^^^^^^^^ Use type of this production ``` With this each production can have any return type as long as case like `[ a | b ... ]` the return type of the `a,b,...` production have a commond type (or common derived type). Let me know what you think? Matthias From: Olivier Duhart ***@***.***> Sent: Freitag, 25. November 2022 08:18 To: b3b00/csly ***@***.***> Cc: matthiasbannach ***@***.***>; Mention ***@***.***> Subject: Re: [b3b00/csly] Welcome to csly Discussions! (Discussion #276) @matthiasbannach <https://github.com/matthiasbannach> , are you ok with my answers ? do you need more explanations ? have you found another way ? please answer. Olivier — Reply to this email directly, <#276 (reply in thread)> view it on GitHub, or <https://github.com/notifications/unsubscribe-auth/ASNFY3ONM2WPR22WDUOKP5LWKBRZNANCNFSM5QUV6GFA> unsubscribe. You are receiving this because you were mentioned. <https://github.com/notifications/beacon/ASNFY3ITVWMDOAJIIYM6RPDWKBRZNA5CNFSM5QUV6GFKYY3PNVWWK3TUL52HS4DFWFCGS43DOVZXG2LPNZBW63LNMVXHJKTDN5WW2ZLOORPWSZGOABAJ35Y.gif> Message ID: < ***@***.***> ***@***.***>

[b3b00]

Thanks for your reply @matthiasbannach , i do not understand you proposal. Could you elaborate or give an example of parser that you imagine ?

I don't see what's the purpose of your idea.

anyway thanks for contributing

Olivier

[matthiasbannach]

Thanks Olivier - Will do.On Nov 26, 2022, at 17:12, Olivier Duhart ***@***.***> wrote: Thanks for your reply @matthiasbannach , i do not understand you proposal. Could you elaborate or give an example of parser that you imagine ? I don't see what's the purpose of your idea. anyway thanks for contributing Olivier —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[matthiasbannach]

I have encouterer another challenge:

Using these tokens

        [Lexeme(GenericToken.Identifier, IdentifierType.Custom, "a-z", "_a-zA-Z")]
        LOWER_START,

        [Lexeme(GenericToken.Identifier, IdentifierType.Custom, "A-Z", "_a-zA-Z")]
        UPPER_START,

        [Lexeme(GenericToken.Identifier, IdentifierType.Custom, "0-9", "_a-zA-Z")]
        NUMBER_START,

        [Lexeme(GenericToken.Identifier, IdentifierType.Alpha)]
        ALPHA,

Is it possible that I got the definition wrong? I get two ALPHA tokens and none of the preceding. When I take out the last (ALPHA) lexer definition it fails at the first character.

[b3b00]

hello, i m not sure it is quite possible to mix many identifier patterns in the same lexer, I will look at it soon.

[b3b00]

Indeed many identifier patterns do not work in the same lexer.
Looking at the generated FSM we've got this :

Which obviously can not tokenize the 12NUMBER idnetifier.

In fact when I've never thought that many identifier patterns could be of interest. I am not sure it could be that easy to allow it now the generic lexer is fully done. There may be a workaround for your case using :

• generic lexer extensions.
• lexer post processing : I guess this would be a better match.

Could you explain your use case so i could guide you ? i will also look at a way to allow many identifiers in the generic lexer but that might take a long time.

One last way is to use the regex lexer instead of the generic lexer at the cost of serious performance degradation.