NotAllInputParsedException : Redundant input, expecting EOF but found

[ixops]

I'm trying to create a wildcard token which captures everything that isn't part of my grammar. To illustrate, lets say I want to match Foo, Bar, and Baz specifically, and any text that isn't one of those, I want it to be captured with a wildcard token.

So given "Foo Bar Baz, This is a cow. ", the Lexer would return [] where wildcard would hold "This is a cow."

So given "Bar Baz Foo, This is a fish. ", the Lexer would return [] where wildcard would hold "This is a fish."

I've got a set of tokens that seem to match everything correctly, including (I think) the wildcard. They look like this...

matchFoo = createToken({ name: 'matchWildcard', pattern: 'Foo'});
matchBar = createToken({ name: 'matchBar', pattern: 'Bar' });
matchBaz = createToken({ name: 'matchBaz', pattern: 'Baz' });
matchWildcard = createToken({ name: 'matchWildcard', pattern: /[a-zA-Z',. ]+/ });

The issue I'm having is with the parsing rule. I'm trying a rule that looks like this...

$.RULE('notation', () => {
    $.AT_LEAST_ONE(() => {
        $.OR([
          { ALT: () => $.CONSUME(matchFoo) },
          { ALT: () => $.CONSUME(matchBar) }, 
          { ALT: () => $.CONSUME(matchBaz) },
          { ALT: () => $.CONSUME(matchWildcard) }
        ]);
    })
});

I "think" this should require at least one token but also consume all tokens, regardless of the order in which they appear. The problem is I'm getting the following error...

 Error {
   name: 'NotAllInputParsedException',
   message: 'Redundant input, expecting EOF but found: , This is a fish.',
   token: [Object],
   resyncedTokens: [],
   context: [Object] } 

I searched the docs, chat channel, and issue history, but haven't been able to unpack why I'm getting the error or how to fix it. I'm sure I'm failing to grasp something here.

[bd82]

Hello @ixops

A possibly related topic may be fuzzy matching:

• Fuzzy Parsing Example #993

Anyhow I think your code snippet should be working as you expect it to.
I suggest you inspect the tokenVector produced by the Lexer before it is passed to the parser and ensure it contains exactly four tokens as you expect.

You can also put a breakpoints just inside and after:

• $.AT_LEAST_ONE

and inspect $.LA(1)/this.LA(1) every iteration that enters and after exiting from the loop, perhaps this will provide some hints to the problem...

If these options fail to produce clues, please create a simple gist/repo I can checkout and reproduce the problem with and I will debug it myself.

Please note that Chevrotain performs Lexical Matching in Before Parsing in a separate phase. Which make your Token Names (match prefix) a bit misleading as no regExp matching is done inside the parser, that already happened during lexing...

Cheers.
😄

[bd82]

please re-open this if you can provide an easily re-producible example.