Random lexing error; native fallback fails

[DrTom]

We get (more or less) random lexing errors like the following:

features/guest_user.feature: Lexing error on line 4: ''

This is on C/MRI Ruby 1.9.3-p392, gherkin 2.11.6

I tried to fall back to the ruby lexer as suggested (e.g. in cucumber/gherkin#182), along the following:

Gherkin::I18n.class_eval do
  alias_method :orig_lexer, :lexer
    def lexer(listener, force_ruby=false)
    orig_lexer(listener,true)
  end
end

Then we will get a 'no lexer found exception':

No lexer was found for en (cannot load such file -- gherkin/lexer/en)

I am not concerned about the performance. Is there a way to make the native ruby lexer work?

[b4mboo]

I ran into the exact same issue. We had random / non deterministic lexing errors and when trying to force usage of the ruby lexer, there was no English lexer to be found.

Solution: You need to compile the lexer yourself.

The problem for me was that initially I didn't have ragel installed on my system, which is why compiling didn't work at first. So before trying to follow my steps, ensure that ragel -v returns something like this:

Ragel State Machine Compiler version 6.7 May 2011
Copyright (c) 2001-2009 by Adrian Thurston

Now for compiling your own lexer:

1. Go to the gherkin gem's directory. (e.g. ~/.rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/gherkin-2.12.0/)
2. Configure your .ruby-version. If you are running rbenv you can do this with the following command: rbenv local 1.9.3-p392
3. Install all necessary gems. Call: bundle or bundle install. Depending on what was installed and if you are running rbenv, another rbenv rehash might be necessary.
4. Compile the lexer. Assuming you want an English lexer this is the command: bundle exec rake compile:gherkin_lexer_en. For other languages you need to have the respective suffix.
5. Enjoy running cucumber with ruby lexer and hopefully without the random lexing errors.

[DrTom]

This indeed solves the lexing errors! Thanks a lot for the hints.

A plea to the gherkin people: this procedure complicates our CI setup considerably and I would appreciate it if I could remove it again. Would it be possible make gherkin work in plain ruby mode without those steps?

[b4mboo]

Here is a script that allows you to automate those steps for your CI: https://github.com/b4mboo/grulec

[aslakhellesoy]

I'll try to make this easier in the next release.

[alexspeller]

Any news on this? It seems a pretty critical bug as it makes cucumber entirely unreliable.

[emiltin]

i had this too, with mac 10.8.5, ruby 2.0.0p195 and cucumber gem 1.1.4. on some runs i would get the lexing error, on others not - with no change in the file or anything else. very frustrating.

the involved files have a @ tag before the feature definition:

@routing @726
Feature: Avoid weird loops caused by rounding errors

after upgrading the gem to 1.3.8 i haven't seen it.. yet.

[jcwilk]

Thank you, turned out that I didn't have ragel installed, though seems strange that the compiler wouldn't abort at that point if it seems to be very much so required. Installed ragel, uninstalled and reinstalled gherkin and it seems to be working fine now.

[caseyding]

@jcwilk Could you tell me how I can install ragel on windows 7?

[jcwilk]

@caseyding I tend to avoid using ruby in windows whenever possible (I run a linux VM in windows) and my issue was in ubuntu, so I can't really help there or say whether it might be the same issue as I don't want to just make a guess and lead you down the wrong road.

It sounds like if you have cucumber/gherkin on recent versions the issue probably isn't still present by now since this was two years ago?

If you suspect it is, you might want to tag some of the others who actually know what they're doing here :P But I suspect it's something else unless you absolutely have to use an old version (again, mostly guessing)

[aslakhellesoy]

Gherkin2 will soon be replaced by Gherkin3 and we don't plan to make any more Gherkin2 releases.

[marciok]

I'm still facing this issue with cucumber 2.0.2, The scenario is falling randomly.

Is there a way to use Gherkin3 with cucumber 2.0.2 ?

[brasmusson]

@marciok If you are willing to use not yet released versions, you can follow the instructions in: cucumber/gherkin#257 (comment).

[marciok]

Awesome @brasmusson, thanks!

[brasmusson]

@marciok Cucumber 2.1 (using Gherkin3) was just released so there is no need for using gems from github anymore.

[marciok]

@brasmusson Thanks again! 🎉🎊