Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parser rewrite #274

Merged
merged 156 commits into from
Jan 20, 2024
Merged
Changes from 1 commit
Commits
Show all changes
156 commits
Select commit Hold shift + click to select a range
7464a02
Start work on eliminating code and moving stuff out
Kampfkarren Apr 2, 2023
086f247
Tests compile
Kampfkarren Apr 2, 2023
83b69e9
Initial tokenizer to pass anonymous-functions-1
Kampfkarren Apr 3, 2023
97e4666
Saner APIs
Kampfkarren Apr 3, 2023
6710796
Decent parsing, TokenReference
Kampfkarren Apr 4, 2023
1b0c946
Start work on expressions (I need to merge)
Kampfkarren Apr 4, 2023
3c0c51c
anonymous-functions-1 passes
Kampfkarren Apr 5, 2023
87bd000
anonymous-functions-2 passing
Kampfkarren Apr 5, 2023
672d3bd
anonymous-functions-3 pass
Kampfkarren Apr 5, 2023
2e20295
anonymous-functions-4
Kampfkarren Apr 5, 2023
e3a5b3e
assignment-1
Kampfkarren Apr 5, 2023
6268bfe
assignment-2
Kampfkarren Apr 5, 2023
ed63d2a
assignment-3 mini boss
Kampfkarren Apr 5, 2023
3825f2e
binops (call-1 and call-2 just worked)
Kampfkarren Apr 5, 2023
c79ea97
do, empty
Kampfkarren Apr 5, 2023
cd8b08e
exponents
Kampfkarren Apr 5, 2023
2df64c0
fractional-numbers
Kampfkarren Apr 5, 2023
1bf2c90
function-declaration-1
Kampfkarren Apr 5, 2023
788e687
function-declaration-2
Kampfkarren Apr 6, 2023
9ff34b9
function-shortcuts
Kampfkarren Apr 6, 2023
bc2dbef
generic-for-loop-1, free: generic-for-loop-2, goto-as-identifier, gt-…
Kampfkarren Apr 7, 2023
d77eec0
if-1, and a bunch of free ones
Kampfkarren Apr 7, 2023
cba1b4d
local-function-1 (2 for free)
Kampfkarren Apr 7, 2023
b62e651
multi-line-comments-1 through 4
Kampfkarren Apr 7, 2023
a82c45f
5 and 6
Kampfkarren Apr 7, 2023
cbdd8f5
7, 8, 9
Kampfkarren Apr 7, 2023
d4adf09
multi-line-string 1-5
Kampfkarren Apr 7, 2023
ae6fd70
Multi-line-strings, negative numbers
Kampfkarren Apr 7, 2023
ba4689e
Numeric for loop
Kampfkarren Apr 7, 2023
09bfddd
Paren expressions
Kampfkarren Apr 7, 2023
fb830f6
repeat-until
Kampfkarren Apr 7, 2023
07dfe39
return-break
Kampfkarren Apr 7, 2023
3da5e9d
semicolons-1, semicolons-2
Kampfkarren Apr 7, 2023
5dd8698
Shebangs, but I changed the behavior to make sense
Kampfkarren Apr 7, 2023
26a4d81
single-line-comment-4 to 5
Kampfkarren Apr 7, 2023
5662b80
Fix bug with character counts
Kampfkarren Apr 7, 2023
a026a1e
A lot
Kampfkarren Apr 7, 2023
46d5ec8
unops, utf-8 for free
Kampfkarren Apr 9, 2023
3d4bf88
while loops
Kampfkarren Apr 9, 2023
10b3769
Fix some regressions when parsing Zombie Strike
Kampfkarren Apr 9, 2023
d925b08
Hexadecimals
Kampfkarren Apr 9, 2023
fc8697f
Better APIs
Kampfkarren Apr 9, 2023
0626512
TokenReference::symbol
Kampfkarren Apr 9, 2023
a9ca905
Start work on fail cases
Kampfkarren Apr 9, 2023
c641526
Keep parsing blocks with unexpected token
Kampfkarren Apr 9, 2023
19cd8db
Some cargo insta test up to a point
Kampfkarren Apr 9, 2023
e967180
I'm ok with these
Kampfkarren Apr 9, 2023
194a145
Up to do-1
Kampfkarren Apr 9, 2023
a7e212d
Every result so far is fine
Kampfkarren Apr 9, 2023
ff19bdb
Start work on more specific function argument parsing
Kampfkarren Apr 10, 2023
c5b378b
Snaps after fixing a panic
Kampfkarren Apr 10, 2023
d76e5e4
Some more passes
Kampfkarren Apr 10, 2023
34004c7
Everything is Okay
Kampfkarren Apr 10, 2023
d5642f9
Some more snaps
Kampfkarren Apr 10, 2023
702ce8f
Some more snaps
Kampfkarren Apr 10, 2023
5847c25
Everything's fine
Kampfkarren Apr 10, 2023
8439276
We can parse every fail case now?
Kampfkarren Apr 10, 2023
a9bbbb9
It's fine
Kampfkarren Apr 10, 2023
2c9e872
Reduce errors until the next stmt
Kampfkarren Apr 10, 2023
4755639
Support recovered tokens
Kampfkarren Apr 10, 2023
6b65557
Remove errors, to be replaced
Kampfkarren Apr 10, 2023
b430b76
Start work on recovered tokens
Kampfkarren Apr 10, 2023
3902515
Unclosed comments
Kampfkarren Apr 11, 2023
fa6a63b
Shebang is weird, but the rest of these are nice
Kampfkarren Apr 11, 2023
c9e6810
Rewrite todo
Kampfkarren Apr 11, 2023
6bd26f3
Initial fuzzers
Kampfkarren Apr 11, 2023
e3ec709
Start cleanup of current().unwrap()
Kampfkarren Apr 11, 2023
bcdebbc
Clean up all BAD uses of current().unwrap()
Kampfkarren Apr 11, 2023
a377528
Quick clippy
Kampfkarren Apr 11, 2023
5655952
Bad numbers
Kampfkarren Apr 11, 2023
090dd38
Fix multiple decimals
Kampfkarren Apr 11, 2023
b02f113
Error with invalid 0x
Kampfkarren Apr 11, 2023
18735cc
This came up with fuzzing
Kampfkarren Apr 11, 2023
e5d03f6
I think this is also an error case
Kampfkarren Apr 11, 2023
06572d3
push_punctuated
Kampfkarren Apr 11, 2023
a4d1b0e
clippy
Kampfkarren Apr 11, 2023
9aadc94
Remove outdated todo
Kampfkarren Apr 11, 2023
d1683be
Update benchmarks
Kampfkarren Apr 11, 2023
c95ca22
Remove todo
Kampfkarren Apr 11, 2023
4a20a9b
Add recovery test, but I want to add coverage testing
Kampfkarren Apr 11, 2023
87f6816
Sanity assert
Kampfkarren Apr 12, 2023
bcfe3fc
Cool, it just works
Kampfkarren Apr 12, 2023
8b7d16c
This doesn't need to use a macro
Kampfkarren Apr 12, 2023
df82b0d
Remove UnOp::precedence
Kampfkarren Apr 12, 2023
bdb522b
Clean up BinOp
Kampfkarren Apr 12, 2023
3cb9410
We didn't
Kampfkarren Apr 12, 2023
c48a306
We changed this
Kampfkarren Apr 12, 2023
3cc6cd1
I can't get this to repro anyway
Kampfkarren Apr 12, 2023
4b47711
New range
Kampfkarren Apr 12, 2023
d0905de
Update ranges (but I didn't really look at them =/)
Kampfkarren Apr 12, 2023
8fab086
Test Recovered coverage
Kampfkarren Apr 12, 2023
beb8f41
Add optional range
Kampfkarren Apr 13, 2023
fb46da3
Add range field
Kampfkarren Apr 13, 2023
75423b9
Start with ranges
Kampfkarren Apr 13, 2023
b9a7bd5
Error displays as part of test
Kampfkarren Apr 13, 2023
a6a81d4
Better error message
Kampfkarren Apr 13, 2023
c1406a3
Better error messages for blocks
Kampfkarren Apr 13, 2023
1ab3909
Better table key errors
Kampfkarren Apr 16, 2023
e5fa0e4
Change parse signature
Kampfkarren Apr 16, 2023
4d1148d
Some fixes with new result type, some last stmt test. I don't love it
Kampfkarren Apr 16, 2023
ccec566
Use new helper
Kampfkarren Apr 16, 2023
eb14dab
Clean some todos
Kampfkarren Apr 16, 2023
6d69027
Remove unused
Kampfkarren Apr 16, 2023
2001be3
It doesn't
Kampfkarren Apr 16, 2023
6951dca
I think it's fine
Kampfkarren Apr 16, 2023
a45f992
Better expect_local_assignment structure
Kampfkarren Apr 16, 2023
d85d40d
Add LuaVersion field
Kampfkarren Apr 17, 2023
53be460
Luau behind feature flag is almost certainly better
Kampfkarren Apr 17, 2023
aefbf85
Make ParserState a bitfield
Kampfkarren Apr 17, 2023
949d2f7
Fix tests
Kampfkarren Apr 17, 2023
35880b3
Fix doc tests
Kampfkarren Apr 17, 2023
2ed8b94
Start work on Lua 5.2 parsing (none of these are right)
Kampfkarren Apr 22, 2023
e1a38a0
Symbol macro
Kampfkarren Apr 22, 2023
cded960
Allow unused in the symbol macro
Kampfkarren Apr 22, 2023
4aac101
goto
Kampfkarren Apr 22, 2023
ff0c0da
Numbers, working on the other stuff
Kampfkarren Apr 23, 2023
c43ece9
Z escaping
Kampfkarren Apr 23, 2023
33a64c6
Lua 5.3
Kampfkarren Apr 24, 2023
0e42181
Add Symbol::from_str test
Kampfkarren Apr 26, 2023
913e572
Fix parsing not using the correct version in tests
Kampfkarren Apr 26, 2023
0aeb645
Lua 5.4
Kampfkarren Apr 26, 2023
b953111
Fix 5.1 compile
Kampfkarren Apr 26, 2023
d42cfb5
Fix doctest without 5.1 feature
Kampfkarren Apr 26, 2023
4132187
Clean version checking macro
Kampfkarren Apr 26, 2023
712dc36
Add Lua 5.4 to default set, and I can't think of anything
Kampfkarren Apr 26, 2023
589901b
Restore ast tests
Kampfkarren Apr 26, 2023
ba5f364
Flatten AstError, since the other variants are no longer reachable
Kampfkarren Apr 26, 2023
0325eeb
Remove optional for additional
Kampfkarren Apr 26, 2023
2e6b06c
Some trivial changelogs
Kampfkarren Apr 26, 2023
bb511ea
Less trivial changelogs
Kampfkarren Apr 26, 2023
1f61ef5
parse_fallible documentation
Kampfkarren Apr 26, 2023
b4cc4ad
Handle spaces with newline whitespace position correctly
JohnnyMorganz Oct 15, 2023
544f464
Luau Parser rewrite (#276)
JohnnyMorganz Jan 7, 2024
52e61f4
Merge branch 'main' of https://github.com/Kampfkarren/full-moon into …
JohnnyMorganz Jan 7, 2024
c9cd219
Support Luau floor division + assignment (`//` and `//=`)
JohnnyMorganz Jan 7, 2024
969c448
luau: consider trivia when testing string interpolation double brace …
JohnnyMorganz Jan 7, 2024
b66b9c0
luau: support \z escape in strings (#287)
JohnnyMorganz Jan 7, 2024
7b25ffb
fix some clippy lints
JohnnyMorganz Jan 7, 2024
624d867
rename `feature = "roblox"` -> `feature = "luau"`
JohnnyMorganz Jan 7, 2024
d50a8da
Switch `multi_line` from `Option<usize>` to `usize` (#294)
JohnnyMorganz Jan 20, 2024
a83708c
CHANGELOG
Kampfkarren Jan 20, 2024
158c6cf
More changelog fixing
Kampfkarren Jan 20, 2024
0a0b6a9
Add changelog about comments
Kampfkarren Jan 20, 2024
6085060
Rename parse_ to expect_
Kampfkarren Jan 20, 2024
035a2e6
Make these not public
Kampfkarren Jan 20, 2024
29ed525
Remove UnexpectedShebang
Kampfkarren Jan 20, 2024
e409e19
InvalidNumber documentation
Kampfkarren Jan 20, 2024
734214b
I don't really have time to test this
Kampfkarren Jan 20, 2024
7405e83
This is fine
Kampfkarren Jan 20, 2024
e60bc5a
This is very rarely hit, and there's bigger problems here
Kampfkarren Jan 20, 2024
bbc1545
Too niche, not worth it
Kampfkarren Jan 20, 2024
9fa7819
I think this was implemented by Johnny
Kampfkarren Jan 20, 2024
6a12257
Sure
Kampfkarren Jan 20, 2024
2d13175
Docs
Kampfkarren Jan 20, 2024
838461b
Fix clippy with no features
Kampfkarren Jan 20, 2024
df8a981
Only have doc comment with lua 5.2
Kampfkarren Jan 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Start work on Lua 5.2 parsing (none of these are right)
Kampfkarren committed Apr 22, 2023
commit 2ed8b94abc6ef0180d71fe44e412c286632eba79
21 changes: 21 additions & 0 deletions full-moon/src/ast/parser_util.rs
Original file line number Diff line number Diff line change
@@ -15,3 +15,24 @@ macro_rules! try_parser {
}
};
}

#[doc(hidden)]
#[macro_export]
macro_rules! version_switch {
($lua_version:expr, {
$(
$($version:ident)|* => $value:expr
)+
}) => {
paste::paste! {
$(
$(
#[cfg(feature = "" $version)]
if $lua_version.[<has_ $version>]() {
$value
}
)*
)+
}
};
}
22 changes: 21 additions & 1 deletion full-moon/src/ast/versions.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// Lua version is handled as a bitfield to support parsing as many languages as possible at once.
// Any new language added does not necessarily need to (or should be) added to the default set.
const VERSION_LUAU: u8 = 1 << 0;
const VERSION_LUA52: u8 = 1 << 1;

#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
pub struct LuaVersion {
@@ -28,12 +29,31 @@ impl LuaVersion {
pub fn has_luau(self) -> bool {
self.bitfield & VERSION_LUAU != 0
}

#[cfg(feature = "lua52")]
pub fn lua52() -> Self {
Self {
bitfield: VERSION_LUA52,
}
}

#[cfg(feature = "lua52")]
pub fn with_lua52(self) -> Self {
Self {
bitfield: self.bitfield | VERSION_LUA52,
}
}

#[cfg(feature = "lua52")]
pub fn has_lua52(self) -> bool {
self.bitfield & VERSION_LUA52 != 0
}
}

impl Default for LuaVersion {
fn default() -> Self {
Self {
bitfield: VERSION_LUAU,
bitfield: VERSION_LUAU | VERSION_LUA52,
}
}
}
30 changes: 23 additions & 7 deletions full-moon/src/tokenizer/lexer.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
use crate::{ast::LuaVersion, tokenizer::StringLiteralQuoteType, ShortString};
use crate::{ast::LuaVersion, tokenizer::StringLiteralQuoteType, version_switch, ShortString};

use super::{
Position, Symbol, Token, TokenReference, TokenType, TokenizerError, TokenizerErrorType,
@@ -442,12 +442,28 @@ impl Lexer {
},
),

':' => self.create(
start_position,
TokenType::Symbol {
symbol: Symbol::Colon,
},
),
':' => {
version_switch!(self.lua_version, {
lua52 | luau => {
if self.source.current() == Some(':') {
self.source.next();
return self.create(
start_position,
TokenType::Symbol {
symbol: Symbol::TwoColons,
},
);
}
}
});

self.create(
start_position,
TokenType::Symbol {
symbol: Symbol::Colon,
},
)
}

',' => self.create(
start_position,
106 changes: 106 additions & 0 deletions full-moon/tests/cases/pass/goto-as-identifier/tokens.snap.new
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
source: full-moon/tests/pass_cases.rs
assertion_line: 40
expression: tokens
input_file: full-moon/tests/cases/pass/goto-as-identifier
---
- start_position:
bytes: 0
line: 1
character: 1
end_position:
bytes: 48
line: 1
character: 49
token_type:
type: SingleLineComment
comment: " goto as an identifier is permitted in lua 5.1"
- start_position:
bytes: 48
line: 1
character: 49
end_position:
bytes: 49
line: 1
character: 49
token_type:
type: Whitespace
characters: "\n"
- start_position:
bytes: 49
line: 2
character: 1
end_position:
bytes: 53
line: 2
character: 5
token_type:
type: Identifier
identifier: self
- start_position:
bytes: 53
line: 2
character: 5
end_position:
bytes: 54
line: 2
character: 6
token_type:
type: Symbol
symbol: "."
- start_position:
bytes: 54
line: 2
character: 6
end_position:
bytes: 58
line: 2
character: 10
token_type:
type: Symbol
symbol: goto
- start_position:
bytes: 58
line: 2
character: 10
end_position:
bytes: 59
line: 2
character: 11
token_type:
type: Symbol
symbol: (
- start_position:
bytes: 59
line: 2
character: 11
end_position:
bytes: 64
line: 2
character: 16
token_type:
type: StringLiteral
literal: foo
quote_type: Double
- start_position:
bytes: 64
line: 2
character: 16
end_position:
bytes: 65
line: 2
character: 17
token_type:
type: Symbol
symbol: )
- start_position:
bytes: 65
line: 2
character: 17
end_position:
bytes: 65
line: 2
character: 17
token_type:
type: Eof

32 changes: 11 additions & 21 deletions full-moon/tests/fail_cases.rs
Original file line number Diff line number Diff line change
@@ -59,20 +59,24 @@ fn process_codespan_display(source: &str, result: &AstResult) {
);
}

#[test]
#[cfg_attr(feature = "no-source-tests", ignore)]
fn test_parser_fail_cases() {
run_test_folder("./tests/cases/fail/parser", |path| {
fn run_parser_fail_cases(folder: &str, lua_version: LuaVersion) {
run_test_folder(folder, |path| {
let source = fs::read_to_string(path.join("source.lua")).expect("couldn't read source.lua");
let tokens = tokenizer::Lexer::new(&source, LuaVersion::lua51())
let tokens = tokenizer::Lexer::new(&source, lua_version)
.collect()
.unwrap();
assert_yaml_snapshot!("tokens", tokens);

process_fail_case(path, &source, LuaVersion::lua51());
process_fail_case(path, &source, lua_version);
});
}

#[test]
#[cfg_attr(feature = "no-source-tests", ignore)]
fn test_parser_fail_cases() {
run_parser_fail_cases("./tests/cases/fail/parser", LuaVersion::lua51());
}

#[test]
#[cfg_attr(feature = "no-source-tests", ignore)]
fn test_tokenizer_fail_cases() {
@@ -128,21 +132,7 @@ fn test_roblox_tokenizer_fail_cases() {
#[cfg(feature = "lua52")]
#[cfg_attr(feature = "no-source-tests", ignore)]
fn test_lua52_parser_fail_cases() {
run_test_folder("./tests/lua52_cases/fail/parser", |path| {
let source = fs::read_to_string(path.join("source.lua")).expect("couldn't read source.lua");

let tokens = tokenizer::tokens(&source).expect("couldn't tokenize");

assert_yaml_snapshot!("tokens", tokens);

match ast::Ast::from_tokens(tokens) {
Ok(_) => panic!("fail case passed for {:?}", path),
Err(error) => {
println!("error {:#?}", error);
assert_yaml_snapshot!("error", error);
}
}
})
run_parser_fail_cases("./tests/lua52_cases/fail/parser", LuaVersion::lua52());
}

#[test]
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
---
source: full-moon/tests/fail_cases.rs
expression: error
assertion_line: 23
expression: result.ast
input_file: full-moon/tests/lua52_cases/fail/parser/goto-1

---
UnexpectedToken:
nodes:
stmts: []
eof:
leading_trivia: []
token:
start_position:
bytes: 4
@@ -16,5 +19,5 @@ UnexpectedToken:
character: 5
token_type:
type: Eof
additional: "expected identifier after `goto`"
trailing_trivia: []

Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
source: full-moon/tests/fail_cases.rs
assertion_line: 28
expression: "full_moon::print(&ast)"
input_file: full-moon/tests/lua52_cases/fail/parser/goto-1
---
""

Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
source: full-moon/tests/fail_cases.rs
assertion_line: 56
expression: "String::from_utf8(output.into_inner()).unwrap()"
input_file: full-moon/tests/lua52_cases/fail/parser/goto-1
---
error[ast]: unexpected token, this needs to be a statement
┌─ source.lua:1:1
1 │ goto
│ ^^^^


22 changes: 22 additions & 0 deletions full-moon/tests/lua52_cases/fail/parser/goto-1/errors.snap.new
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
source: full-moon/tests/fail_cases.rs
assertion_line: 22
expression: result.errors
input_file: full-moon/tests/lua52_cases/fail/parser/goto-1
---
- AstError:
UnexpectedToken:
token:
start_position:
bytes: 0
line: 1
character: 1
end_position:
bytes: 4
line: 1
character: 5
token_type:
type: Symbol
symbol: goto
additional: "unexpected token, this needs to be a statement"

Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
source: full-moon/tests/fail_cases.rs
assertion_line: 23
expression: result.ast
input_file: full-moon/tests/lua52_cases/fail/parser/goto-as-identifier
---
nodes:
stmts: []
eof:
leading_trivia: []
token:
start_position:
bytes: 69
line: 2
character: 17
end_position:
bytes: 69
line: 2
character: 17
token_type:
type: Eof
trailing_trivia: []

Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
source: full-moon/tests/fail_cases.rs
assertion_line: 28
expression: "full_moon::print(&ast)"
input_file: full-moon/tests/lua52_cases/fail/parser/goto-as-identifier
---
""

This file was deleted.

Loading