Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve startup time (regression) #525

Open
Tracked by #447
sharkdp opened this issue Aug 9, 2024 · 3 comments
Open
Tracked by #447

Improve startup time (regression) #525

sharkdp opened this issue Aug 9, 2024 · 3 comments

Comments

@sharkdp
Copy link
Owner

sharkdp commented Aug 9, 2024

The new type checker from #443 introduced a severe regression in startup times:

hyperfine --warmup 3 --reference "./numbat-master -e '1+2'" "./numbat-1.12 -e '1+2'" --export-markdown -
Command Mean [ms] Min [ms] Max [ms] Relative
./numbat-master -e '1+2' 78.9 ± 1.2 77.1 83.8 2.28 ± 0.09
./numbat-1.12 -e '1+2' 34.5 ± 1.2 32.4 42.5 1.00

We should look into potential measures to improve this again.

If the constraint checker is the culprit, we might want to look into de-duplication of constraints before attempting to solve them. But it might be something else entirely (e.g. the mutation/traversal/cloning of ASTs during elaboration)

@sharkdp sharkdp changed the title Fix performance regression (startup time benchmark) Improve startup time (regression) Aug 9, 2024
@sharkdp
Copy link
Owner Author

sharkdp commented Aug 19, 2024

Thanks to the changes in #543 by @GuerricChupin, we've moved a significant step in the right direction:

Command Mean [ms] Min [ms] Max [ms] Relative
./numbat-master -e '1+2' 59.9 ± 1.2 57.9 64.3 1.78 ± 0.08
./numbat-1.12 -e '1+2' 33.7 ± 1.3 32.6 44.8 1.00

@sharkdp
Copy link
Owner Author

sharkdp commented Aug 29, 2024

Thanks to the changes in #537 by @irevoire (which are not related to the type checker, but still), we've moved another significant step in the right direction:

Command Mean [ms] Min [ms] Max [ms] Relative
./numbat-master -e '1+2' 47.0 ± 0.9 45.5 50.3 1.37 ± 0.04
./numbat-1.12 -e '1+2' 34.2 ± 0.9 32.2 36.7 1.00

@sharkdp
Copy link
Owner Author

sharkdp commented Oct 2, 2024

Thanks to various changes by @rben01 (#559, #569, #578, #579, #585, #597, #598 ), there is another 5% improvement in startup time. For more details, see #579 (comment) (note: the absolute times are not comparable. I measured on different machines).

Also, note that some of the measurements above might be underestimating the true performance benefit, since Numbats prelude is growing. For a true comparison, we need to run both versions against the same version of the prelude (typically the older one, as the new one might not be supported on the old interpreter).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant