-
-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ebisu? #85
Comments
fasiha/ebisu.js#23 |
Ebisu author here 👋 fasiha/ebisu.js#23 has the discussion and links to the results. Ebisu v3 release candidate didn’t do well! For separate reasons, I’ve been working on alternatives to that version and have something I like more (see fasiha/ebisu#66) and I’m happy to support rerunning the benchmarks on that version. I am ashamed to admit it but I haven’t made time to properly understand how the benchmarks here work, and I haven’t made time to figure out how to run FSRS etc. on the benchmarks I personally use to compare Ebisu versions. Part of the reason is, Ebisu and its benchmarks handle not just binary quizzes but also binomial and noisy-binary and passive quizzes; and it’s not been obvious how to adapt those quiz styles to various other probabilistic SRS systems to ensure we’re doing apples-to-apples comparisons.
I know I should just wrap up working on Ebisu v3 and release it so folks can do benchmarks without being confused what version to run 😓 sorry! I’m hoping to release v3 this… year 🤞 |
We can benchmark any algorithm as long as it:
Also, Anki has 4 grades (answer buttons), so previously I suggested using different values of |
Oh, perfect! Thanks for sharing that thread. I like Ebisu’s approach in principle, still curious if its empirical deficits can be overcome. I like that its theory more directly handles issues like, say, the fact that if we’re targeting 90% retrievability, we should expect to miss 10% of items, even if their underlying stabilities are identical. Most algorithms handle that with an ad-hoc solution (eg FSRS’s low-pass back to default stability), and maybe that’s fine, but Bayesian stats seem like a better approach in principle (tho evidently perhaps not in practice!) I’ll leave this issue open since Ebisu is still not in the official benchmark results for this repo but feel free to close if you like, since I got what I wanted! :) |
The PR is here: #11. It doesn't perform well, so I haven't merged it. And the dataset has been updated, so the PR is outdated. If you're interested in the result, I would rerun the benchmark when I'm available. But I need to check whether my implementation is correct at first. It requires the help from @fasiha. |
@fasiha we're working on FSRS-5, and I will make another Reddit post about benchmarking, so if you are still interested, you can come back to implementing Ebisu in the benchmark. |
Thank you for this very interesting analysis! If you all feel inclined to include it, I'd be curious to see how Ebisu compares.
The text was updated successfully, but these errors were encountered: