From 700b8616201be611c23618527db680b6d28ea1d5 Mon Sep 17 00:00:00 2001 From: Niels Thykier Date: Fri, 17 May 2024 13:55:49 +0000 Subject: [PATCH] Speed up spellchecking by ignoring whitespace-only lines The new API has introduced extra overhead per line being spellchecked. One way of optimizing out this overhead, is to spellcheck fewer lines. An obvious choice here, is to optimize out empty and whitespace-only lines, since they will not have any typos at all (on account of not having any words). A side-effect of this change is that we now spellcheck lines with trailing whitespace stripped. Semantically, this gives the same result (per "whitespace never has typos"). Performance-wise, it is faster in theory because the strings are now shorter (since we were calling `.rstrip()` anyway). In pratice, I am not sure we are going to find any real corpus where the trailing whitespace is noteworthy from a performance point of view. On the performance corpus from #3491, this takes out ~0.4s of runtime brining us down to slightly above the 5.6s that made the baseline. --- codespell_lib/_codespell.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/codespell_lib/_codespell.py b/codespell_lib/_codespell.py index 8db03096108..693f078f321 100644 --- a/codespell_lib/_codespell.py +++ b/codespell_lib/_codespell.py @@ -881,7 +881,8 @@ def parse_file( ) for i, line in enumerate(lines): - if line.rstrip() in exclude_lines: + line = line.rstrip() + if not line or line in exclude_lines: continue extra_words_to_ignore = set()