-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mariadb deadlocked occasionally #39
Comments
Attempt at addressing this: lobsters/lobsters@f054967 |
Fewer branches, much less redundancy. Remove transaction. It occasionally deadlocked in prod (lobsters-ansible issue #39), and it's unneeded: the comment/story updates are done relatively rather than absolutely.
This is still present, unfortunately. We've seen three of them on comment creation since the previous commit. |
FWIW, they've entirely stopped happening in voting code since that tweak. Strong suggestion this only happens inside of transactions. |
Haven't seen this in a while. Or the lobsters/lobsters#783, now that I think about it. |
Broken in Ruby 3, has been eating exceptions: qertoip/transaction_retry#12 Related to lobsters/lobsters-ansible#39 which hopefully won't recur.
Dammit. We're immediately back to this.
|
I've been rerunning
So the deadlock is happening because Comment creation (and upvoting, or Story creation) are locking the same tables in different orders. Probably the right fix is to remove those transactions. I really don't want to try to reorganize that rat's nest of callbacks to hit tables in the same order because there's no way to enforce it, we'd surely reintroduce the deadlock before long. |
My understanding is that we needed the transactions to ensure that all of the calculations worked properly. |
Writes a vote directly to avoid vote_thusly doing round trips to check if one exists, etc. Removes redundant transactions from controllers from #899. Rails already creates a transaction for the .save. Unifies Story cache updating. Previously recalculate_hotness! was called twice on comment creation. Moves comment counting into the db. Shorter transaction should reduce the frequence of lobsters/lobsters-ansible/issues/39 but seems unlikely to eliminate it as the create + upvote transactions for stories + comments still read/write from stories, comments, and votes.
Broken in Ruby 3, has been eating exceptions: qertoip/transaction_retry#12 Related to lobsters/lobsters-ansible#39 which hopefully won't recur.
Writes a vote directly to avoid vote_thusly doing round trips to check if one exists, etc. Removes redundant transactions from controllers from lobsters#899. Rails already creates a transaction for the .save. Unifies Story cache updating. Previously recalculate_hotness! was called twice on comment creation. Moves comment counting into the db. Shorter transaction should reduce the frequence of lobsters/lobsters-ansible/issues/39 but seems unlikely to eliminate it as the create + upvote transactions for stories + comments still read/write from stories, comments, and votes.
In the last couple weeks, every couple days I get an exception from a query failing with a complaint about a deadlock:
(Headers, env vars, and traceback follow, but there's nothing interesting there.) This almost always happens on an story or comment upvote (the most common kind of write on this table), but I have seen a couple on creating a comment.
I poked around a bit and saw someone suggest running
SHOW ENGINE INNODB STATUS;
, so here's that output:Given our generally low db usage we probably should never see errors like this. Can anyone read this to guess at what's misconfigured?
The text was updated successfully, but these errors were encountered: