Repository navigation
Conversation
…meouts The timeout sweep issued one blind, unbounded UPDATE over every deferred task instance past its trigger timeout. On MySQL that statement waits on rows the triggerer is writing while the triggerer waits on rows the sweep already holds, so InnoDB breaks the cycle by killing one of them. The retry the sweep already had does not save it: the deadlock recurs on every tick, and once the retries are exhausted the exception ends the scheduler loop. Deployments running more than one triggerer hit this often enough to lose runs. Claiming the rows up front with SKIP LOCKED means the sweep never waits on a row another writer holds -- it leaves those for the next tick -- so it can no longer be one side of a lock cycle. The claim also bounds how many rows a single tick may lock, so a large backlog of timed-out tasks cannot stall the scheduler loop, and rolling back a failed attempt lets the existing retry start from a clean transaction on the backends where row locks are unavailable and a cycle is still possible.
fat-catTW
suggested changes
Oct 29, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #72595, closed by the open PR limit.
Summary
The deferred-task timeout sweep fails every timed-out task instance in one unbounded
UPDATE. On MySQL that statement can wait on rows the triggerer is writing while the triggerer waits on rows the sweep holds, and InnoDB kills one side.The sweep's retry does not prevent this: under the same triggerer traffic the cycle can form again, and once the retries run out the exception ends the scheduler loop, which is the crash #56670 reports.
closes: #56670
Change
FOR UPDATE SKIP LOCKEDand update only those.adopt_or_reset_orphaned_tasksdoes.With
SKIP LOCKEDthe sweep no longer waits on rows the triggerer holds, and theUPDATEtouches only rows its own transaction already claimed. Wherewith_row_locksis a no-op (SQLite, MariaDB,use_row_level_locking = False), the rollback is what lets the retry start from a clean transaction.Tests
MySQL 8.0.46, 450 timed-out deferred task instances, four writers re-deferring ten rows per transaction, three 30-second runs. The writers lock rows in descending id order: one order across writers, so they do not deadlock each other, and the reverse of the sweep's scan.
With one of three timed-out rows locked by another transaction, the sweep now skips that row and returns in 0.03s. Before, it waited out
innodb_lock_wait_timeoutand raised 1205 without failing any of the three.Behavior change
A tick fails at most 500 timed-out task instances; a larger backlog drains over later ticks.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines