Skip to content

[3/3] Verify throttle recovery across interruption and cutover - #474

Draft
driv3r wants to merge 1 commit into
throttle-recovery/02-gradual-recoveryfrom
throttle-recovery/03-resume-coverage
Draft

driv3r wants to merge 1 commit into
throttle-recovery/02-gradual-recoveryfrom
throttle-recovery/03-resume-coverage

Conversation

@driv3r

@driv3r driv3r commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Refs #471

Stack: 3/3. Base: #473 (throttle-recovery/02-gradual-recovery). Previous: #472 → #473. Merge order: #472 → #473 → #474 (this PR). This diff is relative to #473. It adds regression coverage only and does not change scheduling.

Scope

  • test/lib/go/throttle_recovery_ghostferry/main.go is a test executable using the tf.Setup/tf.Run pattern. Its BeforeInitialize sets concurrency 1, base batch size 200, DataIterationThrottleRecovery.RecoveryInterval: "30s", and the control server on 127.0.0.1:18000.
  • test_interrupt_resume_during_throttle_recovery (test/integration/interrupt_resume_test.rb):
    1. The first BEFORE_ROW_COPY sends POST /api/pause to the control server.
    2. The first AFTER_ROW_COPY starts a thread that sends /api/unpause one second later. The callback returns while the pause is still set.
    3. The second BEFORE_ROW_COPY pauses again, and the second AFTER_ROW_COPY calls term_and_wait_for_exit.
    4. The interrupted target holds exactly 200 + 25 rows. Its max id matches the dumped LastSuccessfulPaginationKeys, and the dumped state has no throttle or recovery keys.
    5. The unpause thread is joined after exit so request failures propagate. Resuming with minimal_ghostferry yields an identical table.
  • TestCopyThrottleStandaloneCopyBypassesPause (test/go/data_iterator_test.go) seeds non-empty source tables and empty target tables, with a throttler that is both paused and disabled. It runs the real Ferry.RunStandaloneDataCopy, without Ferry.Run or its supporting services, and requires it to finish with ordered rows identical to the source.

Checks run (MySQL 8.0.32 from docker-compose_8.0.yml)

  • MYSQL_VERSION=8.0 bundle exec rake test TEST=test/integration/interrupt_resume_test.rb: 27/27 pass.
  • The same with GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid: 27/27 pass on rerun. The first GTID run hung after test_interrupt_resume_during_throttle_recovery had passed, and the hanging test was not identified (see Gradual copy throttle recovery #471).
  • MYSQL_VERSION=8.0 go test ./test/go -run '^TestCopyThrottleStandaloneCopyBypassesPause$' -count=1 -p=1: pass.
  • MYSQL_VERSION=8.0 go test ./sharding/test -run 'TestJoinedTablesTestSuite|TestPrimaryKeyTableTestSuite' -count=1 -p=1: pass.
  • Verification commands 1–5 from Gradual copy throttle recovery #471 at this stack tip: pass.

- test_interrupt_resume_during_throttle_recovery pauses through the
  control server during the first batch, unpauses, pauses again during the
  first 25-row recovery batch and interrupts. The interrupted target holds
  200 + 25 rows matching the dumped state, and resuming with the minimal
  Ghostferry copies the table identically.
- TestCopyThrottleStandaloneCopyBypassesPause runs RunStandaloneDataCopy
  with a paused and disabled throttler and no Ferry.Run supporting
  services; it must finish with identical rows.

Refs #471

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant