[driv3r]

Refs #471

Stack: 2/3. Base: #472 (throttle-recovery/01-safe-inputs). Next: #474. Merge order: #472 → #473 (this PR) → #474. This diff is relative to #472.

Behavior

Today all copy workers stall while throttled and all resume with full-sized batches as soon as the throttle clears. After this change, copy load recovers gradually:

• While throttled, no new copy batch starts. Batches that are already running finish normally.
• Once the throttle clears, one batch runs at a time with max(1, configured / 8) rows.
• After each healthy DataIterationThrottleRecovery.RecoveryInterval (default 5s), the batch-size divisor steps 8 → 4 → 2 → 1. Then concurrency grows by one per interval up to DataIterationConcurrency. With the defaults that is 25 → 50 → 100 → 200 rows at concurrency 1, then concurrency 2 → 3 → 4. Only one control moves per interval.
• What makes a stage healthy:
  • The stage starts when its first batch is admitted.
  • At least one non-empty batch admitted in that stage must complete successfully, with no throttle observed.
  • With the LagThrottler, the stage also needs a lag sample newer than the one at stage start and no older than 2 × UpdateInterval. NULL or missing rows do not count as samples.
• What resets progress:
  • A failed batch restarts the stage interval without changing the limits.
  • A new throttle restarts recovery from one batch at 1/8 size.
  • Completions of batches admitted in an earlier stage do not count.
• Fairness: admission is FIFO, so every assigned table keeps making progress at reduced concurrency.
• Disabling the throttler (as cutover and RunStandaloneDataCopy do) restores the configured limits at the next admission and clears recovery history. Re-enabling it does not replay an old timer.
• Cold start without an observed throttle keeps today's full speed.
• Opt-out: DataIterationThrottleRecovery.Disabled: true keeps today's immediate resume. The hard throttle still applies.

Unchanged: the Throttler interface, WaitForThrottle, the binlog writer's hard stop, the pause API, lag semantics, binlog coordinates, serialized state, and status fields.

Implementation

• copy_throttle.go (new) adds copyThrottleController. It uses a mutex, a condition variable, FIFO tickets, in-flight/limit/divisor state, and a stage generation. Permits are value types. A single 500 ms ticker re-reads the throttler while batches wait, and acquire and release also re-read it. Cancelling closes the controller, and queued acquisitions return context.Canceled.
• DataIterator.Run builds one controller per run, outside Ferry.Run, so RunStandaloneDataCopy works too. It starts and joins the controller's ticker and stops it before the done listeners run. Ferry.NewDataIterator copies Config.DataIterationThrottleRecovery. Verifier and binlog-writer cursors are unaffected.
• Cursor.Each acquires a permit inside each read retry, before DB.Begin. It reads the configured base after admission, and copyThrottlePermit.batchSize scales it. The permit is held through fetch, listeners, target write and source rollback, and is released exactly once on every path. Exported Fetch keeps the full configured size and delegates to the new private fetch(db, batchSize).
• LagThrottler records a sample sequence and observation time for successful non-NULL samples only, and exposes them through the private recoverySample(). NewLagThrottler rejects a non-positive UpdateInterval. ValidateConfig rejects a negative DataIterationConcurrency and an invalid RecoveryInterval.
• Docs: CHANGELOG.md and docs/technicaloverview.md.

Tests

• copy_throttle_test.go: controller tests using a fake clock, FIFO contenders and no sleeps. Cases:
  • Full ramp (1,25) (1,50) (1,100) (1,200) (2,200) (3,200) (4,200), asserting admitted and waiting counts at every stage.
  • A throttle while 4 batches are in flight; earlier-stage completions; restart after flapping.
  • No growth without a completed batch, after a failure, or with a stale or unchanged lag sample; empty fetches stay neutral.
  • FIFO order; a re-throttle during a fetch retry; batch-size scaling (1 and 7 stay at least 1, 80 → 10, 104 → 13).
  • Disable precedence and re-enable; cancellation of queued waiters, including a cursor that returns without touching the database; config bounds.
• test/go/race_conditions_integration_test.go, real MySQL with a full Ferry:
  • TestCopyThrottleResumesOneSmallBatchWithoutSourceLocks pauses four running table copies and drains them. The first resumed batch has 25 rows (or 10 with an 80-row override). While it is held:
    • An UPDATE on its rows hits Lock wait timeout (positive control).
    • UPDATEs on every waiting table complete with innodb_lock_wait_timeout=1.
    • A sentinel insert beyond the max key is replayed to the target.
    • No second batch starts.
    • After a live base update to 104, recovery batches have 13 rows (10 for the 80-row override). Final checksums and ordered rows match.
  • TestCopyThrottleLagThrottlerRecoversCopyGradually uses a real LagThrottler on meta.lag_table. Lag above the threshold stops both copy and replay. Fresh low lag resumes with 25 rows and replays the sentinel. Batch sizes step 25 → 50 → 100 → 200 without skipping a stage. With all lag rows deleted, sizes stay at 50.
• test/go/data_iterator_test.go: with recovery disabled, the hard pause still holds and copying resumes at full size. A paused cold start resumes with 1-row batches (7 / 8) and finishes the table with exact rows.
• test/go/config_test.go and test/go/lag_throttler_test.go: validation bounds.

Checks run (MySQL 8.0.32 from docker-compose_8.0.yml)

1. go test -race . -run '^TestCopyThrottle' -count=1: pass. Also passed with -count=30, and with -count=20 -cpu 1,2,8.
2. MYSQL_VERSION=8.0 go test ./test/go -run 'TestCopyThrottle|TestDataIterator|TestThrottler|TestNewThrottler|TestSelectUpdateBinlogCopy|TestUpdateBinlogSelectCopy|TestSelectCopyUpdateBinlog|TestOnlyDeleteRowWithMaxPaginationKey' -count=1 -p=1: pass.
3. MYSQL_VERSION=8.0 go test -race ./test/go -run 'TestCopyThrottle|TestThrottlerThrottlesAndUnthrottles|TestThrottlerMultipleServerIDs' -count=1 -p=1: pass.
4. MYSQL_VERSION=8.0 go test . ./test/go ./copydb/test ./sharding/test -count=1 -p=1: pass.
5. Command 2 with GHOSTFERRY_BINLOG_COORDINATE_MODE=gtid: pass on rerun. The first run hit TestThrottlerMultipleServerIDs, an existing timing flake: in -count=100 loops, unmodified origin/main failed 3 of 200 runs, and in a -count=40 loop this branch failed 5 of 80.
6. MYSQL_VERSION=8.0 bundle exec rake test TEST=test/integration/interrupt_resume_test.rb: 26/26 passed in 4 consecutive runs on this branch. One earlier run hung. A hang captured later on the stack tip traced to an existing test flake in test_interrupt_resume_inline_verifier_will_verify_entries_in_reverify_store, where the deleted seed id can be a hole; details in Gradual copy throttle recovery #471.

• Mutation check: moving admission after Begin + fetch makes TestCopyThrottleResumesOneSmallBatchWithoutSourceLocks fail with Error 1205 on the waiting-table UPDATE.
• Real-program smoke in both coordinate modes: evidence comment on #471.