Skip to content

fix: hold the reload lock in short slices and stop on an unresponsive client - #450

Merged
3alpha merged 2 commits into
mainfrom
fix/reload-lock-chunks
Nov 17, 2026
Merged

3alpha merged 2 commits into
mainfrom
fix/reload-lock-chunks

Conversation

@3alpha

@3alpha 3alpha commented Nov 17, 2026

Copy link
Copy Markdown
Member

Problem

The reload cron holds the validator operation lock for its whole pass, including one fee recipient POST per key, sent one after another. UI and Launchpad API operations wait at most 30s (CLIENT_OPERATION_MAX_WAIT_MS) for the lock and then fail as busy (HTTP 503 for Lido CSM and Stader). 0.1.33 had no lock, so this is new in 0.2.0.

On a node with many validators, or a client that is slow per write (for example, Lighthouse rewrites its validator definitions file on every POST), a pass can exceed 30s, and users get intermittent busy errors. If the validator client accepts connections but never answers, each POST waits the 120s default timeout, so the lock can be held for hours and blocks deletes.

Change

  • Key sync (steps 0–5) still runs under one lock. It is a few bulk calls and returns the keys the signer holds.
  • Fee recipients (step 6) and builder configs are applied in slices that hold the lock for at most 1s (forEachInLockedSlices). Each slice re-reads the DB inside the lock, so it never re-sends a fee recipient a user has since replaced, and it skips keys deleted mid-pass. User operations now wait at most about one slice plus one request.
  • Early stop: reload fee recipient POSTs time out after 10s. A request with no HTTP response (timeout, refused or reset) stops the pass, and the next pass retries every key. HTTP errors concern a single key, so the pass continues, as before.
  • API errors now carry their cause (ErrorOptions), and receivedHttpResponse() checks it. setFeeRecipient takes an optional timeout; other callers are unchanged.

Tests

New reloadLockSlices.unit.test.ts:

  • With 800 keys at 5ms per write, a user fee update made mid-pass finishes within one slice, every key is still configured, and the pass sends the new value for the updated key, never the old one. This test fails if a slice is allowed to last the whole pass.
  • A key deleted mid-pass is skipped.
  • Against a real local HTTP server: a timeout stops after one request and the next pass covers every key; a 404 does not stop the pass; a refused connection stops after the first attempt.

Lint, build and all suites pass (brain unit 116 + 1 skipped, e2e 10, UI 65, common 13). The timing test passed 5 runs in a row.

Trade-offs

  • Each slice re-reads and validates the DB: one extra read per second of pass time.
  • When keys are added or removed, open UIs are notified at the end of key sync, before fee recipients are re-sent. A UI that lists in that gap may briefly show a fee mismatch. It gets another refresh when the pass ends (covered by the existing e2e repair tests).

… client

The reload pass held the validator operation lock for its whole run, including one
fee recipient POST per key. On large nodes a pass can exceed the 30s wait budget of
user operations, which then failed as busy (HTTP 503 on the Launchpad API).

Key membership (steps 0-5) is still reconciled under one lock. Fee recipients and
builder configs are then applied in slices that hold the lock for at most 1s and
re-read the database inside the lock, so they never re-send a value a user has
since replaced and skip keys deleted mid-pass.

Reload fee recipient writes now time out after 10s. A request with no HTTP response
(timeout, refused or reset) stops the pass; the next pass retries every key. HTTP
errors concern one key and the pass continues, as before.
@3alpha
3alpha requested a review from a team as a code owner November 17, 2026 12:05
…nding

When the fee recipient step stopped because the validator client gave no response,
reload still applied every builder configuration, each waiting up to 10s. With the
builder feature enabled this could delay the next pass for hours on large nodes.

The slice helper now reports whether a step stopped the run. Reload skips the builder
phase when the fee step stopped, and the builder step itself stops at the first
request without a response. Builder errors keep a cause with only the HTTP status,
never the response body, so echoed authorization data stays out of logs.
@3alpha
3alpha merged commit 5d0c759 into main Nov 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant