Repository navigation
fix(server): let the shared shutdown budget outlive the subsystem stops - #314
Merged
Merged
Conversation
Every clean shutdown SIGKILLed the llama-server sidecar with no grace period
and logged `ERROR embeddings stop` with a context error. Both are gone.
Cause: defer ordering in `run()`. The embeddings stop is deferred at the
declaration of `shutdownCtx`, the tunnel stop a few hundred lines later, and
the shared 10s budget was created in the signal-handling path with a
`defer cancel()` right beside it. Defers run last-registered-first, so that
`cancel()` always fired BEFORE both stops and handed each of them an
already-cancelled context. The M7 comment above the budget states the intent
-- "single shared shutdown budget for HTTP drain + embeddings supervisor" --
and the ordering silently denied it: nothing but the HTTP drain ever got to
use the budget.
Evidence from a clean SIGTERM before the change, timestamps relative:
INFO +0.000s server stopped
INFO +0.001s sending SIGTERM to llama-server
WARN +0.001s SIGTERM timed out, sending SIGKILL
INFO +0.005s llama-server exited on shutdown err=signal: killed
ERROR +0.005s embeddings stop err=context canceled
The escalation at +0.001s is the tell: no timeout elapsed. `context canceled`
rather than `context deadline exceeded` says the same thing -- the budget was
cancelled, not spent.
Fix: park the cancel in `shutdownCancel` (via `newShutdownBudget`) and release
it from a defer registered at the declaration, i.e. BEFORE both stop defers,
so it runs after them. One budget now covers the HTTP drain and both
subsystem stops, which is what M7 asked for; the 10s total is unchanged, so
this does not reintroduce the 20s-of-grace problem that comment was written
about.
The cancel is parked in a helper rather than assigned inline because
`go vet`'s lostcancel check cannot see through the defer ordering and reports
a false "not used on all paths" when the assignment is made in `run()`.
Verified live, same harness, same config, only the binary differing:
- after: `server stopped` -> `sending SIGTERM to llama-server` ->
`llama-server exited on shutdown` in 10ms. No SIGKILL, no `signal:
killed`, zero ERROR lines in the whole run. llama-server never needed a
long grace -- it just needed to be given any.
- the startup-error path, where `shutdownCtx` is still nil, was exercised
on purpose (unwritable `CIX_VECTORS_DIR`, which fails after the provider
is constructed): the pre-existing fresh-context fallback still stops the
sidecar cleanly, exit 1, no ERROR and no SIGKILL.
- `go build`, `go vet`, `gofmt -l` and `go test ./...` all clean, 0 test
failures.
The tunnel stop shares the same `shutdownCtx` and is fixed by the same
change, but no tunnel was enabled in the harness, so that path is reasoned
rather than observed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every clean shutdown SIGKILLed the llama-server sidecar with no grace period and logged
ERROR embeddings stopwith a context error. Both are gone.Cause
Defer ordering in
run(). The embeddings stop is deferred at the declaration ofshutdownCtx, the tunnel stop a few hundred lines later, and the shared 10s budget was created in the signal-handling path with adefer cancel()right beside it. Defers run last-registered-first, so thatcancel()always fired before both stops and handed each of them an already-cancelled context.The M7 comment above the budget states the intent — "single shared shutdown budget for HTTP drain + embeddings supervisor" — and the ordering silently denied it. Nothing but the HTTP drain ever got to use the budget.
Evidence
A clean SIGTERM before the change, timestamps relative:
The escalation at +0.001s is the tell: no timeout elapsed.
context canceledrather thancontext deadline exceededsays the same thing — the budget was cancelled, not spent.Fix
Park the cancel in
shutdownCancel(vianewShutdownBudget) and release it from a defer registered at the declaration, i.e. before both stop defers, so it runs after them. One budget now covers the HTTP drain and both subsystem stops, which is what M7 asked for. The 10s total is unchanged, so this does not reintroduce the 20s-of-grace problem that comment was written about.The cancel is parked in a helper rather than assigned inline because
go vet's lostcancel check cannot see through the defer ordering and reports a false "not used on all paths" when the assignment is made inrun().Verification
Live, same harness and config, only the binary differing:
10 ms, no SIGKILL, no
signal: killed, zero ERROR lines in the whole run. llama-server never needed a long grace — it just needed to be given any.shutdownCtxis still nil, was exercised on purpose (unwritableCIX_VECTORS_DIR, which fails after the provider is constructed): the pre-existing fresh-context fallback still stops the sidecar cleanly, exit 1, no ERROR and no SIGKILL.go build,go vet,gofmt -l,go test ./...— all clean, 0 test failures.The tunnel stop shares the same
shutdownCtxand is fixed by the same change, but no tunnel was enabled in the harness, so that path is reasoned rather than observed.🤖 Generated with Claude Code