Repository navigation
Conversation
Every clean shutdown SIGKILLed the llama-server sidecar with no grace period
and logged `ERROR embeddings stop` with a context error. Both are gone.
Cause: defer ordering in `run()`. The embeddings stop is deferred at the
declaration of `shutdownCtx`, the tunnel stop a few hundred lines later, and
the shared 10s budget was created in the signal-handling path with a
`defer cancel()` right beside it. Defers run last-registered-first, so that
`cancel()` always fired BEFORE both stops and handed each of them an
already-cancelled context. The M7 comment above the budget states the intent
-- "single shared shutdown budget for HTTP drain + embeddings supervisor" --
and the ordering silently denied it: nothing but the HTTP drain ever got to
use the budget.
Evidence from a clean SIGTERM before the change, timestamps relative:
INFO +0.000s server stopped
INFO +0.001s sending SIGTERM to llama-server
WARN +0.001s SIGTERM timed out, sending SIGKILL
INFO +0.005s llama-server exited on shutdown err=signal: killed
ERROR +0.005s embeddings stop err=context canceled
The escalation at +0.001s is the tell: no timeout elapsed. `context canceled`
rather than `context deadline exceeded` says the same thing -- the budget was
cancelled, not spent.
Fix: park the cancel in `shutdownCancel` (via `newShutdownBudget`) and release
it from a defer registered at the declaration, i.e. BEFORE both stop defers,
so it runs after them. One budget now covers the HTTP drain and both
subsystem stops, which is what M7 asked for; the 10s total is unchanged, so
this does not reintroduce the 20s-of-grace problem that comment was written
about.
The cancel is parked in a helper rather than assigned inline because
`go vet`'s lostcancel check cannot see through the defer ordering and reports
a false "not used on all paths" when the assignment is made in `run()`.
Verified live, same harness, same config, only the binary differing:
- after: `server stopped` -> `sending SIGTERM to llama-server` ->
`llama-server exited on shutdown` in 10ms. No SIGKILL, no `signal:
killed`, zero ERROR lines in the whole run. llama-server never needed a
long grace -- it just needed to be given any.
- the startup-error path, where `shutdownCtx` is still nil, was exercised
on purpose (unwritable `CIX_VECTORS_DIR`, which fails after the provider
is constructed): the pre-existing fresh-context fallback still stops the
sidecar cleanly, exit 1, no ERROR and no SIGKILL.
- `go build`, `go vet`, `gofmt -l` and `go test ./...` all clean, 0 test
failures.
The tunnel stop shares the same `shutdownCtx` and is fixed by the same
change, but no tunnel was enabled in the harness, so that path is reasoned
rather than observed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(server): let the shared shutdown budget outlive the subsystem stops
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
developtomain. One commit, one file: the shutdown-budget fix from #314.This publishes nothing and is not a release
Deliberately no tag is cut. Fixes are being accumulated on
mainso a release can batch several of them later instead of one tag per fix.Verified rather than assumed — every workflow that fires on a push to
main:ci-cli.ymlcli/**ci-plugin.ymlci-server.ymlserver/**ci-site.ymlsite/**codeql.ymlsecurity.ymldeploy-site.ymlsite/**The only publishing one is
deploy-site.yml, and this promotion touches nosite/**path, so it does not fire. The release workflows are tag-triggered only —release-server.ymlonserver/v*,release-cli.ymloncli/v*,release-mac.ymlonmac/v*— so no image, binary or GitHub release is produced by merging this.Practical consequence: the fix is on
mainand ready, and it ships with whicheverserver/vX.Y.Ztag is cut next.What it carries
Before this, every clean shutdown SIGKILLed the llama-server sidecar with no grace period and logged
ERROR embeddings stop, because adefer cancel()beside the shared 10s budget ran before the subsystem stops that were deferred earlier inrun(). #314 has the full diagnosis, the before/after logs and the verification; nothing has changed since it was merged.🤖 Generated with Claude Code