Skip to content

fix(server): let the shared shutdown budget outlive the subsystem stops - #314

Merged
dvcdsys merged 1 commit into
developfrom
fix/shutdown-budget
Nov 3, 2026
Merged

dvcdsys merged 1 commit into
developfrom
fix/shutdown-budget

Conversation

@dvcdsys

@dvcdsys dvcdsys commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Every clean shutdown SIGKILLed the llama-server sidecar with no grace period and logged ERROR embeddings stop with a context error. Both are gone.

Cause

Defer ordering in run(). The embeddings stop is deferred at the declaration of shutdownCtx, the tunnel stop a few hundred lines later, and the shared 10s budget was created in the signal-handling path with a defer cancel() right beside it. Defers run last-registered-first, so that cancel() always fired before both stops and handed each of them an already-cancelled context.

The M7 comment above the budget states the intent — "single shared shutdown budget for HTTP drain + embeddings supervisor" — and the ordering silently denied it. Nothing but the HTTP drain ever got to use the budget.

Evidence

A clean SIGTERM before the change, timestamps relative:

INFO  +0.000s  server stopped
INFO  +0.001s  sending SIGTERM to llama-server
WARN  +0.001s  SIGTERM timed out, sending SIGKILL
INFO  +0.005s  llama-server exited on shutdown   err=signal: killed
ERROR +0.005s  embeddings stop                   err=context canceled

The escalation at +0.001s is the tell: no timeout elapsed. context canceled rather than context deadline exceeded says the same thing — the budget was cancelled, not spent.

Fix

Park the cancel in shutdownCancel (via newShutdownBudget) and release it from a defer registered at the declaration, i.e. before both stop defers, so it runs after them. One budget now covers the HTTP drain and both subsystem stops, which is what M7 asked for. The 10s total is unchanged, so this does not reintroduce the 20s-of-grace problem that comment was written about.

The cancel is parked in a helper rather than assigned inline because go vet's lostcancel check cannot see through the defer ordering and reports a false "not used on all paths" when the assignment is made in run().

Verification

Live, same harness and config, only the binary differing:

INFO  server stopped
INFO  sending SIGTERM to llama-server
INFO  llama-server exited on shutdown

10 ms, no SIGKILL, no signal: killed, zero ERROR lines in the whole run. llama-server never needed a long grace — it just needed to be given any.

  • The startup-error path, where shutdownCtx is still nil, was exercised on purpose (unwritable CIX_VECTORS_DIR, which fails after the provider is constructed): the pre-existing fresh-context fallback still stops the sidecar cleanly, exit 1, no ERROR and no SIGKILL.
  • go build, go vet, gofmt -l, go test ./... — all clean, 0 test failures.

The tunnel stop shares the same shutdownCtx and is fixed by the same change, but no tunnel was enabled in the harness, so that path is reasoned rather than observed.

🤖 Generated with Claude Code

Every clean shutdown SIGKILLed the llama-server sidecar with no grace period
and logged `ERROR embeddings stop` with a context error. Both are gone.

Cause: defer ordering in `run()`. The embeddings stop is deferred at the
declaration of `shutdownCtx`, the tunnel stop a few hundred lines later, and
the shared 10s budget was created in the signal-handling path with a
`defer cancel()` right beside it. Defers run last-registered-first, so that
`cancel()` always fired BEFORE both stops and handed each of them an
already-cancelled context. The M7 comment above the budget states the intent
-- "single shared shutdown budget for HTTP drain + embeddings supervisor" --
and the ordering silently denied it: nothing but the HTTP drain ever got to
use the budget.

Evidence from a clean SIGTERM before the change, timestamps relative:

    INFO  +0.000s  server stopped
    INFO  +0.001s  sending SIGTERM to llama-server
    WARN  +0.001s  SIGTERM timed out, sending SIGKILL
    INFO  +0.005s  llama-server exited on shutdown   err=signal: killed
    ERROR +0.005s  embeddings stop                   err=context canceled

The escalation at +0.001s is the tell: no timeout elapsed. `context canceled`
rather than `context deadline exceeded` says the same thing -- the budget was
cancelled, not spent.

Fix: park the cancel in `shutdownCancel` (via `newShutdownBudget`) and release
it from a defer registered at the declaration, i.e. BEFORE both stop defers,
so it runs after them. One budget now covers the HTTP drain and both
subsystem stops, which is what M7 asked for; the 10s total is unchanged, so
this does not reintroduce the 20s-of-grace problem that comment was written
about.

The cancel is parked in a helper rather than assigned inline because
`go vet`'s lostcancel check cannot see through the defer ordering and reports
a false "not used on all paths" when the assignment is made in `run()`.

Verified live, same harness, same config, only the binary differing:

  - after: `server stopped` -> `sending SIGTERM to llama-server` ->
    `llama-server exited on shutdown` in 10ms. No SIGKILL, no `signal:
    killed`, zero ERROR lines in the whole run. llama-server never needed a
    long grace -- it just needed to be given any.
  - the startup-error path, where `shutdownCtx` is still nil, was exercised
    on purpose (unwritable `CIX_VECTORS_DIR`, which fails after the provider
    is constructed): the pre-existing fresh-context fallback still stops the
    sidecar cleanly, exit 1, no ERROR and no SIGKILL.
  - `go build`, `go vet`, `gofmt -l` and `go test ./...` all clean, 0 test
    failures.

The tunnel stop shares the same `shutdownCtx` and is fixed by the same
change, but no tunnel was enabled in the harness, so that path is reasoned
rather than observed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dvcdsys
dvcdsys merged commit 655a197 into develop Nov 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant