Skip to content

chore: promote develop to main (shutdown budget fix, no tag) - #315

Merged
dvcdsys merged 2 commits into
mainfrom
develop
Nov 3, 2026
Merged

dvcdsys merged 2 commits into
mainfrom
develop

Conversation

@dvcdsys

@dvcdsys dvcdsys commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Promotes develop to main. One commit, one file: the shutdown-budget fix from #314.

server/cmd/cix-server/main.go | 41 +++++++++++++++++++---

This publishes nothing and is not a release

Deliberately no tag is cut. Fixes are being accumulated on main so a release can batch several of them later instead of one tag per fix.

Verified rather than assumed — every workflow that fires on a push to main:

workflow publishes? path filter
ci-cli.yml no cli/**
ci-plugin.yml no all
ci-server.yml no server/**
ci-site.yml no site/**
codeql.yml no all
security.yml no all
deploy-site.yml yes site/**

The only publishing one is deploy-site.yml, and this promotion touches no site/** path, so it does not fire. The release workflows are tag-triggered only — release-server.yml on server/v*, release-cli.yml on cli/v*, release-mac.yml on mac/v* — so no image, binary or GitHub release is produced by merging this.

Practical consequence: the fix is on main and ready, and it ships with whichever server/vX.Y.Z tag is cut next.

What it carries

Before this, every clean shutdown SIGKILLed the llama-server sidecar with no grace period and logged ERROR embeddings stop, because a defer cancel() beside the shared 10s budget ran before the subsystem stops that were deferred earlier in run(). #314 has the full diagnosis, the before/after logs and the verification; nothing has changed since it was merged.

🤖 Generated with Claude Code

dvcdsys and others added 2 commits November 3, 2026 16:55
Every clean shutdown SIGKILLed the llama-server sidecar with no grace period
and logged `ERROR embeddings stop` with a context error. Both are gone.

Cause: defer ordering in `run()`. The embeddings stop is deferred at the
declaration of `shutdownCtx`, the tunnel stop a few hundred lines later, and
the shared 10s budget was created in the signal-handling path with a
`defer cancel()` right beside it. Defers run last-registered-first, so that
`cancel()` always fired BEFORE both stops and handed each of them an
already-cancelled context. The M7 comment above the budget states the intent
-- "single shared shutdown budget for HTTP drain + embeddings supervisor" --
and the ordering silently denied it: nothing but the HTTP drain ever got to
use the budget.

Evidence from a clean SIGTERM before the change, timestamps relative:

    INFO  +0.000s  server stopped
    INFO  +0.001s  sending SIGTERM to llama-server
    WARN  +0.001s  SIGTERM timed out, sending SIGKILL
    INFO  +0.005s  llama-server exited on shutdown   err=signal: killed
    ERROR +0.005s  embeddings stop                   err=context canceled

The escalation at +0.001s is the tell: no timeout elapsed. `context canceled`
rather than `context deadline exceeded` says the same thing -- the budget was
cancelled, not spent.

Fix: park the cancel in `shutdownCancel` (via `newShutdownBudget`) and release
it from a defer registered at the declaration, i.e. BEFORE both stop defers,
so it runs after them. One budget now covers the HTTP drain and both
subsystem stops, which is what M7 asked for; the 10s total is unchanged, so
this does not reintroduce the 20s-of-grace problem that comment was written
about.

The cancel is parked in a helper rather than assigned inline because
`go vet`'s lostcancel check cannot see through the defer ordering and reports
a false "not used on all paths" when the assignment is made in `run()`.

Verified live, same harness, same config, only the binary differing:

  - after: `server stopped` -> `sending SIGTERM to llama-server` ->
    `llama-server exited on shutdown` in 10ms. No SIGKILL, no `signal:
    killed`, zero ERROR lines in the whole run. llama-server never needed a
    long grace -- it just needed to be given any.
  - the startup-error path, where `shutdownCtx` is still nil, was exercised
    on purpose (unwritable `CIX_VECTORS_DIR`, which fails after the provider
    is constructed): the pre-existing fresh-context fallback still stops the
    sidecar cleanly, exit 1, no ERROR and no SIGKILL.
  - `go build`, `go vet`, `gofmt -l` and `go test ./...` all clean, 0 test
    failures.

The tunnel stop shares the same `shutdownCtx` and is fixed by the same
change, but no tunnel was enabled in the harness, so that path is reasoned
rather than observed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(server): let the shared shutdown budget outlive the subsystem stops
@dvcdsys
dvcdsys merged commit 87b6bff into main Nov 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant