Skip to content

Feature/model gateway - #24

Merged
SURYANSH-RAI merged 27 commits into
feature/durable-matching-jobsfrom
feature/model-gateway
Nov 13, 2026
Merged

SURYANSH-RAI merged 27 commits into
feature/durable-matching-jobsfrom
feature/model-gateway

Conversation

@SURYANSH-RAI

Copy link
Copy Markdown
Collaborator

R22: Model gateway

Embed and rerank calls obey the same limits; provider outages return the defined fallback; unrelated embedding spaces are never mixed.

Decisions (recommended options):

  • One gateway for both ports: per-provider bulkhead and circuit breaker, per-operation time budgets covering retries, and retries for transient errors only.
  • Fallbacks: query embedding failure → word matching; profile embedding failure → job backoff; rerank failure → scored order.
  • Search budget: 6 seconds total.
  • Usage table kept 90 days.
  • Vectors are compared only within one model.

Changes:

  • Part 1: CircuitBreaker, Bulkhead, model-gateway.ts, abort signals in providers, search deadline, migration 0013 model_usage.
  • Part 2: gateway tests and npm run models:usage (optional MODEL_PRICE_PER_1K_TOKENS).
  • Part 3: embedding_model stored and respected in retrieval, scoring and relevance; re-embed on model change.

CircuitBreaker and Bulkhead were empty classes, and provider calls only
had their own timeouts: no shared budget, concurrency limit, retry
policy, fallback or usage accounting.

- CircuitBreaker (opens after 5 failures, one trial call after 30s) and
  Bulkhead (bounded concurrency and queue per provider)
- model-gateway.ts wraps both ports: per-operation time budgets that
  cover retries, abort signals so a timed-out request really stops,
  one or two jittered retries for 429/5xx/timeouts/network errors only
- Fallbacks: a failed query embedding returns null (search matches by
  words); a failed profile embedding throws (the job's own backoff
  retries later); a failed rerank keeps the scored order
- Search has a 6-second total budget split between query embedding and
  rerank; a rerank is skipped when under 750 ms remain
- Migration 0013: model_usage records every call (provider, operation,
  outcome, latency, characters), kept 90 days by the sweep
- Unit tests: circuit breaker open/half-open/close, bulkhead limits and
  queue timeout, transient-only retries with every attempt recorded,
  budget timeouts, query vs profile fallbacks, circuit stopping calls,
  and rerank skipped when the request is nearly out of time
- npm run models:usage [-- <hours>] reports calls, failures, average
  and p95 latency, approximate tokens and, with
  MODEL_PRICE_PER_1K_TOKENS, estimated cost per provider and operation
Vectors from different providers or models live in different spaces,
but nothing recorded which model produced a stored embedding, so
switching providers would compare unrelated vectors.

- event_participants.embedding_model (migration 0013) is written with
  every embedding (embedding_model of the gateway's provider id)
- Retrieval ranks by vector only among candidates embedded by the
  query's model; others keep their place by text relevance
- The semantic scorer and query relevance compare vectors only within
  one model, falling back to text otherwise
- A profile embedded by another model (or before this change) is
  embedded again with the current one on its next matching run
- Tests: scorer and relevance fallbacks, re-embedding, and real-Postgres
  retrieval that ignores an exact match from another model
Notification e-mails were sent inline after the action and a failure
was only logged, so a mail outage lost the message.

- Migration 0014: notification_outbox (unique dedupe key, attempts,
  lease, last error) and location_requests (used by part 2); the sweep
  removes sent notifications after 30 days and location requests after
  a day
- An intro request and its notification, and an acceptance and its
  notice to the requester, commit in one transaction
- The worker delivers queued notifications with a 2-minute lease and
  backoff retries (30s ... 15 min, 5 attempts); details are read at
  send time and an interaction that no longer applies is closed
  without sending; replays never send twice
- E-mails carry a deep link to the connection on the Connections page;
  the notifier gains location-request and reply messages
- Real-Postgres tests: queued once with the request, delivered with the
  link, acceptance notice, and a failed send kept for retry
…rt 2)

"Ping zone" only overwrote the sender's own zone for every connection;
there was no way to ask a connection where they are, and no reply.

- domain/connection/location.rules.ts: a request is a question, not
  tracking. One open request per direction of an accepted connection;
  unanswered requests expire after 30 minutes; a shared zone is
  visible only to the person who asked, for 15 minutes; 10 requests
  per person and hour
- LocationRepository (Postgres): creating a request and answering it
  (a configured zone, or "not now") each queue their notification in
  the same transaction, so the interaction is never lost
- Connections page: "Ask where they are", an incoming request with
  "Share with <name>" / "Not now", the reply or decline for the asker,
  and the existing broadcast labelled "Share with all my connections";
  each connection has an anchor for e-mail deep links
- Tests: state and rules, the use cases, real-Postgres visibility,
  expiry and replacement, and a browser run where the worker e-mails
  the request, acceptance, location request and reply
Integration tests skip themselves without Postgres or Keycloak, which let
CI pass with the Keycloak suites never running. CI now starts the compose
Postgres and Keycloak, syncs the realm, sets REQUIRE_SERVICES so a global
setup refuses to start without them, and checks the JSON report so any
skipped test fails the job. Workers accept an optional event scope for
tests.
When every explanation of a shortlist was already cached the model was
not asked again and the list fell back to score order, so a refresh that
changed nothing reshuffled a stored, reranked list. The model's order is
now kept for the exact shortlist (keyed by all of its pair keys) and
reused on a full cache hit; erasure, removal, hiding and housekeeping
clear it with the explanations.
Runs the real embed and match workers against Postgres with deterministic
fake providers: the first person to save gets matches once others arrive,
the reranked order is stored and survives a refresh, and a worker killed
between stages is recovered without publishing a stale revision.
Two people asking each other at the same moment both passed the
check-then-insert and created two crossing requests. The repository now
takes a lock on the pair of people and applies the same refusal rule
inside the insert's transaction, so exactly one request is created.
The same request sent twice, crossing requests, an accept racing a
decline, a repeated location question and overlapping roster imports
each leave exactly one outcome against a real Postgres.
Two organizations' events and one person attending both: profiles,
candidates, stored matches, intros, contacts, location requests and
removal never cross from one event to the other, even with ids taken
from it.
Playwright (pinned 1.63.0, approved) drives the built app and worker
against the compose Postgres and Keycloak: organizer setup and import,
invited and uninvited sign-in, profiles, background matches, search,
intro consent, the venue question, isolation from a third attendee,
analytics and closing the event; plus phone sign-in, single-use magic
links, a tampered cookie, sign-out and phone-width layouts. Login codes
come from the app's dev e-mail log. The CI job keeps the report and logs
on failure.
Cookies are read with SESSION_SECRET and then any SESSION_SECRET_PREVIOUS
value, and always sealed with the current one. Terraform keeps two secret
slots and a session_secret_active switch, so a rotation regenerates the
inactive slot and makes it active in one apply. Login codes get their own
OTP_PEPPER so a rotation leaves codes in flight valid.
The worker reports queue backlog, waiting time and permanent failures
once a minute with a heartbeat; every search reports its latency and
failures; every model call reports its outcome. They are written as
CloudWatch embedded metrics (no agent or new service) when
METRICS_NAMESPACE is set. Terraform adds alarms for a stopped worker,
matching and notification backlogs, failed jobs and notifications, slow
or failing search and a sustained model failure rate, plus a dashboard.
Each alarm names its runbook section.
Two workers saving match lists that named each other deadlocked: each
locked its own profile row FOR UPDATE, then its inserts took key-share
locks on the other's row through the foreign keys. Found by the capacity
check (300 attendees, 2 workers). The revision check now locks FOR NO KEY
UPDATE, which still blocks a concurrent profile save but not those
foreign-key locks.
Retrieval took the 50 nearest people and only then dropped intents the
viewer does not welcome, so a mentor whose nearest neighbours were all
mentors got no matches even with a mentee in the event. Found by the
capacity check (1 of 300 attendees). Retrieval now applies the viewer's
welcomed intents in the query, before the limit, for profile matches and
search (part of the shared search cache key; retrieval version r3).
npm run capacity:check creates a throwaway event of the chosen size,
saves every profile, runs the real embed and match workers until the
queue is quiet, runs concurrent searches through the real use case, and
reports time to first match lists, search p50/p95, failures and model
calls per attendee and search against the budgets in the runbook, then
removes everything. It uses a free deterministic embedding by default
(EMBEDDING_PROVIDER=hash, refused in production) or the deployment's real
providers with --providers configured. Locally: 300 attendees matched in
6 s with search p95 83 ms; 1000 attendees in 25 s with search p95 169 ms.
npm run db:verify checks a database is usable: pgvector, every migration
applied from the same file (either line ending), core tables present, and
row counts against a source with --compare. npm run db:restore-rehearsal
dumps the compose database, restores it into a scratch database, verifies
it and removes it, timing each step (locally 1.3 s). CI now refuses a
change to a released migration or a new migration that would break the
previous image (drop, rename, type change, truncate) unless the line is
marked as a deliberate contract step.
The GitHub deploy role could push to the web, Keycloak and migrate image
repositories but not the worker's, while every release pushes a worker
image, so a real deploy would stop at "Build & push worker" with access
denied. The worker repository is now in the push statement.
The deploy workflow takes an environment (production or staging). Staging
is its own Terraform stack (match-maker-staging-*, example variables and
separate state key), its jobs run in the GitHub 'staging' environment and
refuse to start unless that environment names the staging stack, so they
never fall back to production settings. promote_tag copies the images a
staging deploy tested into production unchanged (buildx imagetools, same
account) and then migrates and rolls out as usual; production's deploy
role may read the staging repositories when image_promotion_source is
set. Pushes to main still deploy to production as before.
docs/OPERATIONS.md: the service targets and measured capacity, the
release path through staging and promotion, migration rules and
rollback, what to do for every alarm, database restore with
verification, rotation of each secret, running tools against the private
database, and the launch checklist. The worker image now carries the
operator tools so account commands and the Keycloak sync can run as
one-off tasks inside the VPC.
TypeScript 6 (used by the editor) deprecates baseUrl and reported it on
apps/web and packages/auth (inherited from tsconfig.base.json). Path
aliases do not need it: the entries are now relative to the file that
declares them. Type-checks clean with TypeScript 5.7 and 6.0.
@SURYANSH-RAI
SURYANSH-RAI changed the base branch from dev to feature/durable-matching-jobs November 2, 2026 18:17
@SURYANSH-RAI
SURYANSH-RAI merged commit 4fdb5b5 into feature/durable-matching-jobs Nov 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant