Repository navigation
Feature/production hardening - #13
Merged
Merged
Conversation
- DATABASE_SSL=verify checks the server certificate against DATABASE_CA_CERT or DATABASE_CA_CERT_FILE and refuses to run without one; "require" stays encrypted-but-unverified. ensure-database uses the same rules instead of always skipping verification - Production startup (web and worker) refuses a remote database without verify + CA, and the web app refuses a non-https APP_BASE_URL; local hosts stay exempt so docker-compose still runs - HSTS is sent when APP_BASE_URL is https - Web, worker, migrate and Keycloak images ship Amazon's RDS CA bundle - ensure-roles.ts (run by the migration task as the master user): a runtime login with row-level read/write and no DDL, and a Keycloak login that owns the Keycloak database, including tables an earlier deploy created as master (handed over object by object, since REASSIGN OWNED would also move databases). CONNECT is revoked from PUBLIC on both databases. Re-running rotates passwords - Real-Postgres test of what each login can and cannot do
- An environment = "prod" plan fails unless domain_name and acm_certificate_arn are set, so production never serves over HTTP - /admin on Keycloak is no longer routed publicly: a 404 rule blocks it at the load balancer, with an allow rule only for var.keycloak_admin_cidrs (empty by default). The app's admin API calls already go over Cloud Map inside the VPC - Web and worker connect as app_runtime (row access only), Keycloak as its own keycloak login over JDBC sslmode=verify-full, and only the migration task holds the master login; it runs ensure-roles.ts after migrations to create/rotate the other two - Secrets split into app (web/worker), identity (Keycloak) and migrate (master login and the passwords it sets) - Every task uses DATABASE_SSL=verify with the RDS bundle in the image - deploy.yml pins images onto the family's newest task definition revision instead of the one the service is running, so environment and secret changes applied by Terraform (these, and R09's TRUSTED_PROXY_HOPS) actually reach the services
- Migration 0006: event_participants gains invited_at, invite_count, invite_error, claimed_at and revoked_at; existing profiles count as claimed - Revoked attendees lose event sign-in (isParticipant), page and action access (hasEventAccess, so existing sessions too), matching (participantIds) and every results path (ELIGIBLE_CANDIDATE). Re-importing the roster does not silently restore them - The first signed-in visit to the event records the claim time - sendInvitations e-mails everyone never invited, or one attendee as a resend (at most every 10 minutes, never after claiming or removal); each attempt is recorded, failures with their reason - setAttendeeAccess removes or restores access - Invitation e-mail template and NotificationPort.invitation - Analytics count invited, claimed and removed attendees separately
- Roster page: each attendee's state (not invited, invited, invitation failed with the reason, signed in, profile complete, access removed) with times and counts; "Send invitations" for everyone not yet invited; per-row Resend (after 10 minutes), Remove access, Restore - Organizer actions for invitations and access; both use cases check the caller organizes the event - A signed-in attendee without access is sent to the event page, which says whether the organizer removed them or they are on a different account (sign out and use the invited e-mail or phone), instead of a bare 404 - The event page also records the claim on first visit - Analytics shows invitations sent, claimed accounts and removed count
Session cookies were trusted until they expired, so disabling or deleting someone had no effect on a browser already signed in. - Migration 0007: registered_attendees.disabled_at and sessions_revoked_at - isSessionActive (packages/auth) runs on every request in getSession and in the /login middleware (otherwise a revoked session would loop between /login and the page): the account must exist and not be disabled, the session must be newer than sessions_revoked_at, and the Keycloak user must exist and be enabled (cached 5 minutes per replica; a Keycloak outage does not sign everyone out, the database checks still apply) - Every sign-in path refuses a disabled account (canReturnToPath) - npm run account:disable | account:enable | account:revoke-sessions update the database and the Keycloak user (disable also ends its Keycloak sessions) - Tests: unit rules, and the commands against the real database and Keycloak realm, including that repeated sign-ins keep one identity
The visually hidden "Actions" header added to the roster table in R12 is absolutely positioned, and the table's scroll wrapper was not, so the header escaped the wrapper and widened the page by 175px on a phone. The wrapper is now the positioning context.
…(R13 part 2) - OTP then Google hit Keycloak's default first-login flow, which asks the existing account (created by e-mail sign-in, no password) to confirm with a password: a dead end. keycloak:sync now installs "mr-bridge link existing account" (create the user if the e-mail is new, otherwise link it; no review-profile form) and binds the Google provider to it. Google addresses are verified (trustEmail), and the app still rejects unverified ones. Re-running changes nothing - Sign out (button and /api/auth/logout) ends the Keycloak session of a Google sign-in through the admin API instead of sending the browser to Keycloak's "Do you want to log out?" page - A mr-bridge login theme (child of keycloak.v2) gives the remaining Keycloak screens the site's colours; shipped in the Keycloak image, mounted in compose, selected by keycloak:sync when installed - Supported shared-domain hostname setup: strict hostname (the KC_HOSTNAME_STRICT=false override is gone) and KC_HOSTNAME_BACKCHANNEL_DYNAMIC for the app's Cloud Map calls - README documents keycloak:sync and the account commands
…ctive (R14) Roles existed but every organizer could do everything, and imports rewrote people's names, phones and details across organizations in batches that could fail halfway. Roles (domain/shared/organizer-permissions.ts): - Organization owners/admins manage every event in the organization - Event owners/admins configure the event, import and remove attendees - An event "organizer" views, sends invitations and starts matching - Use cases authorize per action; pages hide what the role cannot use (read-only settings, no import or remove controls) - Event lists show only events the viewer may open; organizers of one event may open the list (not the new-event page) Imports (events.importRoster): - One transaction with a per-event lock: all applied or none, and retrying the same file changes nothing - Existing people keep their name, phone and details; the sheet only fills empty fields, and the summary says how many were kept - A new person whose phone belongs to someone else is imported without it instead of failing the whole file - The roster page reports added, already on the roster, skipped, kept and phone conflicts
R14 lets organization owners and admins configure every event in their organization, but events.update still required an event_organizers row in its own SQL, so an organization admin who did not create the event got "Event not found" when saving its settings. The query now applies the same rule as the use case. Covered by a real-Postgres test that fails on the old query.
events.form_config existed, but the participant form was fixed. - domain/participant/profile-form.ts: a catalog of the profile fields matching already reads (role, domain, about, focus, build, years, interests). Per event, organizers set each field's label, whether it is shown and whether it is required. A required field must be shown, and at least one descriptive field stays required so matching always has text. Each save bumps the form version - Organizer dashboard: "Profile form" settings (owners/admins); errors keep the edits on screen - The participant form renders from the config, and the server validates against the same config with its labels - Hidden fields keep what was saved before, and every stored field still feeds matching, so changing the form never wipes profiles or changes embeddings - Migration 0008: profile_form_version per participant; someone saved under an older form is asked for newly required answers, while their matches keep working
Search replaced the searcher's profile embedding with the query's, so query relevance and personal compatibility were blurred together, and without embeddings (no API key) retrieval was alphabetical, so every query returned the same people. There were no filters. - domain/matching/search-query.ts: query words, query relevance (semantic when both vectors exist, else word-prefix matches), and a final score of 60% query relevance and 40% personal compatibility; both appear in the score breakdown - The pipeline's search path keeps the searcher's own profile for compatibility and uses the query only for retrieval and relevance, so the same query ranks differently for different people - Without a query embedding, retrieval uses Postgres full-text search (any word, prefix match) ranked by ts_rank - Intent and domain filters, validated against the event's intents and the domains present among eligible participants; applied in SQL and again on shared-cache hits, and part of the cache key with the normalized query and retrieval kind - Search page: filter selects with their own labels
mutualInterest was set from intent compatibility alone, so any two people with compatible goals were shown "Your event goals align" however poorly they fit. Every scorer was also symmetric, so a two-direction rule would have meant nothing. - domain/matching/scoring-config.ts: scorer weights, the mutual interest threshold and a version in one place; the version is part of the persisted computation version (v2.s2), so older results can be found and recomputed - Interests are now directional (how much of what the viewer cares about the candidate shares), so score(A->B) and score(B->A) can differ - ScoreStage computes both directions; the flag needs both to reach the threshold (0.65, calibrated on documented examples). The reverse score is stored in the breakdown - Wording: "Likely a strong fit for each other"; the fallback reason no longer presents intent compatibility as a prediction - Calibration tests: both high flags; compatible goals alone, or a one-way fit, do not; works with and without embeddings
Every rerank asked the model to explain the whole shortlist again, even when nothing about the pairs had changed. - Migration 0009: pair_explanations (cache key, event, viewer, candidate, reason, expiry), deleted with the event - domain/matching/explanation-key.ts: the key covers event, the viewer -> candidate direction, both profile revisions (last saved), model, prompt version, scoring version, and the context: "profile" or the normalized search query - LlmRerankStage reads cached reasons first. When every shortlisted pair has one, the model is not called and the scored order stands: a cached reason never decides a shortlist's order. Otherwise the model ranks as before and its new reasons are cached for a week. If the model fails, cached reasons are used where they exist; a cache outage never blocks matching - ExplanationCache port with a Postgres adapter; LlmProvider exposes a prompt version; the hourly sweep removes expired explanations
- A declined request is final for the event: re-requesting used to reset it to pending, letting someone keep asking - If the other person already asked, the requester is pointed to their request instead of creating a second one; accepted pairs are refused - At most 20 intro requests an hour and 25 waiting for an answer per person and event; self-requests are refused in the use case too - An attendee removed from the event disappears from other people's connections and their contact is no longer shown; restoring access brings both back. Accepting a request from a removed person fails - Participants can hide their profile from matches and search; hidden, they keep browsing and their accepted connections (consent already given), and nobody new can request them - Tests: request rules, abuse limits, and real-Postgres checks of decline finality and removal
- Migration 0010: audit_events (event, actor, action, subject, details with counts and field names only). Recorded: roster imports, invitations, access removed/restored, settings and profile-form changes, introductions accepted (contacts shared), and account disable/enable/revoke/erase. Recording failures are logged and never block the action. Organizers see "Recent activity" on the analytics page. Records are kept for a year by the hourly sweep and survive their event's deletion - Removing someone from an event deletes cached explanations written about or for them; hiding a profile deletes explanations of it - npm run account:erase -- <email> --yes for deletion requests: deletes the Keycloak user first (so a failure leaves data to retry), then the attendee and everything cascading from it, cached explanations and login challenges; refuses without --yes - Tests: real Postgres for records, sweep, retention on removal and hiding, and erasure against the local Keycloak realm
Concurrent identical searches each embedded the query and retrieved candidates; the cache key lacked the corpus state, embedding space and limit, and a profile change could serve candidates from the old corpus. - Migration 0011: cache_locks (expiring, owner-only release) and events.corpus_revision, bumped by statement-level triggers whenever an event's participants change (one bump per statement, so large imports stay cheap) - application/shared-computation.ts computeOnce: the first caller takes the lock and computes; others wait up to 2 seconds for its result, then compute themselves. Cache or lock failures fall back to computing directly, so an outage costs work but never fails a search - Retrieval keys: retrieval version, event, corpus revision, normalized query, filters, embedding provider/model (or lexical) and limit; results are re-checked for eligibility and filters after lookup - Query embeddings are cached per provider/model and normalized query for a day and coalesced the same way - CandidateCache gains vectors and locks in both the Postgres and Upstash adapters (Redis SET NX PX and compare-and-delete); embedding providers expose their id; the sweep clears expired locks - Tests: coalescing, waiting, fallbacks; real Postgres locks, vectors, and two pipeline instances coalescing one search while ranking personally, with fresh retrieval after a corpus or filter change
Profile saves and their jobs were separate writes, a running job made later edits get dropped, a crashed worker's job stayed "running", and a slow job could publish results for an older profile. - Migration 0012: profile_revision per participant; jobs carry the revision they reflect, a lease, the last error and why they exist (own profile change or refresh). One job per person and kind may be pending; a new edit updates it instead of waiting behind a running one - saveProfile bumps the revision and queues the embedding in the same transaction - Embeddings and match lists are written only while the profile is still at the job's revision (the match write locks the row); a stale run completes quietly and the newer job publishes - Claims take a 5-minute lease; an expired lease is reclaimed, or failed for good if it was the last attempt. Failures retry after 30s, 60s, 120s... (capped at 15 minutes) up to 5 attempts, then stay failed with the error kept; a newer pending job supersedes a failing one. Replaces R08's stuck-job workaround in enqueue - listFailed / retryFailed for operators; claim can be scoped to one event (used by tests so they never claim other local jobs) - Real-Postgres tests for each of these
Other people's saved matches never changed when someone joined or edited their profile, so the first participant to save had no matches until they pressed refresh. - After a participant's own profile change is matched, the worker queues refreshes for the people it affects: those whose saved matches include them, and those whose list is not full. Refreshes wait MATCH_REFRESH_DELAY_SECONDS (default 120) so bursts coalesce into one run per person, keep the latest revision, and never trigger further refreshes; a pending profile job stays a profile job - npm run jobs:failed lists jobs that failed after every retry with their last error; npm run jobs:retry -- <id|all> queues them again - Tests: who gets refreshed (and who does not), coalescing, no cascade; browser check that the first participant receives later arrivals without pressing anything, and that rapid edits end on the latest revision
CircuitBreaker and Bulkhead were empty classes, and provider calls only had their own timeouts: no shared budget, concurrency limit, retry policy, fallback or usage accounting. - CircuitBreaker (opens after 5 failures, one trial call after 30s) and Bulkhead (bounded concurrency and queue per provider) - model-gateway.ts wraps both ports: per-operation time budgets that cover retries, abort signals so a timed-out request really stops, one or two jittered retries for 429/5xx/timeouts/network errors only - Fallbacks: a failed query embedding returns null (search matches by words); a failed profile embedding throws (the job's own backoff retries later); a failed rerank keeps the scored order - Search has a 6-second total budget split between query embedding and rerank; a rerank is skipped when under 750 ms remain - Migration 0013: model_usage records every call (provider, operation, outcome, latency, characters), kept 90 days by the sweep
- Unit tests: circuit breaker open/half-open/close, bulkhead limits and queue timeout, transient-only retries with every attempt recorded, budget timeouts, query vs profile fallbacks, circuit stopping calls, and rerank skipped when the request is nearly out of time - npm run models:usage [-- <hours>] reports calls, failures, average and p95 latency, approximate tokens and, with MODEL_PRICE_PER_1K_TOKENS, estimated cost per provider and operation
Vectors from different providers or models live in different spaces, but nothing recorded which model produced a stored embedding, so switching providers would compare unrelated vectors. - event_participants.embedding_model (migration 0013) is written with every embedding (embedding_model of the gateway's provider id) - Retrieval ranks by vector only among candidates embedded by the query's model; others keep their place by text relevance - The semantic scorer and query relevance compare vectors only within one model, falling back to text otherwise - A profile embedded by another model (or before this change) is embedded again with the current one on its next matching run - Tests: scorer and relevance fallbacks, re-embedding, and real-Postgres retrieval that ignores an exact match from another model
Notification e-mails were sent inline after the action and a failure was only logged, so a mail outage lost the message. - Migration 0014: notification_outbox (unique dedupe key, attempts, lease, last error) and location_requests (used by part 2); the sweep removes sent notifications after 30 days and location requests after a day - An intro request and its notification, and an acceptance and its notice to the requester, commit in one transaction - The worker delivers queued notifications with a 2-minute lease and backoff retries (30s ... 15 min, 5 attempts); details are read at send time and an interaction that no longer applies is closed without sending; replays never send twice - E-mails carry a deep link to the connection on the Connections page; the notifier gains location-request and reply messages - Real-Postgres tests: queued once with the request, delivered with the link, acceptance notice, and a failed send kept for retry
…rt 2) "Ping zone" only overwrote the sender's own zone for every connection; there was no way to ask a connection where they are, and no reply. - domain/connection/location.rules.ts: a request is a question, not tracking. One open request per direction of an accepted connection; unanswered requests expire after 30 minutes; a shared zone is visible only to the person who asked, for 15 minutes; 10 requests per person and hour - LocationRepository (Postgres): creating a request and answering it (a configured zone, or "not now") each queue their notification in the same transaction, so the interaction is never lost - Connections page: "Ask where they are", an incoming request with "Share with <name>" / "Not now", the reply or decline for the asker, and the existing broadcast labelled "Share with all my connections"; each connection has an anchor for e-mail deep links - Tests: state and rules, the use cases, real-Postgres visibility, expiry and replacement, and a browser run where the worker e-mails the request, acceptance, location request and reply
Integration tests skip themselves without Postgres or Keycloak, which let CI pass with the Keycloak suites never running. CI now starts the compose Postgres and Keycloak, syncs the realm, sets REQUIRE_SERVICES so a global setup refuses to start without them, and checks the JSON report so any skipped test fails the job. Workers accept an optional event scope for tests.
When every explanation of a shortlist was already cached the model was not asked again and the list fell back to score order, so a refresh that changed nothing reshuffled a stored, reranked list. The model's order is now kept for the exact shortlist (keyed by all of its pair keys) and reused on a full cache hit; erasure, removal, hiding and housekeeping clear it with the explanations.
Runs the real embed and match workers against Postgres with deterministic fake providers: the first person to save gets matches once others arrive, the reranked order is stored and survives a refresh, and a worker killed between stages is recovered without publishing a stale revision.
Two people asking each other at the same moment both passed the check-then-insert and created two crossing requests. The repository now takes a lock on the pair of people and applies the same refusal rule inside the insert's transaction, so exactly one request is created.
The same request sent twice, crossing requests, an accept racing a decline, a repeated location question and overlapping roster imports each leave exactly one outcome against a real Postgres.
Two organizations' events and one person attending both: profiles, candidates, stored matches, intros, contacts, location requests and removal never cross from one event to the other, even with ids taken from it.
Playwright (pinned 1.63.0, approved) drives the built app and worker against the compose Postgres and Keycloak: organizer setup and import, invited and uninvited sign-in, profiles, background matches, search, intro consent, the venue question, isolation from a third attendee, analytics and closing the event; plus phone sign-in, single-use magic links, a tampered cookie, sign-out and phone-width layouts. Login codes come from the app's dev e-mail log. The CI job keeps the report and logs on failure.
Cookies are read with SESSION_SECRET and then any SESSION_SECRET_PREVIOUS value, and always sealed with the current one. Terraform keeps two secret slots and a session_secret_active switch, so a rotation regenerates the inactive slot and makes it active in one apply. Login codes get their own OTP_PEPPER so a rotation leaves codes in flight valid.
The worker reports queue backlog, waiting time and permanent failures once a minute with a heartbeat; every search reports its latency and failures; every model call reports its outcome. They are written as CloudWatch embedded metrics (no agent or new service) when METRICS_NAMESPACE is set. Terraform adds alarms for a stopped worker, matching and notification backlogs, failed jobs and notifications, slow or failing search and a sustained model failure rate, plus a dashboard. Each alarm names its runbook section.
Two workers saving match lists that named each other deadlocked: each locked its own profile row FOR UPDATE, then its inserts took key-share locks on the other's row through the foreign keys. Found by the capacity check (300 attendees, 2 workers). The revision check now locks FOR NO KEY UPDATE, which still blocks a concurrent profile save but not those foreign-key locks.
Retrieval took the 50 nearest people and only then dropped intents the viewer does not welcome, so a mentor whose nearest neighbours were all mentors got no matches even with a mentee in the event. Found by the capacity check (1 of 300 attendees). Retrieval now applies the viewer's welcomed intents in the query, before the limit, for profile matches and search (part of the shared search cache key; retrieval version r3).
npm run capacity:check creates a throwaway event of the chosen size, saves every profile, runs the real embed and match workers until the queue is quiet, runs concurrent searches through the real use case, and reports time to first match lists, search p50/p95, failures and model calls per attendee and search against the budgets in the runbook, then removes everything. It uses a free deterministic embedding by default (EMBEDDING_PROVIDER=hash, refused in production) or the deployment's real providers with --providers configured. Locally: 300 attendees matched in 6 s with search p95 83 ms; 1000 attendees in 25 s with search p95 169 ms.
npm run db:verify checks a database is usable: pgvector, every migration applied from the same file (either line ending), core tables present, and row counts against a source with --compare. npm run db:restore-rehearsal dumps the compose database, restores it into a scratch database, verifies it and removes it, timing each step (locally 1.3 s). CI now refuses a change to a released migration or a new migration that would break the previous image (drop, rename, type change, truncate) unless the line is marked as a deliberate contract step.
The GitHub deploy role could push to the web, Keycloak and migrate image repositories but not the worker's, while every release pushes a worker image, so a real deploy would stop at "Build & push worker" with access denied. The worker repository is now in the push statement.
The deploy workflow takes an environment (production or staging). Staging is its own Terraform stack (match-maker-staging-*, example variables and separate state key), its jobs run in the GitHub 'staging' environment and refuse to start unless that environment names the staging stack, so they never fall back to production settings. promote_tag copies the images a staging deploy tested into production unchanged (buildx imagetools, same account) and then migrates and rolls out as usual; production's deploy role may read the staging repositories when image_promotion_source is set. Pushes to main still deploy to production as before.
docs/OPERATIONS.md: the service targets and measured capacity, the release path through staging and promotion, migration rules and rollback, what to do for every alarm, database restore with verification, rotation of each secret, running tools against the private database, and the launch checklist. The worker image now carries the operator tools so account commands and the Keycloak sync can run as one-off tasks inside the VPC.
TypeScript 6 (used by the editor) deprecates baseUrl and reported it on apps/web and packages/auth (inherited from tsconfig.base.json). Path aliases do not need it: the entries are now relative to the file that declares them. Type-checks clean with TypeScript 5.7 and 6.0.
Feature/operations readiness
Feature/browser acceptance suite
Feature/ci acceptance coverage
…requests Feature/notifications location requests
Feature/model gateway
Feature/durable matching jobs
Feature/shared search retrieval
Feature/privacy abuse controls
Feature/pair explanation cache
Feature/mutual interest scoring
Feature/query aware search
Feature/profile form config
Feature/organizer roles import
Feature/identity sessions
Feature/invitation lifecycle
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
R11: Production transport, Keycloak admin and database credentials
Production can no longer silently run with insecure transport or one shared master database login.
Decisions (recommended options):
/adminat the load balancer, except an IP allowlist that is empty by default.Changes:
DATABASE_SSL=verifychecks the certificate againstDATABASE_CA_CERT/DATABASE_CA_CERT_FILEand refuses to run without one.APP_BASE_URL. Local hosts are exempt, so docker-compose still works. HSTS is sent on https.sslmode=verify-full.ensure-roles.ts(run by the migrate task as master): createsapp_runtime(row access, no DDL) andkeycloak(owns only its database; older tables are handed over object by object, not via REASSIGN OWNED). CONNECT is revoked from PUBLIC. Re-running rotates passwords.environment = "prod"plan fails withoutdomain_nameandacm_certificate_arn;/admin→ 404 exceptkeycloak_admin_cidrs;app/identity/migrate;DATABASE_SSL=verify.deploy.ymlnow builds on the family's newest task definition revision, so Terraform's env/secret changes (these, and R09'sTRUSTED_PROXY_HOPS) actually ship.Only a real apply/deploy can prove:
Deploy notes:
keycloak_admin_cidrsif the console is needed.domain_nameandacm_certificate_arn.