Skip to content

Add BM25 over stock PostgreSQL, with a rare-term candidate gate - #77

Merged
manuc66 merged 6 commits into
mainfrom
feat/postgres-bm25-and-match-mode
Nov 16, 2026
Merged

manuc66 merged 6 commits into
mainfrom
feat/postgres-bm25-and-match-mode

Conversation

@manuc66

@manuc66 manuc66 commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

What this adds

Three things, in three commits.

  1. PostgresBm25SearchEngine — Okapi BM25 over stock PostgreSQL, with no extension beyond unaccent. It reads the lexical engine's table, DDL and tsv expression, and ranks the term frequencies the vector already stores with Bm25Scorer. This is the portability gap: pg_search needs a preloaded library and no major managed service lets you install one, so the only BM25 backend could not run on RDS, Cloud SQL or Azure.

  2. PostgresIndexOptions.MatchMode — AllTerms (websearch_to_tsquery's AND, the previous behaviour) or AnyTerm (the query's lexemes OR-ed). On long queries the AND rule was what buried the corpus: SciFact moves 0.0188 → 0.0988 nDCG@10 (5.3×), while the ranker is still the rest — 0.0988 is far from BM25's 0.662 — and NFCorpus barely moves (0.1864 → 0.1881).

  3. A rare-term candidate gate (RareTermGate, on by default) — a GIN index cannot return rows in score order, so the alternative is to rank every match before the pool cut. The gate generates candidates from the query's rarest terms instead: ascending document frequency while their summed frequency stays inside RareTermGateBudget (1,000 rows by default), scored with the whole query. It falls back to the whole match set when even the rarest term exceeds the budget, and a quoted phrase bypasses it.

Measured quality — nDCG@10, pool 1,000

corpus ts_rank_cd all ts_rank_cd any BM25 gate 1,000 BM25 no gate BM25 pool 0 in-memory BM25 ParadeDB
NFCorpus 0.1864 0.1881 0.3072 0.2999 0.3072 0.308 0.3084
SciFact 0.0188 0.0988 0.6665 0.5636 0.6665 0.662 0.6619

At the default budget the gate reads the exact ranking's nDCG on both corpora. Without it, the candidates are chosen by ts_rank_cd coverage and truncation costs 0.10 on SciFact. The score is Bm25Scorer's in every BM25 column; what changes is which documents are candidates.

Measured latency — p50 / mean, one thread, pooled data source, page of 10

corpus ts_rank_cd any BM25 no gate gate 250 gate 500 gate 1,000
NFCorpus (3,633 docs) 6.6 / 28.3 ms 25.7 / 50.0 ms 5.0 / 7.0 ms 7.0 / 10.5 ms 10.0 / 17.8 ms
SciFact (5,183 docs) 219 / 240 ms 265 / 284 ms 15.7 / 17.8 ms 31.4 / 34.3 ms 58.8 / 63.9 ms

The measurements are reproducible with bench/LexiSharp.Benchmarks postgres (new lane); the quality rows with bench/LexiSharp.Eval --backend bm25 --backend-gate rare|off --backend-gate-budget <n>.

Behaviour change

RareTermGate is on by default. It is a recall bound: a document holding none of the gated terms cannot be returned, however much of the common part of the query it matches. RareTermGate = false restores the previous (ungated) behaviour, and --backend-gate off measures that on another corpus.

What this is not

  • Not a speed replacement for pg_search. The ParadeDB backend answers from its own inverted index instead of ranking a candidate set inside PostgreSQL. What this buys is portability; the latency table is its price.
  • Not a generality claim for the gate. Both corpora are scientific and entity-rich, which is the gate's best case. A rare term that is a false friend can gate out relevant documents; that is stated in the docs and measurable with the harness flag.
  • Not an analysis change. The terms on both sides are PostgreSQL's configuration (simple + unaccent), so these rows are comparable with the in-memory defaults and not with an aligned (--analyzer english) row.
  • No incremental statistics. N, the average length and df are one ts_stat snapshot; writes through the engine invalidate it and the next search rebuilds it (RefreshStatistics() for writes from elsewhere). A write/search loop pays ~200-450 ms per corpus of this size.

Tests and docs

  • New: a conformance suite for the engine, parity with the in-memory Bm25Scorer (same scores on an analysis-compatible corpus), pool arithmetic, the tsvector reader (including the filtered path), the gate selection (budget, fallback, deterministic order) and the gate's behaviour (a document matching only common terms is excluded when on, returned when off).
  • Full suite: 2,909 passed, 31 skipped (optional engines/models), 0 failed. api-docs check and the public API baseline are regenerated.
  • docs/backends.md (BM25 section, match rule, managed-PostgreSQL matrix), docs/evaluation.md, docs/benchmarks.md (new postgres lane), docs/index.md, docs/querying.md, the changelog and both READMEs.

Commits

  1. Add BM25 over stock PostgreSQL and a tsquery match mode
  2. Measure the stock-PostgreSQL engines and read BM25 candidates in two phases
  3. Generate BM25 candidates from the query's rare terms
  4. Split the three methods CodeFactor flags as complex — no behaviour change; BackendLane.Parse, PostgresBm25SearchEngine.Search and PostgresTsvText.Parse are decomposed so the PR's CodeFactor check is green.

pg_search is the faster route to BM25 on PostgreSQL, but it needs a
preloaded library and none of RDS, Cloud SQL or Azure Database for
PostgreSQL lets you install one, so the only BM25 backend was
unavailable on managed deployments.

PostgresBm25SearchEngine ranks the same tsvector term frequencies
with Bm25Scorer, over the lexical engine's table, DDL and analysis.
N, the average document length and each term's document frequency
come from one ts_stat snapshot, invalidated by writes and rebuildable
with RefreshStatistics(); term frequencies and the length come from
the candidate row's own tsvector. PostgreSQL selects the best
CandidatePool matches by ts_rank_cd (default 1,000, 0 for the exact
ranking up to the engine's 100,000-row ceiling) and the engine
re-ranks them, because scoring every match would make a query
proportional to the corpus.

PostgresIndexOptions.MatchMode separates the other half of ts_rank_cd's
gap: AllTerms is websearch_to_tsquery's AND (the previous behaviour),
AnyTerm ORs the query's lexemes. The gate and the rank expression now
come from one PostgresMatchExpression, and the documents upsert from
one PostgresDocumentWriter, so the two text engines cannot drift.

Measured by the harness on the same corpora, queries and qrels
(nDCG@10, harness defaults):

                      all    any   pool 1k  pool 0   in-mem  ParadeDB
  NFCorpus         0.1864 0.1881   0.2999   0.3072   0.308    0.3084
  SciFact          0.0188 0.0988   0.5636   0.6665   0.662    0.6619

At pool 0 the portable BM25 reproduces the in-memory row to within the
analysis. The default pool is the whole 0.103-point SciFact difference,
which is why --backend-pool exists and why the row prints the pool it
ran with.

The evaluation page had attributed ts_rank_cd's NFCorpus gap to its
matching rule; OR-ing the terms moves NFCorpus 0.1864 -> 0.1881, so
that gap is the ranker (coverage has no idf). The rows behind both
statements are now published, and the backend guide carries a
managed-PostgreSQL matrix: contrib and pgvector are available,
pg_search is not.
…phases

The quality harness reports no latency on purpose, so the portable BM25's
cost was a claim -- "bounded by the pool" -- without a number. A `postgres`
lane measures it: index build, ts_stat build and p50/p95/p99/mean per
engine and per pool size, interleaved per query so a pool size does not
inherit the previous row's cache state.

Measured on NFCorpus (3,633 documents) and SciFact (5,183), one process,
one thread, a pooled data source, a page of 10:

  corpus    engine              stats   p50     p95     mean
  NFCorpus  ts_rank_cd all        -     0.6     1.3     0.7
  NFCorpus  ts_rank_cd any        -     4.1    74.6    21.3
  NFCorpus  bm25 pool 1,000     220ms  16.0   107.3    37.3
  NFCorpus  bm25 pool 0         201ms  15.8   177.1    58.1
  SciFact   ts_rank_cd all        -     0.7     0.8     0.7
  SciFact   ts_rank_cd any        -   170.1   347.2   177.6
  SciFact   bm25 pool 1,000     252ms 203.3   385.2   210.7
  SciFact   bm25 pool 0         246ms 299.7   490.0   301.0

The candidate selection dominates, not the BM25 arithmetic: on SciFact
the lexical any-term row alone is 170 ms, because a thirteen-term query
matches almost the whole corpus and ts_rank_cd is computed for every
match before the pool cut. That is the price of the recall the rule buys,
and it is published rather than absorbed. The pool is mostly a quality
knob: 500 candidates cost 0.005 nDCG@10 on NFCorpus and 0.077 on SciFact
and barely move the median, so 1,000 stays the default.

Two changes make the BM25 rows cheaper. The candidate query ranks ids and
carries one tsvector per candidate; the page's documents are fetched
afterwards by id, so a thousand candidates no longer drag a thousand
contents and jsonb rows through a sort PostgreSQL answered by spilling to
disk. The per-candidate reader keeps only the query's terms and counts
every position for the length without allocating the skipped lexemes.
On NFCorpus, pool 1,000: 26.9 -> 16.0 ms at the median and 63.7 -> 37.3
ms at the mean.

`--keep` leaves the benchmark table in place for EXPLAIN.
A GIN index has no score order, so the portable BM25 had to rank every match
before the pool cut. On SciFact that is 265 ms p50, and worse: the pool then
truncated the ranking by ts_rank_cd coverage, reading 0.5636 nDCG@10 against
0.6665 for the exact ranking. The scan was the cost, and the pool was the
wrong selector.

The gate generates candidates from the query's rarest terms instead: terms
are taken by ascending document frequency while their summed document
frequency stays inside RareTermGateBudget (1,000 rows by default), and only
those matches are ranked. The whole query still scores them. This is what
BM25's idf says to do -- a term held by ten documents carries ~6 nats, one
held by half the corpus ~0.4 -- and it is a recall bound: a document holding
none of the gated terms cannot be returned. It falls back to the whole match
set when even the rarest term exceeds the budget, and a quoted phrase bypasses
it, because the phrase is the query's contract.

Measured on the same corpora, queries and qrels (nDCG@10, pool 1,000):

  corpus     no gate   gate 250   gate 500   gate 1,000   exact
  NFCorpus    0.2999     0.3036     0.3053       0.3072   0.3072
  SciFact     0.5636     0.6643     0.6665       0.6665   0.6665

and latencies (p50 / mean, one thread, pooled data source):

  corpus     no gate        gate 250      gate 500      gate 1,000
  NFCorpus   25.7/50.0 ms    5.0/7.0 ms    7.0/10.5 ms   10.0/17.8 ms
  SciFact   265.0/284.4 ms  15.7/17.8 ms  31.4/34.3 ms   58.8/63.9 ms

At the default budget the gate reads the exact ranking's nDCG on both corpora,
4-9x faster than the ungated scan. The claim is bounded by the evidence: both
corpora are scientific and entity-rich, which is the gate's best case, and a
rare term that is a false friend can gate out relevant documents.
`--backend-gate off --backend-gate-budget <n>` measures that on another
corpus.

Tests pin the selection (budget, fallback, deterministic order) and the
behaviour (a document matching only common terms is excluded when the gate is
on and returned when it is off).
@manuc66
manuc66 force-pushed the feat/postgres-bm25-and-match-mode branch from f0902f9 to 9a42ad7 Compare November 16, 2026 22:12
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.05068% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.18%. Comparing base (8ab690c) to head (7492995).

Files with missing lines Patch % Lines
src/LexiSharp.Postgres/PostgresBm25SearchEngine.cs 98.11% 0 Missing and 4 partials ⚠️
src/LexiSharp.Postgres/PostgresTsvText.cs 95.87% 0 Missing and 4 partials ⚠️
src/LexiSharp.Postgres/PostgresDocumentStore.cs 95.55% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #77      +/-   ##
==========================================
+ Coverage   88.94%   89.18%   +0.24%     
==========================================
  Files         262      270       +8     
  Lines       20900    21350     +450     
  Branches     4531     4610      +79     
==========================================
+ Hits        18589    19041     +452     
+ Misses       1397     1385      -12     
- Partials      914      924      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

No behaviour change: the same statements, in named pieces.

- `BackendLane.Parse` (55) was one method holding every backend's rule. Each
  group is now a `Refuse...` method that returns early for the backends it does
  not concern, so the parse itself reads as a list of the rules applied. This
  also fixes a stale message: `--backend-gate-df` is `--backend-gate-budget`.
- `PostgresBm25SearchEngine.Search` (22) is now the sequence of its steps:
  build the candidate command, read the candidates, score them, read the page,
  assemble the results.
- `PostgresTsvText.Parse` (26) keeps the loop and hands each lexeme to a reader
  and a recorder, which is also where the `keep` decision now lives.
The BM25 guide and the benchmark lane both said why the other backend is
faster by naming its index. The comparison stands without the name: that
backend answers from its own inverted index instead of ranking a candidate
set inside PostgreSQL.
… hang

The coverage report on the new files was mostly measuring the scorer's path:
the pool's other IReadOnlyTextIndex members, the engine's schema and option
paths, and the parser's defensive branches were never reached. Writing the
tests found a real bug.

- An empty quoted lexeme (`'':1`) left the ':' unread, so the outer loop saw
  it forever: `Parse` did not terminate. The position list is now consumed
  even for a lexeme that is skipped, and a test pins the case.
- A statistics factory gives the pool's view something to be built from
  without a server, and the view's members are pinned: the snapshot answers,
  the candidate answers, the absent document throws, and Documents /
  GetTermPositions throw rather than returning an empty answer.
- The engine gains tests for the second EnsureSchema call, EnsureSchemaAsync,
  autoCreateSchema: false, a query of stop words (no lexeme at all), the
  minimum score, Clear, and ParseQuerySyntax: false.
- The parser gains the unquoted lexeme, the unterminated quote, an escaped
  lexeme under keep, a malformed position list, and a trailing doubled quote.

Line coverage of the five new files under the Postgres/Conformance filter is
now 100% except one defensive line in the row reader's json guard.
@manuc66
manuc66 merged commit 495cb1c into main Nov 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants