Repository navigation
Add BM25 over stock PostgreSQL, with a rare-term candidate gate - #77
Merged
Merged
Conversation
pg_search is the faster route to BM25 on PostgreSQL, but it needs a
preloaded library and none of RDS, Cloud SQL or Azure Database for
PostgreSQL lets you install one, so the only BM25 backend was
unavailable on managed deployments.
PostgresBm25SearchEngine ranks the same tsvector term frequencies
with Bm25Scorer, over the lexical engine's table, DDL and analysis.
N, the average document length and each term's document frequency
come from one ts_stat snapshot, invalidated by writes and rebuildable
with RefreshStatistics(); term frequencies and the length come from
the candidate row's own tsvector. PostgreSQL selects the best
CandidatePool matches by ts_rank_cd (default 1,000, 0 for the exact
ranking up to the engine's 100,000-row ceiling) and the engine
re-ranks them, because scoring every match would make a query
proportional to the corpus.
PostgresIndexOptions.MatchMode separates the other half of ts_rank_cd's
gap: AllTerms is websearch_to_tsquery's AND (the previous behaviour),
AnyTerm ORs the query's lexemes. The gate and the rank expression now
come from one PostgresMatchExpression, and the documents upsert from
one PostgresDocumentWriter, so the two text engines cannot drift.
Measured by the harness on the same corpora, queries and qrels
(nDCG@10, harness defaults):
all any pool 1k pool 0 in-mem ParadeDB
NFCorpus 0.1864 0.1881 0.2999 0.3072 0.308 0.3084
SciFact 0.0188 0.0988 0.5636 0.6665 0.662 0.6619
At pool 0 the portable BM25 reproduces the in-memory row to within the
analysis. The default pool is the whole 0.103-point SciFact difference,
which is why --backend-pool exists and why the row prints the pool it
ran with.
The evaluation page had attributed ts_rank_cd's NFCorpus gap to its
matching rule; OR-ing the terms moves NFCorpus 0.1864 -> 0.1881, so
that gap is the ranker (coverage has no idf). The rows behind both
statements are now published, and the backend guide carries a
managed-PostgreSQL matrix: contrib and pgvector are available,
pg_search is not.
…phases The quality harness reports no latency on purpose, so the portable BM25's cost was a claim -- "bounded by the pool" -- without a number. A `postgres` lane measures it: index build, ts_stat build and p50/p95/p99/mean per engine and per pool size, interleaved per query so a pool size does not inherit the previous row's cache state. Measured on NFCorpus (3,633 documents) and SciFact (5,183), one process, one thread, a pooled data source, a page of 10: corpus engine stats p50 p95 mean NFCorpus ts_rank_cd all - 0.6 1.3 0.7 NFCorpus ts_rank_cd any - 4.1 74.6 21.3 NFCorpus bm25 pool 1,000 220ms 16.0 107.3 37.3 NFCorpus bm25 pool 0 201ms 15.8 177.1 58.1 SciFact ts_rank_cd all - 0.7 0.8 0.7 SciFact ts_rank_cd any - 170.1 347.2 177.6 SciFact bm25 pool 1,000 252ms 203.3 385.2 210.7 SciFact bm25 pool 0 246ms 299.7 490.0 301.0 The candidate selection dominates, not the BM25 arithmetic: on SciFact the lexical any-term row alone is 170 ms, because a thirteen-term query matches almost the whole corpus and ts_rank_cd is computed for every match before the pool cut. That is the price of the recall the rule buys, and it is published rather than absorbed. The pool is mostly a quality knob: 500 candidates cost 0.005 nDCG@10 on NFCorpus and 0.077 on SciFact and barely move the median, so 1,000 stays the default. Two changes make the BM25 rows cheaper. The candidate query ranks ids and carries one tsvector per candidate; the page's documents are fetched afterwards by id, so a thousand candidates no longer drag a thousand contents and jsonb rows through a sort PostgreSQL answered by spilling to disk. The per-candidate reader keeps only the query's terms and counts every position for the length without allocating the skipped lexemes. On NFCorpus, pool 1,000: 26.9 -> 16.0 ms at the median and 63.7 -> 37.3 ms at the mean. `--keep` leaves the benchmark table in place for EXPLAIN.
A GIN index has no score order, so the portable BM25 had to rank every match before the pool cut. On SciFact that is 265 ms p50, and worse: the pool then truncated the ranking by ts_rank_cd coverage, reading 0.5636 nDCG@10 against 0.6665 for the exact ranking. The scan was the cost, and the pool was the wrong selector. The gate generates candidates from the query's rarest terms instead: terms are taken by ascending document frequency while their summed document frequency stays inside RareTermGateBudget (1,000 rows by default), and only those matches are ranked. The whole query still scores them. This is what BM25's idf says to do -- a term held by ten documents carries ~6 nats, one held by half the corpus ~0.4 -- and it is a recall bound: a document holding none of the gated terms cannot be returned. It falls back to the whole match set when even the rarest term exceeds the budget, and a quoted phrase bypasses it, because the phrase is the query's contract. Measured on the same corpora, queries and qrels (nDCG@10, pool 1,000): corpus no gate gate 250 gate 500 gate 1,000 exact NFCorpus 0.2999 0.3036 0.3053 0.3072 0.3072 SciFact 0.5636 0.6643 0.6665 0.6665 0.6665 and latencies (p50 / mean, one thread, pooled data source): corpus no gate gate 250 gate 500 gate 1,000 NFCorpus 25.7/50.0 ms 5.0/7.0 ms 7.0/10.5 ms 10.0/17.8 ms SciFact 265.0/284.4 ms 15.7/17.8 ms 31.4/34.3 ms 58.8/63.9 ms At the default budget the gate reads the exact ranking's nDCG on both corpora, 4-9x faster than the ungated scan. The claim is bounded by the evidence: both corpora are scientific and entity-rich, which is the gate's best case, and a rare term that is a false friend can gate out relevant documents. `--backend-gate off --backend-gate-budget <n>` measures that on another corpus. Tests pin the selection (budget, fallback, deterministic order) and the behaviour (a document matching only common terms is excluded when the gate is on and returned when it is off).
manuc66
force-pushed
the
feat/postgres-bm25-and-match-mode
branch
from
November 16, 2026 22:12
f0902f9 to
9a42ad7
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #77 +/- ##
==========================================
+ Coverage 88.94% 89.18% +0.24%
==========================================
Files 262 270 +8
Lines 20900 21350 +450
Branches 4531 4610 +79
==========================================
+ Hits 18589 19041 +452
+ Misses 1397 1385 -12
- Partials 914 924 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
No behaviour change: the same statements, in named pieces. - `BackendLane.Parse` (55) was one method holding every backend's rule. Each group is now a `Refuse...` method that returns early for the backends it does not concern, so the parse itself reads as a list of the rules applied. This also fixes a stale message: `--backend-gate-df` is `--backend-gate-budget`. - `PostgresBm25SearchEngine.Search` (22) is now the sequence of its steps: build the candidate command, read the candidates, score them, read the page, assemble the results. - `PostgresTsvText.Parse` (26) keeps the loop and hands each lexeme to a reader and a recorder, which is also where the `keep` decision now lives.
The BM25 guide and the benchmark lane both said why the other backend is faster by naming its index. The comparison stands without the name: that backend answers from its own inverted index instead of ranking a candidate set inside PostgreSQL.
… hang The coverage report on the new files was mostly measuring the scorer's path: the pool's other IReadOnlyTextIndex members, the engine's schema and option paths, and the parser's defensive branches were never reached. Writing the tests found a real bug. - An empty quoted lexeme (`'':1`) left the ':' unread, so the outer loop saw it forever: `Parse` did not terminate. The position list is now consumed even for a lexeme that is skipped, and a test pins the case. - A statistics factory gives the pool's view something to be built from without a server, and the view's members are pinned: the snapshot answers, the candidate answers, the absent document throws, and Documents / GetTermPositions throw rather than returning an empty answer. - The engine gains tests for the second EnsureSchema call, EnsureSchemaAsync, autoCreateSchema: false, a query of stop words (no lexeme at all), the minimum score, Clear, and ParseQuerySyntax: false. - The parser gains the unquoted lexeme, the unterminated quote, an escaped lexeme under keep, a malformed position list, and a trailing doubled quote. Line coverage of the five new files under the Postgres/Conformance filter is now 100% except one defensive line in the row reader's json guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Three things, in three commits.
PostgresBm25SearchEngine— Okapi BM25 over stock PostgreSQL, with no extension beyondunaccent. It reads the lexical engine's table, DDL andtsvexpression, and ranks the term frequencies the vector already stores withBm25Scorer. This is the portability gap:pg_searchneeds a preloaded library and no major managed service lets you install one, so the only BM25 backend could not run on RDS, Cloud SQL or Azure.PostgresIndexOptions.MatchMode—AllTerms(websearch_to_tsquery's AND, the previous behaviour) orAnyTerm(the query's lexemes OR-ed). On long queries the AND rule was what buried the corpus: SciFact moves 0.0188 → 0.0988 nDCG@10 (5.3×), while the ranker is still the rest — 0.0988 is far from BM25's 0.662 — and NFCorpus barely moves (0.1864 → 0.1881).A rare-term candidate gate (
RareTermGate, on by default) — a GIN index cannot return rows in score order, so the alternative is to rank every match before the pool cut. The gate generates candidates from the query's rarest terms instead: ascending document frequency while their summed frequency stays insideRareTermGateBudget(1,000 rows by default), scored with the whole query. It falls back to the whole match set when even the rarest term exceeds the budget, and a quoted phrase bypasses it.Measured quality — nDCG@10, pool 1,000
ts_rank_cdallts_rank_cdanyAt the default budget the gate reads the exact ranking's nDCG on both corpora. Without it, the candidates are chosen by
ts_rank_cdcoverage and truncation costs 0.10 on SciFact. The score isBm25Scorer's in every BM25 column; what changes is which documents are candidates.Measured latency — p50 / mean, one thread, pooled data source, page of 10
ts_rank_cdanyThe measurements are reproducible with
bench/LexiSharp.Benchmarks postgres(new lane); the quality rows withbench/LexiSharp.Eval --backend bm25 --backend-gate rare|off --backend-gate-budget <n>.Behaviour change
RareTermGateis on by default. It is a recall bound: a document holding none of the gated terms cannot be returned, however much of the common part of the query it matches.RareTermGate = falserestores the previous (ungated) behaviour, and--backend-gate offmeasures that on another corpus.What this is not
pg_search. The ParadeDB backend answers from its own inverted index instead of ranking a candidate set inside PostgreSQL. What this buys is portability; the latency table is its price.simple+unaccent), so these rows are comparable with the in-memory defaults and not with an aligned (--analyzer english) row.N, the average length anddfare onets_statsnapshot; writes through the engine invalidate it and the next search rebuilds it (RefreshStatistics()for writes from elsewhere). A write/search loop pays ~200-450 ms per corpus of this size.Tests and docs
Bm25Scorer(same scores on an analysis-compatible corpus), pool arithmetic, thetsvectorreader (including the filtered path), the gate selection (budget, fallback, deterministic order) and the gate's behaviour (a document matching only common terms is excluded when on, returned when off).api-docs checkand the public API baseline are regenerated.docs/backends.md(BM25 section, match rule, managed-PostgreSQL matrix),docs/evaluation.md,docs/benchmarks.md(newpostgreslane),docs/index.md,docs/querying.md, the changelog and both READMEs.Commits
Add BM25 over stock PostgreSQL and a tsquery match modeMeasure the stock-PostgreSQL engines and read BM25 candidates in two phasesGenerate BM25 candidates from the query's rare termsSplit the three methods CodeFactor flags as complex— no behaviour change;BackendLane.Parse,PostgresBm25SearchEngine.SearchandPostgresTsvText.Parseare decomposed so the PR's CodeFactor check is green.