Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@
against 25.7 / 50.0 and 265.0 / 284.4 ms with the gate off. `--keep` leaves the table for
`EXPLAIN`.

- **`SearchTrace` on the SQL backends, and a `TraceStage.Gate`.** The PostgreSQL engines now record
the `Score` stage for the page they return — `Postgres lexical`, `Postgres BM25`, `pgvector`,
`sparsevec`, `pg_trgm`, `ParadeDB BM25` — where they previously ignored `SearchOptions.Trace`
entirely. The stock-PostgreSQL BM25 engine also records one query-level `Gate` step naming the
rule that chose its candidate set (the rare terms and their budget, `phrase gate`, or the whole
query) and the candidate count, so a result missing because no gate term matched is
distinguishable from one that scored badly. `SearchTrace.RecordScoreStage` is public for the same
reason: a custom engine outside the core assembly needs to report the stage under the same name.

### Changed

- The shared documents table's upsert and row reader are now one implementation
Expand Down
4 changes: 3 additions & 1 deletion docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,9 @@ a document holding none of the gated terms cannot be returned by any score.
Two fallbacks keep it honest. When even the rarest term exceeds the budget, the thread falls back to
the whole match set, so a corpus of common terms keeps the previous behaviour. And a quoted phrase
bypasses the gate entirely: the phrase is the query's contract, and a disjunction of terms must not
stand in for it. `CandidatePool` still bounds how many of the gate's candidates are read (default
stand in for it. The choice is recorded as a `TraceStage.Gate` step — the terms, the budget and the
candidate count — when `SearchOptions.Trace` is set; see
[observability](observability.md#the-candidate-gate-and-the-sql-backends). `CandidatePool` still bounds how many of the gate's candidates are read (default
1,000); with the gate on it rarely binds.

**What it buys, in quality.** nDCG@10 over the same corpora, queries and qrels:
Expand Down
26 changes: 24 additions & 2 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ allocates and records nothing.

`Explain` breaks down one scorer's arithmetic. `SearchTrace` goes further: it records **every stage
a document passed through**, so a final rank can be walked back stage by stage — which engine
scored it, what each merger contributed, what the reranker changed, what the boost did, and which
route ran.
scored it, what each merger contributed, what the reranker changed, what the boost did, which
route ran, and — on the PostgreSQL backends — which candidate set the query was scored over.

```csharp
using LexiSharp.Core;
Expand Down Expand Up @@ -72,6 +72,28 @@ allocates **exactly the same bytes** with and without this row (60.1 KB per sear
nothing below the `trace is null` guard executes; the traced path allocates **1.4 KB more**
(71.0 → 72.4 KB), one step and one string per search against the ten already written per document.

### The candidate gate, and the SQL backends

The PostgreSQL engines record the same `Score` stage as the in-memory ones, under a name that says
where the score came from — `Postgres lexical`, `Postgres BM25`, `pgvector`, `sparsevec`, `pg_trgm`
or `ParadeDB BM25` — so a hybrid pipeline that mixes a SQL channel with an in-memory one explains
both halves the same way. The page is what is recorded: the stage is bounded by `Limit`, never by
the corpus.

The stock-PostgreSQL BM25 engine adds one query-level `Gate` row, with an empty document id and
`NaN` scores, before any document row:

```
// gate NaN -> NaN rare-term gate [brown, quick] under a 1,000-row budget; 120 candidates
```

It names the rule that chose the candidate set — the terms the rare-term gate matched on, or
`all terms (gate disabled)`, `phrase gate`, `all terms (no term within a 1,000-row budget)` — and
how many candidates it produced. That is the one decision on this backend a reader cannot
reconstruct from the page: the score is `Bm25Scorer`'s, but *which* documents were eligible to be
scored is the gate's, and a result that is missing from the page because no rare term matched it is
otherwise indistinguishable from one that scored badly.

**Cost — what is measured and what is not.** Measured with BenchmarkDotNet's `MemoryDiagnoser` and
stable across repeated runs: a saturated trace allocates **0 additional bytes** per search
(1.34 KB either way), and a trace created per search costs **~0.47 KB** (the object plus its
Expand Down
4 changes: 3 additions & 1 deletion src/LexiSharp.Postgres/ParadeDB/ParadeDBTextSearchEngine.cs
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,9 @@ ORDER BY pdb.score(id) DESC, {PostgresSchema.OrdinalIdTieBreak}
results.Add(new SearchResult(document.Id, score, document));
}

return results.Skip(options.Offset).Take(options.Limit).ToList();
var page = results.Skip(options.Offset).Take(options.Limit).ToList();
options.Trace?.RecordScoreStage("ParadeDB BM25", page);
return page;
}

private static bool PassesFilters(SearchOptions options, SearchDocument document)
Expand Down
73 changes: 62 additions & 11 deletions src/LexiSharp.Postgres/PostgresBm25SearchEngine.cs
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
using System.Globalization;

using LexiSharp.Core;
using LexiSharp.Ranking;
using Npgsql;
Expand Down Expand Up @@ -248,10 +250,27 @@ public IReadOnlyList<SearchResult> Search(
var statistics = _statistics ??= PostgresBm25Statistics.Load(connection, _indexOptions);
var match = PostgresMatchExpression.Build(_indexOptions, query);

using var command = CreateCandidatesCommand(connection, match, statistics, queryTerms, options);
string[] gateTerms = GateTermsFor(match, statistics, queryTerms);

using var command = CreateCandidatesCommand(connection, match, gateTerms, options);

List<PostgresBm25Candidate> candidates = ReadCandidates(command, keptTerms, cancellationToken);

// The gate is a query-level decision, so it is one step with no document id: which rule was
// applied, and how many candidates it produced. Recorded even when the count is zero,
// because "the gate matched nothing" is the answer a reader is looking for.
if (options.Trace is not null)
{
options.Trace.Record(new TraceStep(
TraceStage.Gate,
string.Empty,
double.NaN,
double.NaN,
string.Create(
CultureInfo.InvariantCulture,
$"{GateDescription(match, gateTerms)}; {candidates.Count:N0} candidates")));
}

if (candidates.Count == 0)
return Array.Empty<SearchResult>();

Expand All @@ -268,7 +287,11 @@ public IReadOnlyList<SearchResult> Search(

Dictionary<string, SearchDocument> documents = ReadDocuments(connection, page, cancellationToken);

return BuildResults(page, documents);
List<SearchResult> results = BuildResults(page, documents);

options.Trace?.RecordScoreStage("Postgres BM25", results);

return results;
}

/// <summary>
Expand All @@ -285,8 +308,7 @@ public IReadOnlyList<SearchResult> Search(
private NpgsqlCommand CreateCandidatesCommand(
NpgsqlConnection connection,
PostgresMatchExpression match,
PostgresBm25Statistics statistics,
string[] queryTerms,
string[] gateTerms,
SearchOptions options)
{
var command = connection.CreateCommand();
Expand All @@ -298,15 +320,9 @@ private NpgsqlCommand CreateCandidatesCommand(
// The candidate gate: the query's rare terms when it has any, the whole query otherwise.
// A GIN index cannot rank, so this is what keeps the scan proportional to the
// discriminative part of the query instead of to every document its common part matches.
//
// Only for a free-text query. A mixed query's gate is the phrase conjunction, and the free
// terms beside it are scoring-only by contract: substituting the rare-term disjunction for
// the phrases would let a document without the phrase in.
string gate = match.Gate;

if (_options.RareTermGate
&& !match.HasPhrases
&& GateTerms(queryTerms, statistics.DocumentFrequency, _options.RareTermGateBudget) is { Length: > 0 } gateTerms)
if (gateTerms.Length > 0)
{
gate = "tsv @@ (SELECT string_agg(quote_literal(term), ' | ') FROM unnest(@gateTerms::text[]) AS term)::tsquery";
command.Parameters.AddWithValue("gateTerms", gateTerms);
Expand Down Expand Up @@ -476,6 +492,41 @@ internal static int PoolSize(PostgresBm25Options options, SearchOptions searchOp
PostgresBm25Options.MaximumCandidatePool);
}

/// <summary>
/// The terms the gate applies to this query, or an empty array when it does not apply.
/// </summary>
/// <remarks>
/// Only for a free-text query. A mixed query's gate is the phrase conjunction, and the free
/// terms beside it are scoring-only by contract: substituting the rare-term disjunction for the
/// phrases would let a document without the phrase in.
/// </remarks>
private string[] GateTermsFor(
PostgresMatchExpression match, PostgresBm25Statistics statistics, string[] queryTerms) =>
_options.RareTermGate && !match.HasPhrases
? GateTerms(queryTerms, statistics.DocumentFrequency, _options.RareTermGateBudget)
: [];

/// <summary>The gate's rule in words, for the trace.</summary>
private string GateDescription(PostgresMatchExpression match, string[] gateTerms)
{
if (gateTerms.Length > 0)
{
return string.Create(
CultureInfo.InvariantCulture,
$"rare-term gate [{string.Join(", ", gateTerms)}] under a {_options.RareTermGateBudget:N0}-row budget");
}

if (match.HasPhrases)
return "phrase gate";

if (!_options.RareTermGate)
return "all terms (gate disabled)";

return string.Create(
CultureInfo.InvariantCulture,
$"all terms (no term within a {_options.RareTermGateBudget:N0}-row budget)");
}

/// <summary>
/// The query's terms the candidate gate matches on: the rarest ones, by ascending document
/// frequency, while their summed document frequency stays inside <paramref name="budget"/>.
Expand Down
4 changes: 3 additions & 1 deletion src/LexiSharp.Postgres/PostgresFuzzySearchEngine.cs
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,9 @@ public IReadOnlyList<SearchResult> Search(string query, SearchOptions? options =
results.Add(new SearchResult(document.Id, score, document));
}

return results.Skip(options.Offset).Take(options.Limit).ToList();
var page = results.Skip(options.Offset).Take(options.Limit).ToList();
options.Trace?.RecordScoreStage("pg_trgm", page);
return page;
}

/// <summary>Releases the underlying Npgsql data source.</summary>
Expand Down
4 changes: 3 additions & 1 deletion src/LexiSharp.Postgres/PostgresSparseSearchEngine.cs
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,9 @@ WHERE sparse IS NOT NULL{filters.Fragment}
}
}

return results.Skip(options.Offset).Take(options.Limit).ToList();
var page = results.Skip(options.Offset).Take(options.Limit).ToList();
options.Trace?.RecordScoreStage("sparsevec", page);
return page;
}

/// <summary>Releases the underlying Npgsql data source.</summary>
Expand Down
4 changes: 3 additions & 1 deletion src/LexiSharp.Postgres/PostgresTextSearchEngine.cs
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,9 @@ public IReadOnlyList<SearchResult> Search(string query, SearchOptions? options =
results.Add(new SearchResult(document.Id, score, document));
}

return results.Skip(options.Offset).Take(options.Limit).ToList();
var page = results.Skip(options.Offset).Take(options.Limit).ToList();
options.Trace?.RecordScoreStage("Postgres lexical", page);
return page;
}

/// <summary>Releases the underlying Npgsql data source.</summary>
Expand Down
2 changes: 2 additions & 0 deletions src/LexiSharp.Postgres/PostgresVectorSearchEngine.cs
Original file line number Diff line number Diff line change
Expand Up @@ -987,6 +987,8 @@ private async Task<SearchOutcome> SearchCoreAsync(
.Select(x => new SearchResult(x.Key, x.Value, documents[x.Key]))
.ToList();

options.Trace?.RecordScoreStage("pgvector", results);

return new SearchOutcome(results, contributions);
}

Expand Down
19 changes: 18 additions & 1 deletion src/LexiSharp/Core/SearchTrace.cs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,17 @@

/// <summary>A router selected a route for the query (per query, not per document).</summary>
Route = 4,

/// <summary>
/// An engine chose the candidate set a query is scored over (per query, not per document).
/// </summary>
/// <remarks>
/// The detail is the rule the engine applied and what it cost: on the stock-PostgreSQL BM25
/// engine, the terms a rare-term gate matched on and the candidates it produced, or the whole
/// query when no gate applied. A backend with no candidate choice beyond its index records
/// nothing here.
/// </remarks>
Gate = 5,
}

/// <summary>
Expand Down Expand Up @@ -126,7 +137,7 @@
{
var counts = new Dictionary<TraceStage, int>();

foreach (TraceStep step in _steps)

Check warning on line 140 in src/LexiSharp/Core/SearchTrace.cs

View workflow job for this annotation

GitHub Actions / SonarCloud analysis

Loop should be simplified by calling Select(step => step.Stage)
{
counts.TryGetValue(step.Stage, out int seen);
counts[step.Stage] = seen + 1;
Expand All @@ -152,9 +163,15 @@
/// Records the <see cref="TraceStage.Score"/> step of a page a scoring engine just ranked.
/// Called with the final page rather than per candidate, so the cost is O(page), not O(corpus).
/// </summary>
/// <remarks>
/// Public because a backend outside the core assembly (the PostgreSQL engines) and a custom
/// <see cref="ITextSearchEngine"/> both need to report the same stage under the same name; the
/// alternative was a friend assembly, which would have kept the seam private to this repository's
/// own packages.
/// </remarks>
/// <param name="scorerName">Name of the scorer that produced the scores, used as the step detail.</param>
/// <param name="page">The ranked page, best first.</param>
internal void RecordScoreStage(string scorerName, IReadOnlyList<SearchResult> page)
public void RecordScoreStage(string scorerName, IReadOnlyList<SearchResult> page)
{
for (int i = 0; i < page.Count; i++)
{
Expand Down
26 changes: 25 additions & 1 deletion tests/LexiSharp.Tests/ParadeDBTextSearchEngineTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -396,4 +396,28 @@ public void Search_WithACancelledToken_Throws()
}
}

}
[SkippableFact]
public void Search_RecordsTheScoreStage()
{
Skip.If(ConnectionString is null, "POSTGRES_TEST_CONNECTION not set.");
Skip.If(!ParadeDBAvailable(ConnectionString!), "pg_search extension unavailable.");

using var engine = NewEngine();

try
{
engine.Add(new SearchDocument("a", "quick brown fox"));

var trace = new SearchTrace();
var results = engine.Search("quick", new SearchOptions(Limit: 10, Trace: trace));

Assert.NotEmpty(results);
Assert.Equal(results.Count, trace.Steps.Count(step => step.Stage == TraceStage.Score));
Assert.All(trace.Steps, step => Assert.Equal("ParadeDB BM25", step.Detail));
}
finally
{
engine.DropSchema();
}
}
}
41 changes: 41 additions & 0 deletions tests/LexiSharp.Tests/PostgresBm25SearchEngineTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -547,4 +547,45 @@ public void Constructor_RefusesInvalidRankingParameters()
Assert.Throws<ArgumentException>(() => new PostgresBm25SearchEngine(
connection, new PostgresBm25Options { CandidatePool = -1 }, autoCreateSchema: false));
}

[SkippableFact]
public void Search_RecordsTheGateAndTheScoreStage()
{
Skip.If(ConnectionString is null, "POSTGRES_TEST_CONNECTION not set.");

Run("trace", engine =>
{
engine.Index(Corpus);

var trace = new SearchTrace();
var results = engine.Search("quick brown", new SearchOptions(Limit: 10, Trace: trace));

TraceStep gate = trace.Steps.Single(step => step.Stage == TraceStage.Gate);
Assert.Equal(string.Empty, gate.DocumentId);
Assert.True(double.IsNaN(gate.Before));
Assert.Contains("rare-term gate", gate.Detail!, StringComparison.Ordinal);
Assert.Contains("candidates", gate.Detail!, StringComparison.Ordinal);

var scored = trace.Steps.Where(step => step.Stage == TraceStage.Score).ToList();
Assert.Equal(results.Count, scored.Count);
Assert.All(scored, step => Assert.Equal("Postgres BM25", step.Detail));
});
}

[SkippableFact]
public void Search_RecordsWhyTheGateWasNotApplied()
{
Skip.If(ConnectionString is null, "POSTGRES_TEST_CONNECTION not set.");

Run("tracenotgate", engine =>
{
engine.Index(Corpus);

var trace = new SearchTrace();
engine.Search("quick brown", new SearchOptions(Limit: 10, Trace: trace));

TraceStep gate = trace.Steps.Single(step => step.Stage == TraceStage.Gate);
Assert.Contains("gate disabled", gate.Detail!, StringComparison.Ordinal);
}, options => options with { RareTermGate = false });
}
}
17 changes: 16 additions & 1 deletion tests/LexiSharp.Tests/PostgresFuzzySearchEngineTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -313,4 +313,19 @@ public void Search_WithACancelledToken_Throws()
});
}

}
[SkippableFact]
public void Search_RecordsTheScoreStage()
{
Run(engine =>
{
engine.Add(Doc("a", "classic rock"));

var trace = new SearchTrace();
var results = engine.Search("classic", new SearchOptions(Limit: 10, Trace: trace));

Assert.NotEmpty(results);
Assert.Equal(results.Count, trace.Steps.Count(step => step.Stage == TraceStage.Score));
Assert.All(trace.Steps, step => Assert.Equal("pg_trgm", step.Detail));
});
}
}
21 changes: 20 additions & 1 deletion tests/LexiSharp.Tests/PostgresSparseSearchEngineTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -346,4 +346,23 @@ public void Search_FiltersGateResults()
Filters: new[] { new MetadataFilter("kind", MetadataFilterOperator.Equal, "nope") })));
});
}
}

[SkippableFact]
public void Search_RecordsTheScoreStage()
{
Skip.If(ConnectionString is null, "POSTGRES_TEST_CONNECTION not set.");
Skip.If(!VectorExtensionAvailable(), "vector extension unavailable.");

Run(engine =>
{
engine.Add(Doc("red", "red apple"));

var trace = new SearchTrace();
var results = engine.Search("red apple", new SearchOptions(Limit: 10, Trace: trace));

Assert.NotEmpty(results);
Assert.Equal(results.Count, trace.Steps.Count(step => step.Stage == TraceStage.Score));
Assert.All(trace.Steps, step => Assert.Equal("sparsevec", step.Detail));
});
}
}
Loading
Loading