Find the answer. Understand why.
A working local search engine for engineering documentation, with a React interface and an inspectable ranking pipeline. Import Markdown, text, or JSON, search by terms or exact phrases, and see the contribution of every term to a result's BM25 score.
The retrieval engine is implemented in Python. SQLite stores documents, term statistics, and positional postings. It does not delegate ranking to SQLite FTS, Elasticsearch, a vector database, or a language model.
Requires Python 3.12+ and Node 22.12+.
git clone https://github.com/Vcode2407/ScoutSearch.git
cd ScoutSearch
python -m venv .venv
.\.venv\Scripts\python -m pip install -r requirements-dev.txt
cd frontend
npm ci
npm run build
cd ..
.\.venv\Scripts\python -m scoutOn macOS/Linux, use .venv/bin/python in place of .\.venv\Scripts\python. Open http://127.0.0.1:8765. The first launch creates data/scout.db and indexes 16 original sample notes. Imported documents remain on disk between restarts. The API and built UI run from the same process and origin. Runtime operation needs no external API or network access.
For the real Python documentation corpus shown in the screenshot, follow the corpus guide. It uses a separate database and preserves the original sample library. Stop an existing server before starting another instance on port 8765.
For frontend development, run the API above, then npm run dev from frontend. Vite proxies /api to the local backend. After a production build changes, reload the page. After a backend change, restart the Python process.
An optional Docker Compose configuration is included with loopback-only host binding. It has not been built or verified in this environment.
docker compose up --build- Persistent positional inverted index with Unicode normalization and case folding.
- BM25 ranking with term-frequency saturation, inverse document frequency, and length normalization.
- AND/OR term queries, mandatory quoted phrases, collection filters, prefix suggestions, stable tie ordering, and pagination.
- Covering-index BM25 aggregation for plain-term queries; positional verification and top-k selection for exact phrases.
- Score explanations, highlighted excerpts, full document reading, and a responsive React/TypeScript interface.
- Markdown/text/JSON import, document creation and editing, deletion, content-digest idempotency, and atomic batch updates.
- Explicit typo recovery backed by a persisted deletion-key index, including adjacent transpositions.
- Optimistic concurrency for editor saves and deletes, with draft retention and export on conflict.
- Bounded HTTP metrics, request IDs, and a reproducible real-document workload with concurrent reads/writes.
- Snapshot-consistent search across document content, postings, and corpus statistics while writers commit.
- Input and request-size limits, parameterized SQL, plain-text rendering, host validation, and cross-origin browser write protection.
- Reproducible benchmark with an independent full-scan scoring baseline and a separate small relevance sanity set.
Use "database migration" for an exact phrase or lease recovery for a term query. In Match any term, all quoted phrases are still mandatory. Terms are matched lexically. There is no stemming, synonym expansion, automatic fuzzy matching, or semantic retrieval. Empty unquoted searches can offer explicit one-edit spelling alternatives; accepting one runs a new lexical search.
Search asycnio TaskGroup in the Python corpus to see spelling recovery. Open a document in two browser windows and save different edits: the stale save is rejected with HTTP 409 and the draft remains available to download. The Benchmarks page includes concurrent HTTP results and a link to live request metrics.
See reliability design, workload, and limitations. The workload creates its own temporary database and loopback server; it never writes to your active library.
python scripts/http_workload.py --requests 400 --concurrency 8Tested on 382 real CPython documentation files, split into 1,654 passages. The pinned corpus, transformations, licenses, checksums, profiling output, and independent scan implementation are reproducible.
| Real documentation workload | Before p50 | After p50 | After p95 |
|---|---|---|---|
| Broad queries | 13.91 ms | 2.42 ms | 3.74 ms |
| Selective queries | 1.02 ms | 1.10 ms | 1.88 ms |
| Exact phrases | 2.04 ms | 2.25 ms | 3.70 ms |
| Collection filters | 9.52 ms | 1.35 ms | 1.96 ms |
Broad p50 improved approximately 5.75× compared with the prior implementation. 27/27 match counts, top-10 rankings, and scores matched the independent full-scan oracle. All 8/8 module-navigation sanity checks returned the expected documentation source first. These checks are not an independent human relevance study.
The change uses covering indexes and native SQL arithmetic for our BM25 formula, then loads content only for the selected results. It does not use a result cache or delegate ranking to SQLite FTS. The in-memory full scan still achieved a slightly lower broad p50 of 2.29 ms on this real corpus, and is faster on filtered queries. The report retains those comparisons.
The original 5,000 synthetic document workload also improved, with broad p50 moving from 22.27 to 4.32 ms. The current synthetic scan baseline was 5.57 ms, and all 28/28 rankings matched. Extra covering indexes increase storage and ingestion work; synthetic index build time was 8.72 seconds versus 5.70 seconds in the earlier run.
The engine reports below are retained measurements of the broad-query optimization revision (before the v0.2 spelling index). They were not rerun or relabeled as v0.2 results. These are warm, single-process local engine measurements recorded November 4, 2026 on Windows 11 / Python 3.12.10. They exclude HTTP, network latency, and concurrent load. Timings vary by machine and are not production SLAs.
- Real corpus setup, methodology, tradeoffs, and attribution.
- Real before report and real after report.
- Navigation checks and corpus manifest.
- Current synthetic report and synthetic report before broad-query optimization.
python scripts/prepare_corpus.py
python scripts/evaluate_real.py --output benchmarks/my-real-run.json --profile
python scripts/benchmark.py --documents 5000 --output benchmarks/my-synthetic-run.jsonThe real evaluator uses its own database; the synthetic benchmark uses a temporary one. Neither modifies your active document library. The benchmark page switches between saved real and synthetic results without running workloads.
The synthetic demo's 10-query relevance sanity set yields MRR@10 and nDCG@10 of 1.000 on 16 notes. Those labels are authored for the demo, not held out or independently judged, and do not establish real-world relevance.
python -m pytest -q
cd frontend
npm run buildThe v0.2 implementation passed 27 backend tests, including an independent scoring oracle, atomic validation, stale posting removal, snapshot isolation during concurrent commits, and concurrent writer statistics. TypeScript checking and the production build passed. Browser verification covered phrase search, reading, creation and editing, benchmark rendering, modal keyboard focus, and desktop/narrow layouts. No browser warning/error logs were observed. See verification notes.
GitHub Actions runs the backend tests followed by the frontend production build on each push. Check the repository Actions tab for the current remote status. No public application deployment or Docker build has been performed.
| Endpoint | Behavior |
|---|---|
GET /api/search?q=lease+recovery&mode=all |
Ranked search, explanations, snapshot generation |
GET /api/suggest?prefix=rec |
Up to six vocabulary prefix completions |
GET /api/corrections?q=asycnio+TaskGroup |
Explicit spelling alternatives for an empty search |
GET /api/metrics |
Process-local route/status counts and latency histograms; no query content |
GET /api/workload |
Saved concurrent HTTP report; never runs load |
GET /api/stats |
Index counts, collections, generation, storage size |
POST /api/documents |
Atomic upsert of up to 100 documents |
GET /api/documents/{id} |
Complete document |
DELETE /api/documents/{id} |
Remove document and postings |
GET /api/benchmark?dataset=python |
Saved real corpus report (synthetic selects the synthetic report); never runs a workload |
GET /api/health |
Health response |
GET /openapi.json |
Machine-readable API schema |
For JSON import, use examples/import.json. A document ID is the stable update key; identical imports are no-ops. Bulk imports without preconditions use last-committed-writer semantics. The editor supplies an expected_digests map for atomic compare-and-save; null requires a new ID. A stale save returns 409 without changing any batch member. Editor deletes supply expected_digest too. Markdown and text imports receive a new ID each time. Deleting a document is permanent in this prototype, so keep your originals.
Read the architecture and tradeoffs, demo and interview guide, and next milestones.
This is a single-node, local developer tool. It has no account system, tenant ACLs, encryption at rest, replication, distributed indexing, multi-user conflict resolution, or public hosting. Keep it bound to loopback. Imported content is displayed as text, not executable Markdown/HTML. It does not crawl source links.
scout/engine.py Tokenization, indexing, query planning, scoring, consistency
scout/api.py HTTP boundary, validation, storage lifecycle, static UI
scout/spelling.py Deletion keys and one-edit candidate verification
scout/metrics.py Bounded HTTP instrumentation without content logging
scripts/http_workload.py Isolated concurrent HTTP workload and integrity checks
scout/sample.py Original synthetic demo notes
frontend/src/ React application and responsive styles
tests/ Core, concurrency, and API regression tests
scripts/benchmark.py Synthetic full-scan oracle and timing
scripts/evaluate_real.py Real-corpus oracle, profiling, and timing
scripts/prepare_corpus.py Pinned documentation download and passage preparation
benchmarks/ Measured results and provenance
docs/ Architecture, verification, demo, and next milestones
- SQLite isolation and WAL snapshot behavior.
- Stanford information retrieval text, BM25.
- FastAPI documentation.
- Vite documentation.
The score uses the non-negative log(1 + (N - df + 0.5)/(df + 0.5)) IDF variant, k1=1.2, and b=0.75. Title and body have equal term weighting, with a positional gap that prevents phrases from crossing their boundary.