Skip to content

Replace SQLite row indexes with bounded PostgreSQL access - #51

Merged
jay-m-dev merged 6 commits into
developfrom
feat/69-postgresql-row-indexes
Nov 10, 2026
Merged

jay-m-dev merged 6 commits into
developfrom
feat/69-postgresql-row-indexes

Conversation

@jay-m-dev

@jay-m-dev jay-m-dev commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Replace filesystem SQLite row indexes with dataset-owned PostgreSQL metadata and indexed rows while retaining original CSVs, splits and workspace files as authoritative managed artifacts.

Indexes prepare on demand in bounded bulk transactions, preserve exact strings (including NUL-safe binary headers/values), and publish atomically under source locks. Async bounded reads now cover planning, resume, previews, schema preparation and summaries. Saved jobs resolve dataset UUID/hash/parser identities; each job tick verifies original bytes before progress locks and reuses evidence only while its file fingerprint agrees. Registration locks protect active legacy jobs against deletion races.

Operator migration: after coordinated backups and stopping writers, run python -m asaree.row_index_migration inventory, rebuild, and verify with explicit previous --legacy-dir directories and private --report files. The tool compares originals, legacy rows and PostgreSQL in batches and reports missing/inconsistent evidence. Optional retire re-verifies and renames matched SQLite files to .sqlite.retired; locks, unknown files and source artifacts remain retained. Complete existing #50 relocation plans with their matching image/config before upgrading. Roll back using the coordinated pre-cutover backup and old image/config; new job descriptors are not compatible with the previous SQLite reader. Full procedure and pre-existing execution JSONB NUL limits are documented in docs/managed-storage.md.

Validation: CI run 42745154628 passed on f9db7ca with 2014 regression tests passed, 3 skipped and 14 statistical tests deselected. PostgreSQL ingestion, concurrency, cancellation, deletion-race and legacy-job migration checks passed. The frontend and separate statistical-validation step were skipped by scope. Full mypy remains warn-only (693 findings across 57 files); one existing statistical convergence warning was reported. Locally, 138 focused tests passed and 57 PostgreSQL/opt-in checks skipped without services; Python lint, focused typing for the new index/retention/migration/job/schema modules, migration graph/offline SQL checks and git diff --check passed. The first CI run exposed an obsolete fictional-source batching fixture, corrected to use a real owned registration and verified CSV. No Docker, browser automation, dependency updates or expensive simulations were run locally.

Addresses EpistasisLab/ASAREE_Issues#69. Does not implement #55 or change Motoro's database.

@jay-m-dev
jay-m-dev marked this pull request as ready for review November 10, 2026 02:56
@jay-m-dev
jay-m-dev merged commit 30b7969 into develop Nov 10, 2026
3 checks passed
@jay-m-dev
jay-m-dev deleted the feat/69-postgresql-row-indexes branch November 10, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant