Repository navigation
Replace SQLite row indexes with bounded PostgreSQL access - #51
Merged
Merged
Conversation
added 6 commits
November 9, 2026 18:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace filesystem SQLite row indexes with dataset-owned PostgreSQL metadata and indexed rows while retaining original CSVs, splits and workspace files as authoritative managed artifacts.
Indexes prepare on demand in bounded bulk transactions, preserve exact strings (including NUL-safe binary headers/values), and publish atomically under source locks. Async bounded reads now cover planning, resume, previews, schema preparation and summaries. Saved jobs resolve dataset UUID/hash/parser identities; each job tick verifies original bytes before progress locks and reuses evidence only while its file fingerprint agrees. Registration locks protect active legacy jobs against deletion races.
Operator migration: after coordinated backups and stopping writers, run
python -m asaree.row_index_migration inventory,rebuild, andverifywith explicit previous--legacy-dirdirectories and private--reportfiles. The tool compares originals, legacy rows and PostgreSQL in batches and reports missing/inconsistent evidence. Optionalretirere-verifies and renames matched SQLite files to.sqlite.retired; locks, unknown files and source artifacts remain retained. Complete existing #50 relocation plans with their matching image/config before upgrading. Roll back using the coordinated pre-cutover backup and old image/config; new job descriptors are not compatible with the previous SQLite reader. Full procedure and pre-existing execution JSONB NUL limits are documented indocs/managed-storage.md.Validation: CI run 42745154628 passed on f9db7ca with 2014 regression tests passed, 3 skipped and 14 statistical tests deselected. PostgreSQL ingestion, concurrency, cancellation, deletion-race and legacy-job migration checks passed. The frontend and separate statistical-validation step were skipped by scope. Full mypy remains warn-only (693 findings across 57 files); one existing statistical convergence warning was reported. Locally, 138 focused tests passed and 57 PostgreSQL/opt-in checks skipped without services; Python lint, focused typing for the new index/retention/migration/job/schema modules, migration graph/offline SQL checks and
git diff --checkpassed. The first CI run exposed an obsolete fictional-source batching fixture, corrected to use a real owned registration and verified CSV. No Docker, browser automation, dependency updates or expensive simulations were run locally.Addresses EpistasisLab/ASAREE_Issues#69. Does not implement #55 or change Motoro's database.