Skip to content

fix(lance-tools): add dataset file verification and recovery - #10548

Open
lance-gatefixer[bot] wants to merge 3 commits into
mainfrom
gatekeeper/fix-6694-1
Open

lance-gatefixer[bot] wants to merge 3 commits into
mainfrom
gatekeeper/fix-6694-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Operators lack dataset CLI commands to identify missing or damaged data files and remove their references from a manifest.

Add dataset verify-data-files, repair-manifest, locate-data-file, and restore. Verification checks every referenced object's size and optionally its current-format metadata. Repair validates every explicit file selection, previews affected fragments with --dry-run, and removes whole fragments through a normal delete transaction. File resolution uses the dataset's registered storage bases, including shallow clones. Storage options and optional cloud provider features support remote datasets.

Removing a fragment drops all its rows and columns from the new version; physical objects and historical manifests remain intact. Repair commits skip automatic cleanup so retention settings cannot delete recovery history during repair. Deep verification checks metadata, not every data page or blob payload. The README documents these limits and the recovery commands.

Adding Dataset to the public LanceToolsCommand enum requires callers with exhaustive matches to handle the new variant (breaking-change).

Validation:

  • cargo test -p lance-tools --profile ci --locked — 32 tests passed, including recovery, history preservation under automatic cleanup, table-manifest dispatch, and manifest fixture parsing.
  • cargo fmt --all -- --check — passed.
  • cargo clippy --all --tests --benches --profile ci --locked -- -D warnings — passed.
  • RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --profile ci --locked — passed.

Fixes #6694

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions github-actions Bot added A-deps Dependency updates bug Something isn't working labels Nov 1, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 1, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Applied the breaking-change label for the public LanceToolsCommand::Dataset enum addition, which requires callers with exhaustive matches to handle the new variant. No code change was needed for this labeling request.

@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 1, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 1, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 1, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The recovery commands continue to work alongside table manifest. Repair preserves retained history when automatic cleanup is configured.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates breaking-change bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fast way to verify and eliminate corrupted data files

0 participants