[LuciferYang]

The data_change field on DataReplacement and its spec text landed in #9705. This PR makes the Rust transaction code act on it.

Dataset::column_layout_stats and its Python and Java bindings are #9605, which this branch merges. Once #9605 merges, those files drop out of this diff.

What's wrong today

Each add_columns backfill gives every fragment one more data file. Scans and random reads slow down as the count grows, and compact_files never fixes it: its candidates are fragments with too many deletions or overlays, or too few rows, so a large, clean fragment split across many files is never picked. This is the "horizontal compaction" gap from #4286 and #3995.

What this adds

compact_files gets a second kind of task. Besides rewriting fragments into new fragments, a task can repack the columns of one fragment into fewer data files without moving its rows, so the fragment id, row addresses, deletions, overlays and index coverage don't change.

The default planner plans fragment rewrites exactly as before. Then, for every fragment no rewrite took, it plans a repack when the fragment holds columns in more than max_data_files_per_fragment files, as Dataset::column_layout_stats counts them. The option defaults to None, so nothing changes unless it is set.

A repack merges whole files, and only files the commit will drop, at least two at a time, so the file count always goes down. A file stays if it holds a blob column, a column only partly in the fragment, or the fragment's spilled row lineage, or shares a column with the lineage file. When every file holding a column could go and the limit is above 1, one of them is kept if a merge can leave it: the merge that leaves the largest file in place wins, then the one that leaves the fewest live files.

scope limits a run to one kind of task: RewriteFragments, RepackColumns, or All (the default). max_data_files_per_fragment can also be set as table config (lance.compaction.max_data_files_per_fragment); scope is a per-run option only. Repacks aren't planned under ForceBinaryCopy, since they reencode. plan_groups plans rewrites only.

Callers can read the same stats the planner uses to see why a fragment was or wasn't picked: dataset.stats.column_layout_stats() in Python, Dataset.getColumnLayoutStatistics() in Java. The compaction section of the user guide documents the options and the config key.

How the tasks run and commit

CompactionTask::execute runs a task by its kind: a rewrite task goes through rewrite_files as before, a repack task through the new repack code. compact_files_with_planner runs every task of the plan through it, so a task run in-process and one serialized to a worker take the same path.

A repack task copies the base values of the columns it moves for every physical row, deleted rows included; overlays and the deletion file stay on the fragment and keep applying to the new files. It writes one new file per planned file, row-aligned with the fragment, and deletes what it wrote if it fails.

commit_compaction commits fragment rewrites as one Operation::Rewrite, as before, then repacks as one Operation::DataReplacement { data_change: false } in a separate version. Results that put a fragment in both kinds of task are rejected before either commit. If the repack commit fails, the rewrite stays committed and the dataset handle is left at its version; the run should be planned again, not retried with the same results.

Why DataReplacement with data_change: false

Repacking moves values without changing them. DataReplacement already means "these fragments get new files for these fields", and the composite transactions draft (#8644, AddDataFile) uses a data_change flag for the same idea. With data_change: false:

• index coverage of the moved fields is kept, and a concurrent index build doesn't conflict;
• overlays aren't tombstoned, since they still hold the newer values and keep shadowing the base;
• no row's last-updated version moves, so an incremental reader doesn't see the moved rows as changed;
• the commit doesn't count as supplying values to a column whose nullability a concurrent change is tightening.

DataReplacement is applied to the fragment as it stands at commit. A concurrent change to the fragment's other columns, a delete of some of its rows, or a drop of a column the repack leaves alone is kept, and the repack still commits. A concurrent update or replacement of a moved column, a rewrite of the fragment, a delete of the whole fragment, or a drop of a moved column makes it retry; for the last two, a data_change: true replacement fails instead. A concurrent add_columns also makes it retry, since its Merge conflicts with every DataReplacement.

To carry a repack, DataReplacement now accepts several groups for one fragment (one per new file), applied in order. The check that every new file in one operation has the same fields is gone. data_change is the optional bool #9705 added to the protobuf. It is written only when false, and a transaction written before it existed reads as true.

Where the file-layout rule lives

What a DataReplacement commit does to a fragment's data files is now Fragment::replace_files in lance-table, a pure function: the exact-match swap, appending a file for fields the fragment has no data for, tombstoning and appending, the V1 guard, dropping files left with no live field and no spilled lineage, and the "no changes" error. build_manifest groups the replacements by fragment and calls it once per fragment; overlay tombstoning and row-version stamping stay there, since they depend on data_change and the read version.

The repack planner builds a placeholder file for each merge it considers, applies it with replace_files, and scores the result, so it has no copy of the rule of its own. A repack task's files_removed comes from replace_files too.

The TODO(rmeng) row-count check stays a TODO: DataFile has no row count to check.

What changes for existing callers

• TaskData gains kind and RewriteResult gains repacked_files, both #[serde(default)]. A plan or result serialized by an older release reads as a fragment rewrite. TaskData::repack_columns(fragment, files) builds a repack task, and a repack task refuses output_clustering_id and output_metadata, since it writes no new fragment.
• An older worker given a repack task ignores the unknown field and rewrites the fragment instead, a valid compaction the coordinator commits as a rewrite. The commit tells the two apart by the result's repacked_files, not the task's kind.
• Java TaskData and RewriteResult now declare the serialVersionUID their current shape had, so streams from older workers still deserialize. Java and Python expose max_data_files_per_fragment, scope, the task kind (TaskData.getRepackFiles(), CompactionTask.kind), and DataReplacement.dataChange / data_change.
• max_source_bytes counts only the files a repack task reads, which are the files holding the columns it moves, with no overlays.

Known limits

• A run that both rewrites and repacks commits them as separate versions until composite transactions can commit them as one.
• A repack keeps overlays as overlays. It doesn't fold them into the new base; a fragment rewrite still does that.
• On a table with spilled row lineage, the file carrying it keeps its columns, since the new files carry no lineage.
• Fragments with legacy (V1) files aren't repacked, and blob columns stay where they are, so such a fragment can stay over the limit.
• A V2.0 file holding only a struct header counts toward the limit, and goes only when a repack moves that struct or the fragment is rewritten.
• Grouping columns into files is left to a follow-up that applies to every write, not only compaction: Column groups: keep chosen columns in their own data files, across writes and compaction #10509.

Tests

Test	What it pins
repack_collapses_backfilled_files	two backfills give three files per fragment; one run brings them to one, in one commit, with values and fragment ids unchanged; a second run commits nothing
repack_keeps_the_biggest_file, repack_keeps_deletions, repack_drops_file_left_with_only_dropped_columns	the layout chosen over a limit of 2; deleted rows stay deleted and keep their base values; a file left with only dropped columns is removed
repack_keeps_scalar_index, repack_columns_preserves_vector_index, repack_columns_preserves_data_overlay	BTree and IVF_PQ indices keep uuid and coverage, still serve the query and return the same rows; an overlay on a moved column still shadows its repacked base
compaction_rewrites_and_repacks_in_one_run, compaction_scope_selects_task_kinds, commit_rejects_a_fragment_in_both_kinds_of_task	a mixed run commits Rewrite then DataReplacement; scope selects one kind; a fragment in both kinds of result is refused before either commit
compaction_plans_no_repack_by_default, repack_is_not_planned_at_the_limit	nothing is repacked unless a limit is set, and a fragment at the limit is left alone
repack_tasks_run_distributed, task_data_without_kind_rewrites_fragments, rewrite_result_without_repacked_files_is_a_rewrite, repack_task_refuses_output_properties, repack_task_deletes_its_files_when_the_layout_is_refused	repack tasks and results survive serde and commit together; a task or result serialized before this change reads as a rewrite; a repack task with output properties is refused; a hand-built task whose files the commit would refuse deletes what it wrote
repack_against_concurrent_delete, repack_against_concurrent_drop, repack_against_concurrent_new_value_of_moved_column, repack_against_concurrent_update_of_another_column	which concurrent changes the repack commits over and which make it retry, including an in-place update and a data-changing replacement of a moved column, and an in-place update of a column it leaves alone
repack_keeps_the_spilled_lineage_file, repack_merges_around_a_blob_column, repack_merges_around_a_blob_outside_the_kept_file, repack_merges_every_movable_file_next_to_one_that_stays, repack_never_adds_a_file, repack_skips_legacy_files, repack_skips_a_fragment_holding_a_legacy_file	the lineage, blob and V1 rules; no plan adds a file
repack_moves_a_struct_with_its_header, repack_merges_only_files_it_empties, repack_keeps_the_largest_file_that_shares_a_struct, repack_keeps_the_largest_file_next_to_a_header_only_file, repack_prefers_leaving_the_largest_file_in_place, repack_prefers_the_proposal_leaving_the_fewest_files, repack_plans_nothing_for_header_only_files, repack_merges_all_files_when_none_can_stay, repack_with_a_dropped_struct_child_converges	V2.0 struct headers: a merge counts the files a header keeps alive, which file stays and which proposal wins, that a looser limit than 1 never leaves a fragment over it when a limit of 1 would merge it, and that a second run plans nothing
repack_moves_a_struct_whose_parent_has_no_file_field	from V2.1 on, where a file lists a struct's leaves but not the struct, the struct moves whole and only the new file answers for it
repack_options_parse_from_config, compaction_scope_rejects_an_unknown_name, max_data_files_per_fragment_must_be_positive	option and table-config parsing, and the error a bad value gives
repack_is_not_planned_under_force_binary_copy, repack_counts_only_the_files_it_reads_against_the_byte_budget	the ForceBinaryCopy and max_source_bytes rules
test_defer_index_remap_allows_a_repack_only_plan, test_undrained_fri_allows_a_repack_only_plan, test_column_repack_seeds_moved_column, test_harvest_seeds_from_repacked_file	on a stable-row-id table, neither defer_index_remap nor an undrained fragment reuse index refuses a repack-only run, which moves no rows; a repacked zone-map column gets a seed in its new file
test_conflicts_data_replacement (new cases), test_data_replacement_data_change_roundtrips, test_data_replacement_without_data_change_keeps_overlays, test_data_replacement_applies_several_groups_to_one_fragment, prepare_indices_keeps_coverage_for_moved_values	the DataReplacement changes on their own
test_data_replacement_data_change_round_trips (Python)	DataReplacement(data_change=False) survives commit and read_transaction, and only data_change=True withdraws index coverage of the replaced column
test_compact_files_repacks_columns, test_distributed_repack, test_repack_options_come_from_dataset_config, test_repack_options_override_dataset_config, test_repack_rejects_bad_options, test_repack_keeps_vector_index (Python)	the options as arguments and as table config, which one wins, the errors bad values raise, the task kind, the layout stats, pickling, and an IVF_PQ index kept through a repack
test_commit_leaves_dataset_at_the_rewrite_when_the_repack_fails (Python)	when the repack commit fails after the rewrite landed, Compaction.commit leaves the dataset at the rewrite's version
testRepackColumns, testDeserializeTaskAndResultFromOlderVersion (Java)	the options and layout stats through JNI, Java serialization of tasks and results, and that a TaskData or RewriteResult serialized by the previous release still deserializes

Credit

Repacking started from @dshepelev15's #9291: the repack task writes each new file the way #9291's rewrite_columns did, reading the moved columns back through the fragment updater. Its column_groups option is left for the follow-up above. Dataset::column_layout_stats started in @Xuanwo's #8614. Thanks to both of you.

Co-authored-by: dshepelev15 d-shepelev@list.ru
Co-authored-by: Xuanwo github@xuanwo.io

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[LuciferYang]

@wjones127 @Xuanwo, picking up the horizontal-compaction thread (#4286, discussion #3995, and the stalled #9291). I built the whole thing out on a branch and split it into reviewable PRs; here's where it stands.

The gap, restated: repeated add_columns backfills leave a fragment split across many per-column files, which slows scans and random reads, and compact_files can't help because it's vertical and its candidate check never looks at per-fragment file count. Horizontal compaction repacks a fragment's columns into fewer files without moving rows, so indices and row addresses stay valid.

Two things you each asked for shaped the design:

• @wjones127, your requested-changes on feat: keep column groups through compaction and add rewrite_columns #9291 was to make the compaction planner/executor pluggable first, with horizontal as one implementation rather than a bolt-on. That's feat(compaction): add a pluggable CompactionExecutor #9606: a CompactionExecutor + CompactionCommitter seam with a generic run_compaction_pipeline<E, C> driver, and existing vertical compaction routed through it as RewriteExecutor/RewriteCommitter. Pure refactor, existing tests green. Horizontal is then a second executor/committer pair.
• @Xuanwo, your point was that compaction should expose stats, not decide behind a black box. That's feat(compaction): add Dataset::column_layout_stats #9605: Dataset::column_layout_stats(), the per-fragment live-file and overlay counts a planner keys on.

Correctness rests on committing horizontal compaction as Operation::Update{RewriteColumns} with an empty fields_modified, reusing the existing transaction path instead of adding a new Operation. Empty means index-maintenance prunes nothing (coverage kept) and overlay tombstoning is a no-op (overlays carried forward, still shadowing). I tested the parts that were only argued before: a vector index keeps its uuid and coverage across a rewrite of the indexed column and returns the same KNN; a data overlay survives a rewrite of an unrelated column; the file count drops from three to one after two backfills.

Layout of the stack:

• feat(compaction): repack a fragment's columns as a compact_files task #9604: overview (draft), the whole picture so the seam can be judged with both executors present.
• feat(compaction): add a pluggable CompactionExecutor #9606: pluggable executor/committer framework (ready).
• feat(compaction): add Dataset::column_layout_stats #9605: column_layout_stats (ready).
• Horizontal compaction (rewrite_columns) and compaction column_groups follow, built on the framework.

A couple of things I'd rather settle with you than alone:

• Dataset::rewrite_columns reuses the RewriteColumns name that merge-insert already uses for a different operation. Keep it for continuity with feat: keep column groups through compaction and add rewrite_columns #9291, or rename (say repack_columns) to avoid the collision?
• The framework keeps vertical and horizontal as separate entries committing their own operation, rather than one planner emitting both task kinds (which would need a CompactionPlan wire-format change). Does that match what you had in mind, or did you want the unified planner?

Review of #9606 and #9605 would be much appreciated; #9604 has the full context for the direction question.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[LuciferYang]

@wjones127 Thank you for your careful review. I have some other things to take care of today, and I will revise the corresponding issues tomorrow.

[Xuanwo]

This repack changes soft-deleted rows' stored values while declaring data_change=false. The source retains its deletion file, so updater_with_version routes those positions through DeletionRestorer, which writes blanks or copies of live rows. The merged contract requires the original base values for every physical row, and include_deleted_rows exposes those values. The current repack_keeps_deletions assertion checks only the live scan, so it misses this contract violation.

[LuciferYang]

Good catch. Fixed in 67d4504: the repack now reads the fragment with its deletion file cleared, as it already did with the overlays, so it copies the base values of every physical row, and the deletion file keeps masking the deleted ones. repack_keeps_deletions now also compares an include_deleted_rows scan before and after the repack; it fails without the fix.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The deleted-row issue is fixed: repacks now preserve every physical row's base values while retaining the fragment's deletion metadata. File-count repacking through compact_files addresses backfill-created file fan-out, with the shared replacement rule keeping planning and commit aligned.

Rewrites and repacks commit separately: if repacking fails, the rewrite remains committed. Replan instead of resubmitting those results. The overlapping-value protections continue to reject stale repacks that would overwrite newer values.

Broader grouping across write operations remains in #10509. Prefer merging layout statistics #9605 first, then rebasing this PR to remove the stacked statistics changes.

Please mark this PR with the breaking-change label: Rust CompactionOptions, TaskData, and RewriteResult in rust/lance/src/dataset/optimize.rs gain fields, and Operation::DataReplacement in rust/lance-table/src/transaction/operation.rs gains data_change, so downstream Rust callers must update struct literals and exhaustive patterns.