[wjones127]

Defines the wire format for action-based transactions (Transaction V2): a CompositeOperation that replaces the single legacy Operation with an ordered list of granular, composable Action deltas. This is the format change only — read-side fail-closed, no write path. Every protobuf change the implementation stack needs lands here, so the format is a single vote; the PRs stacked on top of it change no .proto file.

Layout

The V2 vocabulary lives in its own files under protos/transaction/, and its messages in their own lance.transaction package, apart from the table messages in lance.table (which already has an unrelated AddDataFile, a fragment-tree mutation). In Rust, the generated code is lance_table::format::pb_transaction.

• actions.proto — the V2 vocabulary (Ref, CompositeOperation, UserStep, Action + the action messages), with the design rationale summarized in an in-tree file header so it stands on its own.
• common.proto — shared building blocks (UpdateMap, KeyExistenceFilter, …) referenced by both legacy operations and V2 actions, promoted from nested Transaction messages to top-level lance.transaction messages to avoid a circular import. Wire-compatible: field numbers unchanged and none are Any-packed.
• transaction.proto — moved into the directory; keeps the Transaction envelope, legacy operations, and the new composite_operation oneof arm (field 116).

What's in the wire format

CompositeOperation { [UserStep] }
UserStep      { description, [Action] }        // human-readable step; survives squash
Action        { oneof of 19 actions }
Ref           { committed: uint64 | local: uint32 }   // one ref type for field/fragment/base ids

Design principles (fully documented in actions.proto):

• Deltas, not post-images — actions record the change (AddDataFile, TombstoneFieldData), so compound commits and branch merge fall out uniformly.
• Minting vs. reference-stable — minted ids (field/fragment/base) carry a Local token via a single Ref and relocate on merge/rebase; reference-stable changes key off stable coordinates and keep a post-image with a derivable delta.
• Field-level schema — AddField / DropField / identity-preserving AlterField (rename and nullability; logical_type is reserved and rejected at apply, so a cast is DropField + AddField + data). No wholesale SetSchema.
• Index segments — AddIndexSegment / RemoveIndexSegment / AdjustIndexCoverage (a logical index is the segments sharing a name; per-segment config duplication is a pre-existing limitation this draft doesn't fix).
• Assertions — AssertUniqueKeys carries merge-insert's non-derivable key-existence filter as an explicit precondition.
• Off the wire — computed conflict footprints and large derivable row-level deltas (deletion affected-rows, update matched-offsets) are recomputed at conflict time, not serialized.
• Additive-safe — message identities and core mutation fields are pinned; discoveries land as added optional fields.

data_change markers (Delta-style, for CDC skip of compaction) and index-coverage representation carry in-tree TODOs to finalize.

Settled by the implementation

These started out in the PRs stacked above and were pulled down here so the format is voted on once:

• UserOperation becomes CompositeOperation, naming what distinguishes it from the other operations, and loses its description — the per-step UserStep.description is what keeps history readable. UserAction is renamed UserStep (field numbers unchanged), so the step a user recognizes is not confused with the Actions it holds.
• The field actions reference fields by Ref, so an action can name a field the same operation is minting.
• AddIndexSegment gains base, created_at, and dataset_version, and its covered_fragments becomes an optional wrapper rather than a bare repeated. A bare repeated field cannot distinguish "no coverage recorded" — what the MemWAL and fragment-reuse indices carry, and what the query path treats as "serve this segment" — from "covers no fragment", which it treats as "skip it". dataset_version is a correctness gate rather than provenance: an overlay committed at or before it counts as already folded into the index.
• RemoveIndexSegment and AdjustIndexCoverage carry the name of the logical index the segment belongs to, which is what conflict detection compares.
• The update_compacted_ss_tables oneof arm is named to match its UpdateCompactedSsTables message, which is what the action vocabulary keys the wire encoding off.
• AddFragment carries a spilled row lineage sequence as a RowLineageColumn arm, exactly as DataFragment does since feat(format): define hidden row lineage columns #9253: the sequence is a hidden column of one of the data files the fragment's AddDataFile actions add.
• Doc-only: SetDeletionFile.fragment is committed-only because the deletion file path embeds the fragment id; an explicit AddIndexSegment.dataset_version must equal the read version for now; ResetTable discards actions before it in the same operation.

Worked examples

Add a column and its data in one commit

A new field has no id until the commit lands. The data files name it by the token its AddField minted, so readers never see the column without its data:

CompositeOperation
└── UserStep "add column b"
    ├── AddField    { local: 0, parent: none, def: b: int32 }
    ├── AddDataFile { fragment: Committed(0), field_ids: [Local(0)], file: b-0.lance }
    └── AddDataFile { fragment: Committed(1), field_ids: [Local(0)], file: b-1.lance }

At apply, Local(0) resolves to the next field id, and both files are stamped with it. Replaying the set onto a newer version re-resolves the token against that version's counter.

Caveats

• Append + delete, and append + index, need a reservation commit first. A deletion file's path embeds the fragment id (_deletions/{fragment_id}-{read_version}-{id}.{ext}), and apply does no IO, so SetDeletionFile takes a committed id. An index segment's files likewise record real fragment ids and row addresses. Commit ReserveFragmentIds first. Then write the fragments as Ref::Committed(reserved_id) and build the deletion file or segment against those ids.
• Indexing a column added in the same commit is not possible with today's builders. AddIndexSegment.fields accepts Local(0), but every index builder reads the column through the committed schema. Commit the column first, then build and add the segment.

Library changes (minimal, fail-closed)

Read-side rejection only:

• One arm in TryFrom<pb::Transaction> returns Error::NotSupported for a CompositeOperation, with a comment forbidding lenient parsing (the try_collect contract aborts a concurrent V2 commit rather than silently skipping it).
• Regression test test_composite_operation_rejected_on_load.

No Operation enum variant, no apply / translation / conflict resolution, no write path, no Cargo feature gate, no Python/Java changes.

Verification

lance-table + lance build; new test passes; cargo fmt --all and cargo clippy -p lance-table -p lance --tests -- -D warnings clean.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (2)

• WIP
• Draft

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: d6c3e873-ac4a-40f7-89f6-e462d4db9be4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch will/oss-1530-draft-action-based-transaction

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

❌ Patch coverage is 87.75510% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-table/src/format/key_existence.rs	44.44%	5 Missing ⚠️
rust/lance-table/src/transaction/proto.rs	97.50%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

Self-review pass (polish-pr), 2026-09-20

Decision

No code changes. The diff is mechanical apart from the new wire schema and one fail-closed decode arm; the self-review found nothing here that should be auto-applied, because every finding either pins a proto field number or reshapes a message, and the field numbers ripple through all three PRs above this one.

Status

Open. The items below want a decision before the vote pins the tags.

Open questions

Wire schema:

• CompositeOperation duplicates uuid and read_version, which Transaction already carries. The comment says uuid "matches Transaction.uuid semantics", which is the question rather than the answer: two copies that can disagree, with no stated rule for which wins. Drop them, or state that the outer fields are authoritative.
• AssertUniqueKeys embeds KeyExistenceFilter and then documents filter.field_ids as permanently ignored and left empty. A wire field that must be empty and must not be read is a trap once the tag is pinned. Splitting a payload message holding just the oneof { ExactKeySetFilter, BloomFilter } would let both sides carry their own field list.
• Ref is a generic top-level name in lance.table and a Rust keyword, so every generated call site is pb::r#ref::Kind. IdRef would also say what it refers to.
• Ref.committed is uint64, but the id spaces it names are not all that. Fragment id is uint64, base id is uint32, field id is int32 — and a field id has negative sentinels (-1 unassigned, -2 tombstoned, the value TombstoneFieldData writes). Worth stating that the committed value is the non-negative id only, and what apply does with an out-of-range one.
• The Ref-vs-committed-id line is drawn inconsistently across the fragment actions. SetDeletionFile.fragment is a bare uint64 with a good reason. RefreshRowVersionMetadata.fragment_ids and AdjustIndexCoverage.remove_fragments are bare and unexplained. RemoveFragment, TombstoneFieldData, AddOverlays and DropField take a Ref — and AddOverlays's own comment says it targets committed fragments.
• ReserveFragmentIds.count is uint32 while ReserveRowIds.count is uint64, in two messages the file presents as counterparts.
• RemoveIndexSegment.name is derivable from the segment uuid at the read version; the comment justifies it only as an apply-time cross-check, which is worth saying outright.
• AddFragment restates six DataFragment fields rather than embedding it, unlike AddField.def and AddBase.base, with no stated reason.
• The Action oneof tags no longer follow reading order (add_data_file = 2 after add_field = 3; reserve_row_ids = 20 before assert_unique_keys = 19).
• AddIndexSegment's uuid / name / index_details / index_version and ConfigUpdate's three optional UpdateMap fields are the undocumented ones in otherwise well-documented messages; protos/AGENTS.md asks optional fields to state what absent means.

Tests and process:

• decode_inline_transaction must keep returning None for a CompositeOperation, so that opening a dataset whose newest version was written by a newer Lance still works. This PR now depends on that fail-open half of the split and nothing pins it; the function is private and takes &[u8], so a unit test at the bottom of dataset.rs is a few lines.
• test_only_persisted_protos_are_format_changes is blind to a stale glob. A protos/ glob matching nothing contributes nothing to the resolved set, so re-adding the now-deleted protos/transaction.proto to the config still passes — which is the failure mode this PR's config edit exists to repair. Asserting that every protos/-prefixed glob matches at least one file closes it.
• Nothing exercises the pb conversions for the messages hoisted out of Transaction into lance.table (UpdateMap, KeyExistenceFilter, BloomFilter, ExactKeySetFilter). The move is wire-compatible on inspection — field numbers and embedded encodings unchanged, none carried in a google.protobuf.Any — but a round-trip test in key_existence.rs would prove it, and would also cover the zero-means-default branch.
• protos/AGENTS.md asks a persisted-format proto change to land with the matching docs/src/format/ change. The only docs edit here is a link retarget, and CompositeOperation is not described in docs/src/format at all — worth either a short "Transaction V2 (draft)" subsection or a line in the PR description saying the spec text lands with the vote.

🤖 Generated with Claude Code

[wjones127]

Legacy conflict characterization run against this PR (burn-down item 14)

Decision

#9220's characterization suite passes unchanged on this PR: 42 passed, 0 failed, 6 ignored, ~2s.

Status

Verified 2026-09-20. Nothing in the stack changes legacy conflict behaviour.

Options and criteria

The suite's branch (will/legacy-conflict-characterization, base 8cdffd30e) sits 56 commits ahead of this stack's base (93ba1d20c), so the suite does not compile on the stack as-is — FileFragment::write_overlay, which scenarios.rs uses, does not exist at the older base. That is base skew, not a behavioural difference.

Method: for each PR, a throwaway branch off the PR head, merge 8cdffd30e (clean, no conflicts), cherry-pick the suite's five commits, cargo test -p lance --lib conflict_matrix. Throwaway branches deleted afterwards; nothing here is pushed.

Results, all four layers:

PR	Result
#7954	42 passed, 0 failed, 6 ignored
#8644	42 passed, 0 failed, 6 ignored
#8645	42 passed, 0 failed, 6 ignored
#8646	42 passed, 0 failed, 6 ignored

#8644 is the layer that matters most here — it is the one that touches conflict_resolver.rs, adding the fail-closed CompositeOperation arm to every check_*_txn and spelling out check_add_bases_txn's wildcard. The suite covers exactly that file and is unaffected.

Open questions

• The suite's 6 ignored cases are ignored on its own branch too; this run does not change them.
• The stack will need a rebase onto current upstream/main before merge regardless; the clean merge above is evidence that rebase is not expected to be eventful.

🤖 Generated with Claude Code

[wjones127]

question: is this still a worry after we did #9482?

[wjones127]

No. Since #9482 (v14.0.0) the inline transaction is in manifest field 23, which older readers ignore, so any reader can open and scan the table. Only a reader before v14.0.0 that decodes the transaction itself (history, conflict checks) fails, and in a conflict check that aborts the older writer's commit, which is safe. Replaced the danger box with a short note in fd58812.

[wjones127]

nitpick: this is a bad example given the issues with updating index on new data. Let's say instead:

Suggested change

	single named operation covers, such as appending a fragment and updating an
	index together, and makes the transaction a true diff of the manifest rather
	single named operation covers, such as appending a fragment and updating table config together, and makes the transaction a true diff of the manifest rather

[wjones127]

Applied in fd58812.

[wjones127]

suggestion: instead of pushing the squash idea, maybe we should say they are for multi-statement transactions, which is a more canonical database idea and really goes along with the "combining multiple operations into the same transaction".

[wjones127]

Reworded around multi-statement transactions in fd58812.

[wjones127]

nitpick: this is a little convoluted. Please simplify the language. Maybe split these sentences

Suggested change

	identifier (a field, fragment, or base id) names it with a local placeholder
	that resolves against the target's counters at apply time. Both properties are
	identifier (a field, fragment, or base id) names it with a local placeholder. When building the manifest before commit, the local placeholders are translated into concrete identifiers. Both properties—true deltas and local placeholders—are

[wjones127]

Applied in fd58812.

[wjones127]

nitpick: this seems like a detail to leave to the protobuf file.

[wjones127]

Removed in fd58812. actions.proto already covers FieldRef's signedness and how apply stamps fields/base_id.

[wjones127]

question: what does this mean?

[wjones127]

The fragment reuse history translates row addresses: a reader maps an index hit at an old fragment's address to its new address after compaction. On a table with stable row ids an index can store stable row ids instead; those don't change on compaction, so translating them would be wrong. So apply refuses to record a transition while such an index exists (and adding one while a tagged history exists is refused at publish). Reworded the bullet to say this in fd58812.

[wjones127]

question: again, is this still true?

[wjones127]

No, same as the warning above. Rewrote this bullet in fd58812 to point at Inline Transaction Sections.

[wjones127]

question: again, is this true?

[wjones127]

No. Reworded in fd58812, along with the matching comments in transaction.proto and the action.rs module doc (#8644).

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The compatibility text now matches the prerequisite already supplied by #9482 in the base: writers use manifest field 23, which older readers ignore, so an unsupported transaction need not prevent opening and scanning the data. Operations that require interpreting that transaction, including conflict checking, can still fail safely. The four original findings remain closed.

Delta actions and latest-history fragment-reuse append/trim remain reasonable foundations; this revision changes neither their wire encoding nor runtime behavior. Acceptance is still limited to the experimental, format-only scope: there is no V2 writer or executor, and V2 transaction loads fail closed. Replay and trim races, mapping-payload validation, stable-row-ID index admission, reference bounds, drop ordering, and coverage preconditions remain implementation-verification gaps. Paired re-encodes must preserve raw base values, overlays, index coverage, and row versions; missing translation coverage must fall back to scanning.

Snapshot isolation still requires read-derived writers to emit AssertCellsUnchanged. The other adoption costs remain an earlier ID-reservation commit for append-plus-index or append-then-delete, and wide-append metadata overhead. The selected empty-fields representation removes placeholder overhead but retains per-field references. These are not merge prerequisites for this no-writer PR.