From 52fca11d00e65952347ddee3527abe47ed52ea49 Mon Sep 17 00:00:00 2001 From: Brendan Clement Date: Wed, 28 Oct 2026 11:57:32 -0700 Subject: [PATCH 1/2] feat(format): specify expiring pins for ongoing work --- docs/src/format/index/system/frag_reuse.md | 19 ++ docs/src/format/table/.pages | 1 + docs/src/format/table/branch_tag.md | 30 +++ docs/src/format/table/index.md | 4 + docs/src/format/table/ongoing_work_pins.md | 243 +++++++++++++++++++++ docs/src/format/table/transaction.md | 56 +++++ docs/src/format/table/versioning.md | 29 ++- protos/table.proto | 47 ++++ protos/transaction.proto | 65 ++++++ rust/lance-table/src/feature_flags.rs | 42 +++- rust/lance-table/src/format/manifest.rs | 2 + rust/lance-table/src/transaction/proto.rs | 31 +++ 12 files changed, 566 insertions(+), 3 deletions(-) create mode 100644 docs/src/format/table/ongoing_work_pins.md diff --git a/docs/src/format/index/system/frag_reuse.md b/docs/src/format/index/system/frag_reuse.md index 1179234f13a..c9d4008e596 100644 --- a/docs/src/format/index/system/frag_reuse.md +++ b/docs/src/format/index/system/frag_reuse.md @@ -202,6 +202,25 @@ trimmed. Cleanup must retain intermediate transitions still needed to translate old addresses. External mapping files can be deleted only when no retained dataset version references them. +### Ongoing work retention + +[Ongoing-work pins](../../table/ongoing_work_pins.md) are additional history +consumers before their jobs publish an index. A current active pin at source +version V retains the applicable current-chain transitions stamped at or after +V, inclusively; a rewrite stamps its transition with the immediately preceding +dataset version. Retain the union of requirements from all pins and existing +consumers. Validate that union at the final commit target of every operation +that trims or replaces history, including work planned before registration. + +Compaction must record the movements active pins need and recheck those pins +before committing. Reject a result lacking required mappings so the caller can +replan. Restore or +Overwrite can discard a translation path while continuing to retain the source +snapshot. A pin does not require mappings solely into a discarded intermediate +layout, establish index freshness, or authorize publication of an old result. +Referenced external mapping files remain subject to ordinary retained-snapshot +protection and its [maintenance concurrency limits](../../table/ongoing_work_pins.md#maintenance-concurrency-limits). + ## Stable Row IDs Older readers and writers did not expect both stable row IDs and a Fragment Reuse Index but are diff --git a/docs/src/format/table/.pages b/docs/src/format/table/.pages index 460e158230a..12164af5654 100644 --- a/docs/src/format/table/.pages +++ b/docs/src/format/table/.pages @@ -5,6 +5,7 @@ nav: - Table Config: table_config.md - Versioning: versioning.md - Transactions: transaction.md + - Ongoing Work Pins: ongoing_work_pins.md - Layout: layout.md - Branch & Tag: branch_tag.md - Row ID & Lineage: row_id_lineage.md diff --git a/docs/src/format/table/branch_tag.md b/docs/src/format/table/branch_tag.md index b34149bb3f9..7419b80ea48 100644 --- a/docs/src/format/table/branch_tag.md +++ b/docs/src/format/table/branch_tag.md @@ -84,6 +84,36 @@ which means `/` would create a logical subdirectory (e.g., `bugfix/issue-123`, ` 1.manifest ``` +### Deletion with ongoing work pins + +Deletion must resolve the [authoritative head](versioning.md#current-state-for-pin-decisions) +and reject if its active-pin list is nonempty, identifying the blocking pins. +A pin past `expires_at` still blocks deletion until removed by a committed +ReleasePin or ExpirePins. Deletion does not implicitly revoke pins. Force deletion +must perform the same check. Existing tag and dependent-branch checks also apply. + +For force deletion of an orphan with no branch reference, the pin check may be +skipped only when authoritative resolution definitively establishes that no +attached manifest exists. A listing or resolution failure, or a manifest that +exists but cannot be read or validated, must fail deletion. If a selected head +disappears during inspection, restart resolution as specified in +[Format Versioning](versioning.md#current-state-for-pin-decisions); do not treat +that failed read alone as proof of absence. + +A corrupt current manifest therefore requires repair or administrator recovery +outside the ordinary branch-deletion protocol. `force` does not bypass pin +protection; this proposal defines no operator override. + +This follows existing tag protection: the check is separate from physical +deletion and is not atomic with concurrent acquisition. Registration after a +deleter's inspection does not force that deleter to observe the new pin. +Delayed deletion after recovery and name reuse also remains a known gap. See +[Maintenance concurrency limits](ongoing_work_pins.md#maintenance-concurrency-limits). + +Resolving those existing coordination gaps is outside the pin proposal. Branch +names remain reusable; pins add no name-retirement record, deletion marker, +branch layout, or reference-history format. + ## Tagging ### Tag Name diff --git a/docs/src/format/table/index.md b/docs/src/format/table/index.md index d49801c9a00..a5450b1d7d5 100644 --- a/docs/src/format/table/index.md +++ b/docs/src/format/table/index.md @@ -28,6 +28,10 @@ A manifest describes a single version of the dataset. It contains the complete schema definition including nested fields, the list of data fragments comprising this version, a monotonically increasing version number, and an optional reference to the index section that describes a list of index metadata. +[Ongoing work pins](ongoing_work_pins.md) record unfinished jobs' source snapshots +and FRI retention requirements in the current manifest. See that specification +for their lifecycle and the existing maintenance concurrency limits. + `max_allocated_field_id` is optional. If a manifest sets it, the dataset uses non-reusable field IDs from that version onward. See [Field IDs](schema.md#field-ids). diff --git a/docs/src/format/table/ongoing_work_pins.md b/docs/src/format/table/ongoing_work_pins.md new file mode 100644 index 00000000000..fb61e68da9a --- /dev/null +++ b/docs/src/format/table/ongoing_work_pins.md @@ -0,0 +1,243 @@ +# Ongoing Work Pins + +An ongoing-work pin adds a retention requirement for a fixed source snapshot +and the applicable fragment reuse index (FRI) history while a job runs. Pins do +not establish the freshness or publication validity of a job's output. Physical +retention has the [existing maintenance concurrency limits](#maintenance-concurrency-limits) +described below; confirmation does not fence deletion already in flight. + +## Persisted state + +`Manifest.ongoing_work_pins` (field 27) contains `OngoingWorkPin` records. Only +the current attached manifest's list is authoritative. Missing or empty means +no active pins. Historical copies of the list do not retain ended pins. Order +is insignificant; duplicate IDs or incomplete records are invalid for pin +mutation and maintenance. + +Each record contains `id`, `source_version`, `source_transaction_uuid`, +`created_at`, `expires_at`, and optional `job_label`. Presence and absence +semantics are defined in `protos/table.proto`. The pin ID is the registering +transaction's `uuid`; the source version is that transaction's `read_version`. + +`expires_at` is the UTC time after which cleanup may expire the pin. Passing +that time does not automatically remove it. Renewal changes only `expires_at`; +the source identity, creation time, and label stay fixed. No generation counter +is needed because every extension strictly increases `expires_at`. + +The first registration sets writer feature bit 19, `FLAG_ONGOING_WORK_PINS`, +in the same commit as the pin. A nonempty list without the bit is invalid. +The bit is sticky on subsequent versions, including when the list becomes +empty, and introduces no reader requirement solely for pins. Ordinary reads +may ignore the pin list. Pin-aware maintenance must validate it. + +## Identity + +IDs are nonempty UTF-8 strings compared byte-for-byte. Do not case-fold, +normalize, or replace a stored ID with a different textual representation. +This feature imposes no new canonical UUID spelling. Supported writers must +generate fresh transaction IDs for distinct logical transactions, including +Create, Clone, and Restore and after branch-name or table-path reuse. A retry +preserves its original ID. A committed ID must not identify a different +transaction or a second successful publication. This is a writer requirement, +not a persistent registry of all completed transactions. + +The exact source is `(source_version, source_transaction_uuid)`. Read the UUID +from the source manifest's inline transaction, or from its `transaction_file` +name. For the filename carrier, require a single relative filename ending in +`.txn`, split at the first hyphen, validate the nonempty decimal read-version +prefix, and retain all remaining nonempty text before `.txn` as the ID. The +prefix is the transaction's read version, not necessarily the source manifest +version. The sidecar's contents need not still exist. When both carriers are +present, their IDs must agree exactly. Failure to read or consistently identify +the source rejects acquisition; never fall back to version alone. + +If an active source manifest is missing or its identity disagrees with the +record, report a protection/integrity failure and halt the affected destructive +cleanup. The failure is not an implicit release. Explicit Release by an +unambiguous ID remains allowed without reading that source, so an operator can +end the pin. Readers do not need to validate pins to read unrelated data. + +## Time + +Times use `google.protobuf.Timestamp` in UTC: seconds in +`[-62135596800, 253402300799]` and nanoseconds in `[0, 999999999]`. Reject invalid +values; do not clamp or truncate them. Compare normalized seconds and then +nanoseconds, preserving exact precision. Initial `expires_at` must be later +than `created_at`. + +The format stores absolute times. It does not prescribe an API default duration +or renewal schedule. Fix `created_at` and `expires_at` before the first +registration attempt and preserve both on retries. If a client derives +`expires_at` from a duration, use checked arithmetic and reject overflow. +`created_at` is the registration request's time; do not recompute it at commit. +Each renewal likewise fixes one requested `expires_at` before its first attempt. + +Participating clients use reasonably synchronized clocks. Consumers choose a +renewal margin; no table setting or absolute lifetime cap is required. A pin +remains active after `expires_at` until a committed ReleasePin or ExpirePins +removes it from the authoritative list. + +## Current state and acquisition + +Resolve the authoritative current attached head as specified in +[Format Versioning](versioning.md#current-state-for-pin-decisions). A cached +Dataset or a successful write to a previously reclaimed slot is insufficient. + +For a registration attempt with ID P, source V, and source transaction S: + +1. If current membership already contains P with the same `(V, S)`, recover + that pin's current `expires_at` and label without committing or resetting them. + A different source under P is an error. +2. Otherwise the requested snapshot must still be current: its version and + source transaction UUID must equal `(V, S)`. If the branch has moved on or + the snapshot has been replaced, registration fails. Compare those two values + directly; intervening transaction files are not needed for this check. +3. Construct the next manifest from that current source, add the pin, and set + the sticky capability. Publish only at V+1 through the existing atomic + version-slot protocol. A conflict cannot rebase this acquisition onto another + source. After a definite conflict, a caller that has not started work may + explicitly acquire the new head with a new ID. +4. Before acknowledging either a fresh acquisition or recovery, confirm the + current named-branch reference still has the acquired branch ID, the + authoritative head contains P with `(V, S)`, and the source manifest has S. + The pin may have been carried into a newer head. A historical transaction + matching P alone is insufficient. If these facts cannot be established, + return an indeterminate outcome or failure, never acknowledge registration. + +These confirmation reads establish observed current state, not an atomic +handoff against physical deletion. Branch deletion must perform the +[active-pin check](branch_tag.md#deletion-with-ongoing-work-pins), but that check +is not atomic with registration or subsequent physical deletion. +Main has no branch-reference ID that distinguishes table-root recreation; +whole-table deletion must be coordinated by the owning service or catalog. +No permanent table identity is introduced here. + +Jobs begin work only after confirmation. Jobs that need recovery after a restart +must save their pin ID and, until registration is confirmed, the original +request needed to retry it. Recovery returns the current pin, whose `expires_at` +may already have passed; the consumer chooses whether to renew before continuing. +The pin remains active until its removal commits. Ended registration outcomes +are not retained indefinitely. + +Concurrent commits, including other pin operations, can repeatedly move the +head before registration commits. Callers should bound their registration +attempts and retain the original request for each attempt whose outcome remains +unknown. A definitively rejected attempt does not require recovery state. + +## Renewal, release, and expiration + +`RenewPin` requires the target ID to exist. If the current `expires_at` already +equals or exceeds the fixed request, return the current pin without a commit. +Otherwise commit the requested later `expires_at` while preserving every other +field. Re-evaluate after conflicts. A concurrent Release or ExpirePins that +wins first makes renewal fail; renewal never creates a missing pin. + +`ReleasePin` removes only its target ID. Absence is a successful no-op. It does +not stop the worker or cancel an unconfirmed registration still in flight. +Different jobs have independent IDs even when their source files are identical. + +`ExpirePins` contains the inspection time `observed_at` and candidates +`(pin_id, expected_expires_at)`. All timestamps must be valid, candidate IDs +unique, and each `expected_expires_at` at or before `observed_at`. + +Before committing expiration, remove a candidate only if its ID is still active +and its current `expires_at` exactly equals `expected_expires_at`. If renewal +changed that value, skip the candidate. Also skip absent IDs and preserve all +other pins. Retries use the same `observed_at` and candidates; cleanup may make +a new inspection later. If no candidate qualifies, make no commit. + +This orders renewal against expiration: a prior extension invalidates the old +expiration decision; prior expiration prevents renewal. Renewal and other pin +commits count as ordinary versions and cleanup intervals. No-op operations do +not create versions. Expiration commits precede planning physical reclamation +of the resources they release. Cleanup's maintenance commits must not recursively +trigger the same cleanup. + +## Retention and data operations + +Resolve current pin membership when planning retention. Retain each active +source manifest and the resources that normal snapshot retention preserves for +that manifest. Reuse the existing retention machinery: an active pin adds a +retained snapshot. Cleanup may still remove resources that no active pin or +other retention rule requires. Report when pins prevented removal. Removing a +pin does not remove protection supplied by another pin, snapshot, or consumer. + +Compaction must record the FRI row movements active pins need. Before committing +its result, compaction must recheck the active pins, including pins registered +after planning began. If its result lacks required movement records, it must +not commit; the caller can replan. + +History removal must retain the records needed by any current pin or attached +consumer. Within the applicable current history chain, a source at version V +retains transitions stamped at V or later. The boundary includes V because a +transition is stamped with the version immediately before its rewrite commits. +Retain referenced external mapping files through ordinary snapshot retention. + +Restore and Overwrite carry the current active list, fixed sources, timestamps, +and sticky bit. They never import the target snapshot's historical pin list. +Retain movement records needed to connect a pinned snapshot to the current +table layout. After a restore, records needed only for a discarded intermediate +layout may be removed if no other consumer requires them. Pins keep source +files available, but do not guarantee that a job can translate or publish its +results after a restore or overwrite; retention does not supply a missing +mapping. + +New branches and clones have empty pin lists and preserve the sticky bit when +inherited. Detached snapshots cannot register or mutate authoritative pins. + +Other ordinary commits preserve the current list. All operations that rewrite +or trim FRI revalidate its retention requirements at commit. Detailed directional +rules are in [Transactions](transaction.md#ongoing-work-pin-transactions). + +Pins do not protect newly written, uncommitted output. Jobs need a separate +staging and publication strategy for those files. Before releasing a pin, a job +must publish its output and confirm that the result retains the files and FRI +history it needs. Publication and release are separate transactions; the four +pin operations do not add a combined publication-and-release transaction. + +## Compatibility and implementation boundary + +Bit 19 is reserved and unsupported until this contract is implemented. Ordinary +writers that do not support it must refuse flagged manifests. Readers may still +read data if they support the other reader flags. A flag check gates attempts +that reach it; it does not cancel destructive operations already in flight. + +The minimum upstream Rust cleanup guard boundary identified for this proposal +is `v14.0.0-beta.4`, containing commit +`84d1681e07af565837f2d97e72ff1b8b040cd15c`. That release checks writer flags while +inspecting manifests; it rejects an unknown pin bit rather than honoring pins. +The preceding `v14.0.0-beta.3` lacks that check. Rust-based Python/JNI clients +must be assessed by their embedded core revision. This boundary is not a +pin-capable client release, and other cleanup implementations require their own +verification. Activation requires pin-aware maintenance clients, exclusion of +older or unverified destructive clients, and completion or cancellation of +incompatible operations already in flight. Existing branch deleters that never +inspect capability flags are not made safe by this bit. + +## Maintenance concurrency limits + +This feature does not add atomic coordination between protection acquisition, +restore, and destructive maintenance. Existing tags also use separate +acquisition and retention checks. A tag created after cleanup's inspection can +succeed while that sweep still deletes its old source files. An earlier sweep +can likewise delete files adopted by a concurrent restore, and a delayed branch +deletion can damage a replacement after overlapping recovery and name reuse. + +Current-version-only acquisition avoids selecting an arbitrary old version +already eligible for cleanup. A sweep retains the head it inspected, and the +usual grace period protects newly written, unverified files while they remain +recent. This narrows exposure compared with old-version tags, but is not a +general concurrency guarantee: the grace period can elapse or be bypassed by +`delete_unverified`, and restore can reintroduce older files. The old-version +tag case is not evidence that every current-version pin has the same exposure. + +The pin checks do not cancel an earlier deletion plan or prevent a destructive +operation from running after confirmation. A successful registration therefore +does not guarantee safety against these overlapping operations. Callers must +not treat the pin as a lock against destructive maintenance. + +These are existing maintenance and branch-coordination gaps. Their fixes are +outside this proposal; no deletion marker, permanent name retirement, new +branch layout, or reference journal is specified. Branch names remain reusable. +This limitation does not relax the required active-pin checks, the retention +of sources selected by those checks, or the transactional FRI retention rules. diff --git a/docs/src/format/table/transaction.md b/docs/src/format/table/transaction.md index 0af955e89bd..80e092e7c16 100644 --- a/docs/src/format/table/transaction.md +++ b/docs/src/format/table/transaction.md @@ -605,6 +605,62 @@ An UpdateBases operation only modifies the base paths. As a result, it only conf UpdateBases operations and even then only conflicts if the two operations have base paths with the same id, name, or path. +## Ongoing Work Pin Transactions + +The persisted operations are `RegisterPin` (field 116), `RenewPin` (117), +`ReleasePin` (118), and `ExpirePins` (119), defined in `protos/transaction.proto`. +They mutate only the authoritative current pin list and its required capability. +They cannot be used in detached commits. The record, time, identity, no-op, and +confirmation rules are specified in [Ongoing Work Pins](ongoing_work_pins.md). +Their manifest conflict rules do not make physical cleanup or branch deletion +atomic with acquisition; the existing [maintenance concurrency limits](ongoing_work_pins.md#maintenance-concurrency-limits) +still apply. + +### Directional conflict rules + +For an incoming fresh RegisterPin, recover an identical active ID/source first. +Otherwise require the exact current `(version, source transaction UUID)` to equal +its fixed source, and publish only at that version plus one. Every intervening +commit prevents fresh creation; there is no source-changing rebase. This guard +does not require decoding intervening transactions or fetching their sidecars. +A lost-response retry still needs current membership and source confirmation. + +For all other incoming operations, the following rules describe their behavior +across a committed pin operation. Existing conflicts with other data operations +still apply; this does not make a conflicting Append/Overwrite pair compatible. +Unknown transaction operations retain their existing fail-closed conflict rule. + +| Incoming operation | Across committed RegisterPin, RenewPin, ReleasePin, or ExpirePins | +| --- | --- | +| Append, Delete, Merge, ReserveFragments, Update, Project, UpdateConfig, DataReplacement, DataOverlay, UpdateMemWalState, UpdateBases | May rebase under their existing data rules; copy the current pin list unchanged. None may rewrite pins through config or metadata. Any FRI history removal must also obey current pin retention. | +| CreateIndex | May rebase under existing index rules, preserve current pins, and revalidate retained FRI history if the operation replaces or trims it. Index attachment does not release a pin. | +| Rewrite | Revalidate current pins at the final target. A result without required movement records is a conflict requiring replanning, even when planned before registration. Preserve current pin membership and required history. | +| Restore, Overwrite | Preserve the current active list and sticky capability, including current `expires_at` values and fixed sources; never substitute historical or newly empty lists. Existing data conflicts remain. | +| Clone | Does not copy source pins into the new branch/table. Preserve an inherited sticky capability and the source's independent snapshot protections. Source publication/retention rules still apply. | +| RegisterPin | Recover the identical active ID/source or fail the fresh exact-source guard. Never move an existing acquisition to a new source. | +| RenewPin | Re-read the target pin. Missing is an error; an already satisfied request is a no-op; otherwise extend to the fixed later `expires_at`. Another renewal cannot shorten it. | +| ReleasePin | Remove only the target ID from the current list; absent is a no-op. Concurrent renewal does not prevent explicit release. | +| ExpirePins | Compare each candidate's ID and `expected_expires_at` with current state. Skip missing IDs or changed `expires_at` values and remove only still-eligible candidates. | + +Incoming RenewPin, ReleasePin, and ExpirePins can re-evaluate over the existing +known data operations against the resulting current list: ordinary writes, +Restore, and Overwrite carry the list forward. They do not bypass ordinary +writer-capability checks, branch identity, or unknown-operation rules. An +operation cannot use this compatibility text to migrate an old handle to a +recreated branch. Clone creates a separate destination and does not transfer +ownership of a pin mutation there. + +A renewal that commits first changes `expires_at` and defeats an older +expiration decision. If ReleasePin or ExpirePins removes the ID first, RenewPin +fails rather than recreating it. An absent-ID ReleasePin does not fence a +registration still in flight. Distinct IDs operate independently. Pin operations +with no effect do not commit another version; committed mutations otherwise +count as ordinary versions and cleanup intervals. + +All operations that remove FRI transitions must revalidate the union of current +pin and attached-consumer requirements, including the inclusive source-version +stamp boundary. Pin changes never loosen other transaction preconditions. + ## Conflict Resolution ### Terminology diff --git a/docs/src/format/table/versioning.md b/docs/src/format/table/versioning.md index 06c5c7475bd..358a95f82a1 100644 --- a/docs/src/format/table/versioning.md +++ b/docs/src/format/table/versioning.md @@ -41,9 +41,36 @@ they should return an "unsupported" error on any read or write operation. | `2^15` | `FLAG_NON_REUSABLE_FIELD_IDS` | No | Yes | The manifest sets `max_allocated_field_id`, and a writer must assign new field IDs above it. Before activation, all clients that can write to the dataset must enforce writer feature flags. See [Field IDs](schema.md#field-ids). | | `2^16` | `FLAG_ZONE_WRITE_SEEDS` | Yes | Yes | Referenced data files contain version 1 [zone write seeds](../index/scalar/zone_write_seeds.md). Both bits remain set in derived snapshots, including clone and restore. Reserved and unsupported until implemented. | | `2^17` | `FLAG_OBJECT_METADATA` | No | Yes | The snapshot contains [clustering definitions, references, or a nonzero allocation high-water mark](clustering.md), nonempty [fragment metadata](index.md#fragment-metadata), or nonempty [index segment metadata](../index/index.md#index-segment-metadata). Reserved, unsupported until writers implement the maintenance rules. Readers may ignore this metadata. | +| `2^19` | `FLAG_ONGOING_WORK_PINS` | No | Yes | The current manifest owns the active [ongoing-work pin list](ongoing_work_pins.md). Sticky after first activation, including restore and an empty list. Reserved and unsupported until pin maintenance is implemented. | -Bits 16 and 17 are reserved but remain unsupported. Bits at positions 18 and above are unknown. Unknown or unsupported flags cause implementations to reject the corresponding read or write operation with an "unsupported" error. The paired mixed-version, Managed Blob, and zone write-seed reader and writer bits must each either both be set or both be clear; a half-set manifest is invalid. +Bits 16, 17, and 19 are reserved but remain unsupported. Bit 18 is not assigned by this specification and must remain unsupported until separately defined and implemented. Bits at positions 20 and above are unknown. Unknown or unsupported flags cause implementations to reject the corresponding read or write operation with an "unsupported" error. The paired mixed-version, Managed Blob, and zone write-seed reader and writer bits must each either both be set or both be clear; a half-set manifest is invalid. Committing schema metadata `lance.blob.managed = "true"` activates the Managed Blob capability, even without new blob files. Ordinary Blob v2 writes without this opt-in do not activate it. Once activated, removing the opt-in, removing Blob fields, or restoring an older snapshot does not clear it. Before activation, all maintenance clients must support Managed Blobs: the flag cannot revoke old handles or prevent older clients from running cleanup through an unflagged historical snapshot. See [client compatibility](../../guide/blob.md#managed-objects-and-client-compatibility). + +## Current state for pin decisions + +Pin acquisition, recovery, renewal, expiration, ordinary writes carrying pins, +restore, overwrite, compaction, retention planning, and branch-deletion checks +must resolve the authoritative current attached manifest. A previously opened +Dataset, historical transaction read-back, or hint that stops probing at a gap +is insufficient. + +For ordinary file/object-store commit handlers, enumerate attached manifests +using the active naming scheme, exclude detached and staging files, and choose +the highest version. On an unordered store, consume every listing page. A +lexically ordered V2 listing may stop at its first valid attached manifest when +that naming scheme's ordering guarantee applies. Read the selected manifest; +if it disappeared, restart resolution rather than accepting an older cached +result. Propagate listing/read errors. An optimization is allowed only if it +preserves this result in the presence of reclaimed slots and delayed writes. +A hint that moves forward alone is not a correctness proof. + +External commit stores must resolve and finalize their selected authoritative +head through their own commit protocol; do not replace their coordination with +an uncoordinated object listing. A backend must provide the visibility guarantees +required by its commit protocol or refuse pin acquisition. These reads do not +lock current state. Apply the transaction's publication and retry rules after +resolution; the [maintenance concurrency limits](ongoing_work_pins.md#maintenance-concurrency-limits) +remain even when these reads succeed. diff --git a/protos/table.proto b/protos/table.proto index cad83b38aab..d8fce73a4e4 100644 --- a/protos/table.proto +++ b/protos/table.proto @@ -34,6 +34,42 @@ message UUID { bytes uuid = 1; } +/* An active resource pin owned by the branch whose current manifest contains it. + * Historical copies do not impose retention requirements. All fields except + * job_label are required by the pin contract; protobuf presence permits + * detecting an invalid partial record. IDs are compared as exact UTF-8 bytes, + * without normalization. + */ +message OngoingWorkPin { + /* The registering Transaction.uuid. Absent or empty is invalid. + * Distinct logical registrations must use distinct IDs, including after reuse + * of a branch name or table path. Retries preserve the original ID. + */ + optional string id = 1; + /* The fixed attached source manifest version. Absent or zero is invalid. + * Renewal never changes this version. + */ + optional uint64 source_version = 2; + /* UUID of the transaction that committed the source manifest. + * Absent or empty is invalid. Together with source_version, this identifies + * the source; it does not itself prevent deletion of that source. + */ + optional string source_transaction_uuid = 3; + /* Creation time chosen before the first registration attempt, in UTC. + * Absent is invalid. Retries preserve it; it need not equal commit time. + */ + google.protobuf.Timestamp created_at = 4; + /* Absolute UTC expiry time. Absent or not later than created_at is invalid. + * Passing it makes the pin eligible for committed expiration; it does not + * remove the active record. Renewals may only increase it. + */ + google.protobuf.Timestamp expires_at = 5; + /* Optional human-readable context. Absent or empty means no label. + * It has no retention or authorization meaning and is immutable for this pin. + */ + optional string job_label = 6; +} + // Manifest is a global section shared between all the files. message Manifest { // All fields of the dataset, including the nested fields. @@ -182,6 +218,8 @@ message Manifest { * * 1 << 16: paired with the reader bit for version 1 zone-level statistics * seeds. Writers must preserve the seed contract and the required bits * during data writes, index maintenance, cleanup, clone, and restore. + * * 1 << 19: ongoing-work pins. Writer-only and sticky; maintenance must honor + * the current active-pin list. Reserved until the pin contract is implemented. */ uint64 writer_feature_flags = 10; @@ -316,6 +354,15 @@ message Manifest { */ uint32 max_clustering_id = 26; + /* The authoritative active pins when this is the current attached manifest. + * Absent/empty means no active pins. Order is insignificant; IDs must be + * unique. A nonempty list requires writer bit 19. Once set, that bit remains + * set even when this list becomes empty. Restore/overwrite carry current pins; + * new clones/branches start with an empty list. Detached manifests cannot own + * or mutate active pins. No reader capability is required solely for pins. + */ + repeated OngoingWorkPin ongoing_work_pins = 27; + } // Manifest // external dataset base path diff --git a/protos/transaction.proto b/protos/transaction.proto index b65fc21bee2..1d51cba10cf 100644 --- a/protos/transaction.proto +++ b/protos/transaction.proto @@ -6,6 +6,7 @@ syntax = "proto3"; import "file.proto"; import "table.proto"; import "google/protobuf/any.proto"; +import "google/protobuf/timestamp.proto"; package lance.table; @@ -416,6 +417,66 @@ message Transaction { repeated BasePath new_bases = 1; } + /* Register retention for the exact current source. Transaction.uuid is the + * pin ID and Transaction.read_version is the source version; neither is + * duplicated here. A fresh registration cannot rebase to another source. + */ + message RegisterPin { + /* UUID of the source manifest's committing transaction. + * Absent or empty is invalid; compare exact UTF-8 bytes. + */ + optional string source_transaction_uuid = 1; + /* UTC request creation time, fixed before the first send. + * Absent is invalid. The pin retains this value through retries/renewals. + */ + google.protobuf.Timestamp created_at = 2; + /* Absolute UTC expiry time fixed before the first send. + * Absent or not later than created_at is invalid. + */ + google.protobuf.Timestamp expires_at = 3; + /* Optional immutable context. Absent or empty means no label. + * Recovering an active registration returns its existing label. + */ + optional string job_label = 4; + } + + // Extend an existing pin; this operation never creates a missing pin. + message RenewPin { + // Target pin ID. Absent or empty is invalid. + optional string pin_id = 1; + /* Fixed requested absolute UTC expiry time. Absent is invalid. + * A current expires_at that meets or exceeds this request is a successful no-op. + */ + google.protobuf.Timestamp expires_at = 2; + } + + // Remove one pin; an absent target is a successful no-op. + message ReleasePin { + // Target pin ID. Absent or empty is invalid. + optional string pin_id = 1; + } + + /* Revoke pins selected by ordinary cleanup. Re-evaluate each entry against + * current membership and exact expires_at; skip absent or changed entries. + * Duplicate candidate IDs are invalid. An empty effective set is a no-op. + */ + message ExpirePins { + message Candidate { + // Target pin ID. Absent or empty is invalid. + optional string pin_id = 1; + /* The UTC expires_at observed during inspection. Absent is invalid. + * It must be at or before observed_at and still equal the current expires_at. + */ + google.protobuf.Timestamp expected_expires_at = 2; + } + // Pins inspected for revocation. Absent/empty means no candidates. + repeated Candidate candidates = 1; + /* UTC time sampled by cleanup when it selected these candidates. + * Absent is invalid. Retrying this logical decision preserves this cutoff. + */ + google.protobuf.Timestamp observed_at = 2; + } + /* The operation of this transaction. * * Field numbers 100-199 are reserved for operations. A reader that sees a @@ -439,6 +500,10 @@ message Transaction { Clone clone = 113; UpdateBases update_bases = 114; DataOverlay data_overlay = 115; + RegisterPin register_pin = 116; + RenewPin renew_pin = 117; + ReleasePin release_pin = 118; + ExpirePins expire_pins = 119; } // Fields 200/202 (`blob_append` / `blob_overwrite`) previously represented blob dataset ops. diff --git a/rust/lance-table/src/feature_flags.rs b/rust/lance-table/src/feature_flags.rs index 1d0e3f980b3..10192c3e912 100644 --- a/rust/lance-table/src/feature_flags.rs +++ b/rust/lance-table/src/feature_flags.rs @@ -126,8 +126,12 @@ pub const FLAG_ZONE_WRITE_SEEDS: u64 = 1 << 16; /// them must refuse the dataset. Ordinary readers can ignore this metadata; all /// covered features share this one bit. pub const FLAG_OBJECT_METADATA: u64 = 1 << 17; +/// Writer-only retention requirements for ongoing work pins in the current manifest. +/// Reserved and unsupported until pin-aware writes and maintenance are implemented. +/// Ordinary reads do not require support for the pin metadata. +pub const FLAG_ONGOING_WORK_PINS: u64 = 1 << 19; /// The first bit that is unknown as a feature flag -pub const FLAG_UNKNOWN: u64 = 1 << 18; +pub const FLAG_UNKNOWN: u64 = 1 << 20; const _: () = assert!(FLAG_COVERED_INDEX_METADATA < FLAG_UNKNOWN); // The fence needs a bit the current released build already refuses, which means @@ -146,6 +150,7 @@ const _: () = assert!(FLAG_INDEPENDENT_COVERING_FIELDS < FLAG_UNKNOWN); const _: () = assert!(FLAG_MANAGED_BLOBS < FLAG_UNKNOWN); const _: () = assert!(FLAG_ZONE_WRITE_SEEDS < FLAG_UNKNOWN); const _: () = assert!(FLAG_OBJECT_METADATA < FLAG_UNKNOWN); +const _: () = assert!(FLAG_ONGOING_WORK_PINS < FLAG_UNKNOWN); pub(crate) const STICKY_PAIRED_FLAGS: u64 = FLAG_MIXED_DATA_FILE_VERSIONS | FLAG_FRAGMENT_REUSE_INDEX @@ -384,6 +389,9 @@ fn supported_flags_when( mark_supported(&mut supported, FLAG_INDEPENDENT_COVERING_FIELDS, false); mark_supported(&mut supported, FLAG_ZONE_WRITE_SEEDS, true); mark_supported(&mut supported, FLAG_OBJECT_METADATA, true); + // Bit 18 is left for separate format proposals and has no supported feature here. + mark_supported(&mut supported, 1 << 18, false); + mark_supported(&mut supported, FLAG_ONGOING_WORK_PINS, false); supported } @@ -574,12 +582,42 @@ mod tests { use super::*; use crate::format::BasePath; + #[rstest::rstest] + #[case::unassigned_bit_18(1 << 18, 18)] + #[case::ongoing_work_pins(FLAG_ONGOING_WORK_PINS, 19)] + fn test_reserved_writer_flag_fences_writes_without_blocking_reads( + #[case] flag: u64, + #[case] bit: u32, + #[values(false, true)] overlay_enabled: bool, + #[values(false, true)] frag_reuse_with_stable_row_ids_enabled: bool, + ) { + let mut manifest = empty_manifest(); + manifest.writer_feature_flags = flag; + ensure_can_read_manifest(&manifest).unwrap(); + let error = ensure_can_write_manifest(&manifest).unwrap_err(); + assert!(matches!(error, Error::NotSupported { .. })); + assert!(error.to_string().contains("cannot be written")); + assert!(!can_read_dataset(flag)); + assert!(!can_write_dataset(flag)); + assert_eq!(flag, 1 << bit); + assert_eq!( + supported_flags_when(overlay_enabled, frag_reuse_with_stable_row_ids_enabled) & flag, + 0 + ); + // The zone seed bit is supported; ongoing-work pins remain reserved. + assert!(can_read_dataset(FLAG_ZONE_WRITE_SEEDS)); + assert!(can_write_dataset(FLAG_ZONE_WRITE_SEEDS)); + assert_eq!(FLAG_UNKNOWN, 1 << 20); + assert!(!can_read_dataset(FLAG_UNKNOWN)); + assert!(!can_write_dataset(FLAG_UNKNOWN)); + } + #[test] fn test_object_metadata_flag_is_supported_and_derived() { assert_eq!(FLAG_OBJECT_METADATA, 1 << 17); assert!(can_read_dataset(FLAG_OBJECT_METADATA)); assert!(can_write_dataset(FLAG_OBJECT_METADATA)); - assert_eq!(FLAG_UNKNOWN, 1 << 18); + assert_eq!(FLAG_UNKNOWN, 1 << 20); assert!(!can_read_dataset(FLAG_UNKNOWN)); assert!(!can_write_dataset(FLAG_UNKNOWN)); diff --git a/rust/lance-table/src/format/manifest.rs b/rust/lance-table/src/format/manifest.rs index 8205acab3d6..14b3333a211 100644 --- a/rust/lance-table/src/format/manifest.rs +++ b/rust/lance-table/src/format/manifest.rs @@ -1188,6 +1188,8 @@ impl From<&Manifest> for pb::Manifest { branch: m.branch.clone(), clustering: m.clustering.iter().map(pb::Clustering::from).collect(), max_clustering_id: m.max_clustering_id, + // Pin-aware writes are gated by the unsupported writer capability. + ongoing_work_pins: Vec::new(), writer_version: m .writer_version .as_ref() diff --git a/rust/lance-table/src/transaction/proto.rs b/rust/lance-table/src/transaction/proto.rs index 4995b809da6..20cb2f7ffef 100644 --- a/rust/lance-table/src/transaction/proto.rs +++ b/rust/lance-table/src/transaction/proto.rs @@ -404,6 +404,12 @@ impl TryFrom for Transaction { .map(DataOverlayGroup::try_from) .collect::>>()?, }, + // Reserved format operations remain unsupported until runtime + // handling is implemented. Unknown operations fail closed. + Some(pb::transaction::Operation::RegisterPin(_)) + | Some(pb::transaction::Operation::RenewPin(_)) + | Some(pb::transaction::Operation::ReleasePin(_)) + | Some(pb::transaction::Operation::ExpirePins(_)) => Operation::Unknown {}, // prost drops unrecognized oneof fields, so an operation from a // newer writer is indistinguishable from a missing one. None => Operation::Unknown {}, @@ -826,6 +832,31 @@ mod tests { use super::*; use crate::format::DataFile; use crate::format::overlay::OverlayCoverage; + use prost::Message; + + #[rstest::rstest] + #[case::register(pb::transaction::Operation::RegisterPin(Default::default()))] + #[case::renew(pb::transaction::Operation::RenewPin(Default::default()))] + #[case::release(pb::transaction::Operation::ReleasePin(Default::default()))] + #[case::expire(pb::transaction::Operation::ExpirePins(Default::default()))] + fn test_ongoing_work_pin_operations_remain_unsupported( + #[case] operation: pb::transaction::Operation, + ) { + let message = pb::Transaction { + read_version: 12, + uuid: "pin-operation".to_string(), + operation: Some(operation), + ..Default::default() + }; + let decoded = pb::Transaction::decode(message.encode_to_vec().as_slice()).unwrap(); + let transaction = Transaction::try_from(decoded).unwrap(); + assert!(matches!(transaction.operation, Operation::Unknown {})); + assert_eq!(transaction.read_version, 12); + assert_eq!(transaction.uuid, "pin-operation"); + let error = pb::Transaction::try_from(&transaction).unwrap_err(); + assert!(matches!(error, Error::NotSupported { .. })); + assert!(error.to_string().contains("cannot be re-encoded")); + } #[test] fn test_rewrite_frag_reuse_update_stays_in_memory() { From 2dd6550f648aec1f4597ff9c9f3b911f7a8df0a2 Mon Sep 17 00:00:00 2001 From: Brendan Clement Date: Wed, 28 Oct 2026 22:32:43 -0700 Subject: [PATCH 2/2] docs(format): clarify ongoing-work pin retention --- docs/src/format/index/system/frag_reuse.md | 30 +++++---- docs/src/format/table/ongoing_work_pins.md | 74 ++++++++++++++++------ docs/src/format/table/transaction.md | 8 ++- 3 files changed, 78 insertions(+), 34 deletions(-) diff --git a/docs/src/format/index/system/frag_reuse.md b/docs/src/format/index/system/frag_reuse.md index c9d4008e596..95f1777b55e 100644 --- a/docs/src/format/index/system/frag_reuse.md +++ b/docs/src/format/index/system/frag_reuse.md @@ -59,6 +59,9 @@ it is not a dataset version or the sequence number of a rewrite. ordered compaction and stable partition. Each legacy group can be interpreted as an ordered-compaction transition, forming one history with the new records. +A version-1 history is called **tagged**. If it also contains legacy versions, +it is called **mixed**. + Adding a transition or a new mapping type does not itself require increasing `index_version`. Readers may skip unsupported mappings and fall back to scanning where a complete translation path is unavailable. Writers must reject operations @@ -205,21 +208,26 @@ dataset version references them. ### Ongoing work retention [Ongoing-work pins](../../table/ongoing_work_pins.md) are additional history -consumers before their jobs publish an index. A current active pin at source -version V retains the applicable current-chain transitions stamped at or after -V, inclusively; a rewrite stamps its transition with the immediately preceding -dataset version. Retain the union of requirements from all pins and existing -consumers. Validate that union at the final commit target of every operation -that trims or replaces history, including work planned before registration. +consumers before their jobs publish an index. Legacy-only histories use the +inclusive source-version boundary. Tagged and mixed histories retain records +reachable from pinned source fragments, including all downstream dependencies +and whole legacy versions when any group is needed. Native tagged transitions +do not carry a dataset-version stamp. The exact retention rules are defined in +[Retention and data operations](../../table/ongoing_work_pins.md#retention-and-data-operations). +Retain the union of requirements from all pins and existing consumers. Validate +that union at the final commit target of every operation that trims or replaces +history, including work planned before registration. Restore and Overwrite +apply this check to the resulting current history; they may discard translation +paths while preserving current pins and their source snapshots. Compaction must record the movements active pins need and recheck those pins before committing. Reject a result lacking required mappings so the caller can -replan. Restore or -Overwrite can discard a translation path while continuing to retain the source -snapshot. A pin does not require mappings solely into a discarded intermediate +replan. A pin does not require mappings solely into a discarded intermediate layout, establish index freshness, or authorize publication of an old result. -Referenced external mapping files remain subject to ordinary retained-snapshot -protection and its [maintenance concurrency limits](../../table/ongoing_work_pins.md#maintenance-concurrency-limits). +Required records retain their external mapping files, including mappings added +after registration. A metadata replacement does not release those files while +retained snapshots or required history still reference them. Physical cleanup +remains subject to the existing [maintenance concurrency limits](../../table/ongoing_work_pins.md#maintenance-concurrency-limits). ## Stable Row IDs diff --git a/docs/src/format/table/ongoing_work_pins.md b/docs/src/format/table/ongoing_work_pins.md index fb61e68da9a..fdccaee3e90 100644 --- a/docs/src/format/table/ongoing_work_pins.md +++ b/docs/src/format/table/ongoing_work_pins.md @@ -51,11 +51,13 @@ version. The sidecar's contents need not still exist. When both carriers are present, their IDs must agree exactly. Failure to read or consistently identify the source rejects acquisition; never fall back to version alone. -If an active source manifest is missing or its identity disagrees with the -record, report a protection/integrity failure and halt the affected destructive -cleanup. The failure is not an implicit release. Explicit Release by an -unambiguous ID remains allowed without reading that source, so an operator can -end the pin. Readers do not need to validate pins to read unrelated data. +If an active source manifest is missing, unreadable, or its identity disagrees +with the record, report a protection/integrity failure and halt the affected +destructive cleanup. Also refuse any FRI-trimming or fragment-rewrite commit +that cannot establish the pinned fragment set required by its retention checks. +The failure is not an implicit release. Explicit Release by an unambiguous ID +remains allowed without reading that source, so an operator can end the pin. +Readers do not need to validate pins to read unrelated data. ## Time @@ -157,9 +159,10 @@ trigger the same cleanup. Resolve current pin membership when planning retention. Retain each active source manifest and the resources that normal snapshot retention preserves for -that manifest. Reuse the existing retention machinery: an active pin adds a -retained snapshot. Cleanup may still remove resources that no active pin or -other retention rule requires. Report when pins prevented removal. Removing a +that manifest, including base data, overlay files, FRI metadata, and referenced +external mapping files. Reuse the existing retention machinery: an active pin +adds a retained snapshot. Cleanup may still remove resources that no active pin +or other retention rule requires. Report when pins prevented removal. Removing a pin does not remove protection supplied by another pin, snapshot, or consumer. Compaction must record the FRI row movements active pins need. Before committing @@ -167,27 +170,56 @@ its result, compaction must recheck the active pins, including pins registered after planning began. If its result lacks required movement records, it must not commit; the caller can replan. -History removal must retain the records needed by any current pin or attached -consumer. Within the applicable current history chain, a source at version V -retains transitions stamped at V or later. The boundary includes V because a -transition is stamped with the version immediately before its rewrite commits. -Retain referenced external mapping files through ordinary snapshot retention. +History removal must retain the union of requirements from current pins and +existing consumers, except for translation paths discarded by Restore or +Overwrite as described below. An index catching up does not end an active pin's +requirement. An error in an existing consumer check does not permit dropping +pin-required history. If pin requirements cannot be established, refuse the trim. + +For legacy-only FRI (`index_version = 0`), a source at version V retains +applicable current-chain records whose `dataset_version` is V or later. When +publishing a new legacy record, a pin-aware writer must stamp its +`dataset_version` with the version immediately preceding the publishing +manifest, and restamp that new record on retry. This makes the retention +boundary inclusive of V; it does not change stamps on existing records. + +For FRI `index_version = 1`, including histories that also contain legacy +records, determine pin requirements from fragment references rather than +version stamps. Initialize a set with the fragment IDs in all active pinned +source snapshots. Retain each current-history record with a source fragment in +that set, and add its destination fragment IDs to the set. Repeat until no +additional records are selected. If a selected record is a legacy version, +retain all its groups and add every group's destination IDs. Retain whole +records and the downstream records they require. Add these pin requirements +to the existing consumers' requirements; do not replace the existing consumer +checks. + +Retain the external mapping files referenced by required history, including +records added after registration. Physical cleanup may remove a mapping file +only when no retained snapshot or required history references it, subject to +the existing maintenance concurrency limits. Replacing FRI metadata does not +by itself make its mapping files eligible for deletion. + +Pin retention does not change overlay/index correctness requirements or make +an unsupported overlay or rewrite operation valid. Restore and Overwrite carry the current active list, fixed sources, timestamps, and sticky bit. They never import the target snapshot's historical pin list. -Retain movement records needed to connect a pinned snapshot to the current -table layout. After a restore, records needed only for a discarded intermediate -layout may be removed if no other consumer requires them. Pins keep source -files available, but do not guarantee that a job can translate or publish its -results after a restore or overwrite; retention does not supply a missing -mapping. +Apply FRI retention to the resulting current history. Neither operation is +required to reconstruct discarded translation paths. Within that history, +retain movement records needed to connect a pinned snapshot to the current +table layout. Records needed only for a discarded intermediate layout may be +removed if no other consumer requires them. Pins keep source files available, +but do not guarantee that a job can translate or publish its results after a +restore or overwrite; retention does not supply a missing mapping. New branches and clones have empty pin lists and preserve the sticky bit when inherited. Detached snapshots cannot register or mutate authoritative pins. Other ordinary commits preserve the current list. All operations that rewrite -or trim FRI revalidate its retention requirements at commit. Detailed directional -rules are in [Transactions](transaction.md#ongoing-work-pin-transactions). +or trim FRI revalidate the applicable retention requirements at commit, subject +to the Restore and Overwrite rules above. Detailed directional rules are in +[Transactions](transaction.md#ongoing-work-pin-transactions). Pins do not protect newly written, uncommitted output. Jobs need a separate staging and publication strategy for those files. Before releasing a pin, a job diff --git a/docs/src/format/table/transaction.md b/docs/src/format/table/transaction.md index 80e092e7c16..e858332c51b 100644 --- a/docs/src/format/table/transaction.md +++ b/docs/src/format/table/transaction.md @@ -658,8 +658,12 @@ with no effect do not commit another version; committed mutations otherwise count as ordinary versions and cleanup intervals. All operations that remove FRI transitions must revalidate the union of current -pin and attached-consumer requirements, including the inclusive source-version -stamp boundary. Pin changes never loosen other transaction preconditions. +pin and attached-consumer requirements, using the representation-specific rules +in [Ongoing Work Pins](ongoing_work_pins.md#retention-and-data-operations). +Restore and Overwrite apply those rules to the resulting current history and +may discard translation paths as described there; they must still preserve +current pins and their source snapshots. +Pin changes never loosen other transaction preconditions. ## Conflict Resolution