Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/src/format/index/system/frag_reuse.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ it is not a dataset version or the sequence number of a rewrite.
ordered compaction and stable partition. Each legacy group can be interpreted
as an ordered-compaction transition, forming one history with the new records.

A version-1 history is called **tagged**. If it also contains legacy versions,
it is called **mixed**.
Comment on lines +62 to +63

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we trying to move away from the "tagged" terminology (in preference of "version 1")? I don't actually know. But yeah, Claude seems to like "tagged" (maybe we have a bunch of comments referring to it?)


Adding a transition or a new mapping type does not itself require increasing
`index_version`. Readers may skip unsupported mappings and fall back to scanning
where a complete translation path is unavailable. Writers must reject operations
Expand Down Expand Up @@ -202,6 +205,30 @@ trimmed. Cleanup must retain intermediate transitions still needed to translate
old addresses. External mapping files can be deleted only when no retained
dataset version references them.

### Ongoing work retention

[Ongoing-work pins](../../table/ongoing_work_pins.md) are additional history
consumers before their jobs publish an index. Legacy-only histories use the
inclusive source-version boundary. Tagged and mixed histories retain records
reachable from pinned source fragments, including all downstream dependencies
Comment on lines +212 to +213

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is "pinned source fragments" determined? Do we read the pin's version and load all fragments at that version?

and whole legacy versions when any group is needed. Native tagged transitions
do not carry a dataset-version stamp. The exact retention rules are defined in
[Retention and data operations](../../table/ongoing_work_pins.md#retention-and-data-operations).
Retain the union of requirements from all pins and existing consumers. Validate
that union at the final commit target of every operation that trims or replaces
history, including work planned before registration. Restore and Overwrite
apply this check to the resulting current history; they may discard translation
paths while preserving current pins and their source snapshots.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very LLM-ish. It reads like a dump of dense facts. Can we provide a bit of motivation first?

Something like...

An FRI can be removed (cleaned up) once all artifacts referencing the source fragments have been removed. For example, an index that maps search results to fragments A and B will prevent an FRI from getting cleaned up when A and B are source fragments in the FRI. This is because the FRI is still needed to map the results of index searches.

Sometimes we also want to keep an FRI alive even when there are no artifacts referencing source fragments because those artifacts are currently being created by an in-progress task. For example, if an index is being trained against fragments A and B, this process may take several hours. We do not want the an FRI that maps A or B to be cleaned up while this training takes place.

To solve this problem we utilize ongoing-work pins...


Compaction must record the movements active pins need and recheck those pins
before committing. Reject a result lacking required mappings so the caller can
replan. A pin does not require mappings solely into a discarded intermediate
layout, establish index freshness, or authorize publication of an old result.
Required records retain their external mapping files, including mappings added
after registration. A metadata replacement does not release those files while
retained snapshots or required history still reference them. Physical cleanup
remains subject to the existing [maintenance concurrency limits](../../table/ongoing_work_pins.md#maintenance-concurrency-limits).

## Stable Row IDs

Older readers and writers did not expect both stable row IDs and a Fragment Reuse Index but are
Expand Down
1 change: 1 addition & 0 deletions docs/src/format/table/.pages
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ nav:
- Table Config: table_config.md
- Versioning: versioning.md
- Transactions: transaction.md
- Ongoing Work Pins: ongoing_work_pins.md
- Layout: layout.md
- Branch & Tag: branch_tag.md
- Row ID & Lineage: row_id_lineage.md
Expand Down
30 changes: 30 additions & 0 deletions docs/src/format/table/branch_tag.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,36 @@ which means `/` would create a logical subdirectory (e.g., `bugfix/issue-123`, `
1.manifest
```

### Deletion with ongoing work pins

Deletion must resolve the [authoritative head](versioning.md#current-state-for-pin-decisions)
and reject if its active-pin list is nonempty, identifying the blocking pins.
A pin past `expires_at` still blocks deletion until removed by a committed
ReleasePin or ExpirePins. Deletion does not implicitly revoke pins. Force deletion
must perform the same check. Existing tag and dependent-branch checks also apply.

For force deletion of an orphan with no branch reference, the pin check may be
skipped only when authoritative resolution definitively establishes that no
attached manifest exists. A listing or resolution failure, or a manifest that
exists but cannot be read or validated, must fail deletion. If a selected head
disappears during inspection, restart resolution as specified in
[Format Versioning](versioning.md#current-state-for-pin-decisions); do not treat
that failed read alone as proof of absence.

A corrupt current manifest therefore requires repair or administrator recovery
outside the ordinary branch-deletion protocol. `force` does not bypass pin
protection; this proposal defines no operator override.

This follows existing tag protection: the check is separate from physical
deletion and is not atomic with concurrent acquisition. Registration after a
deleter's inspection does not force that deleter to observe the new pin.
Delayed deletion after recovery and name reuse also remains a known gap. See
[Maintenance concurrency limits](ongoing_work_pins.md#maintenance-concurrency-limits).

Resolving those existing coordination gaps is outside the pin proposal. Branch
names remain reusable; pins add no name-retirement record, deletion marker,
branch layout, or reference-history format.

## Tagging

### Tag Name
Expand Down
4 changes: 4 additions & 0 deletions docs/src/format/table/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ A manifest describes a single version of the dataset.
It contains the complete schema definition including nested fields, the list of data fragments comprising this version,
a monotonically increasing version number, and an optional reference to the index section that describes a list of index metadata.

[Ongoing work pins](ongoing_work_pins.md) record unfinished jobs' source snapshots
and FRI retention requirements in the current manifest. See that specification

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we prefer "in-progress" over "unfinished"? When I see the latter term I think "failed" or "abandoned"

for their lifecycle and the existing maintenance concurrency limits.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "existing maintenance concurrency limits" mean?


`max_allocated_field_id` is optional. If a manifest sets it, the dataset uses non-reusable field IDs from
that version onward. See [Field IDs](schema.md#field-ids).

Expand Down
Loading
Loading