Repository navigation
feat(format): specify expiring pins for ongoing work #10428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -59,6 +59,9 @@ it is not a dataset version or the sequence number of a rewrite. | |
| ordered compaction and stable partition. Each legacy group can be interpreted | ||
| as an ordered-compaction transition, forming one history with the new records. | ||
|
|
||
| A version-1 history is called **tagged**. If it also contains legacy versions, | ||
| it is called **mixed**. | ||
|
|
||
| Adding a transition or a new mapping type does not itself require increasing | ||
| `index_version`. Readers may skip unsupported mappings and fall back to scanning | ||
| where a complete translation path is unavailable. Writers must reject operations | ||
|
|
@@ -202,6 +205,30 @@ trimmed. Cleanup must retain intermediate transitions still needed to translate | |
| old addresses. External mapping files can be deleted only when no retained | ||
| dataset version references them. | ||
|
|
||
| ### Ongoing work retention | ||
|
|
||
| [Ongoing-work pins](../../table/ongoing_work_pins.md) are additional history | ||
| consumers before their jobs publish an index. Legacy-only histories use the | ||
| inclusive source-version boundary. Tagged and mixed histories retain records | ||
| reachable from pinned source fragments, including all downstream dependencies | ||
|
Comment on lines
+212
to
+213
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How is "pinned source fragments" determined? Do we read the pin's version and load all fragments at that version? |
||
| and whole legacy versions when any group is needed. Native tagged transitions | ||
| do not carry a dataset-version stamp. The exact retention rules are defined in | ||
| [Retention and data operations](../../table/ongoing_work_pins.md#retention-and-data-operations). | ||
| Retain the union of requirements from all pins and existing consumers. Validate | ||
| that union at the final commit target of every operation that trims or replaces | ||
| history, including work planned before registration. Restore and Overwrite | ||
| apply this check to the resulting current history; they may discard translation | ||
| paths while preserving current pins and their source snapshots. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is very LLM-ish. It reads like a dump of dense facts. Can we provide a bit of motivation first? Something like... An FRI can be removed (cleaned up) once all artifacts referencing the source fragments have been removed. For example, an index that maps search results to fragments A and B will prevent an FRI from getting cleaned up when A and B are source fragments in the FRI. This is because the FRI is still needed to map the results of index searches. Sometimes we also want to keep an FRI alive even when there are no artifacts referencing source fragments because those artifacts are currently being created by an in-progress task. For example, if an index is being trained against fragments A and B, this process may take several hours. We do not want the an FRI that maps A or B to be cleaned up while this training takes place. To solve this problem we utilize ongoing-work pins... |
||
|
|
||
| Compaction must record the movements active pins need and recheck those pins | ||
| before committing. Reject a result lacking required mappings so the caller can | ||
| replan. A pin does not require mappings solely into a discarded intermediate | ||
| layout, establish index freshness, or authorize publication of an old result. | ||
| Required records retain their external mapping files, including mappings added | ||
| after registration. A metadata replacement does not release those files while | ||
| retained snapshots or required history still reference them. Physical cleanup | ||
| remains subject to the existing [maintenance concurrency limits](../../table/ongoing_work_pins.md#maintenance-concurrency-limits). | ||
|
|
||
| ## Stable Row IDs | ||
|
|
||
| Older readers and writers did not expect both stable row IDs and a Fragment Reuse Index but are | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -28,6 +28,10 @@ A manifest describes a single version of the dataset. | |
| It contains the complete schema definition including nested fields, the list of data fragments comprising this version, | ||
| a monotonically increasing version number, and an optional reference to the index section that describes a list of index metadata. | ||
|
|
||
| [Ongoing work pins](ongoing_work_pins.md) record unfinished jobs' source snapshots | ||
| and FRI retention requirements in the current manifest. See that specification | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we prefer "in-progress" over "unfinished"? When I see the latter term I think "failed" or "abandoned" |
||
| for their lifecycle and the existing maintenance concurrency limits. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What does "existing maintenance concurrency limits" mean? |
||
|
|
||
| `max_allocated_field_id` is optional. If a manifest sets it, the dataset uses non-reusable field IDs from | ||
| that version onward. See [Field IDs](schema.md#field-ids). | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we trying to move away from the "tagged" terminology (in preference of "version 1")? I don't actually know. But yeah, Claude seems to like "tagged" (maybe we have a bunch of comments referring to it?)