feat(inspect): add task-based scans for snapshots and history by wgtmac · Pull Request #1066 · apache/iceberg-cpp

wgtmac · 2026-10-27T05:50:13Z

Other Iceberg implementations

Java: Metadata tables reuse TableScan configuration where applicable. Static tables materialize rows into StaticDataTask; other tables use tasks that read manifests or manifest lists. Metadata tasks participate in the file-scan task hierarchy and have JSON parsers.
Go: Table.Inspect() exposes methods for individual metadata tables and returns Arrow record readers. Snapshots and history are materialized into a single batch.
Rust: table.inspect() exposes typed metadata-table objects with schemas and an async scan() returning an Arrow record-batch stream. The inspected snapshots, history, and manifests implementations build a batch before wrapping it in a stream.
PyIceberg: table.inspect exposes methods returning materialized PyArrow tables. Some manifest/file inspection paths use an executor to parallelize reads before collecting the results.

Design decisions

Attempted design	Trade-off / rationale	Decision
Separate metadata scan configuration, task planning, and task reading	Supports individual task consumption and a convenient `ToArrowArrayStream()` path. The builder selects output columns; the reader controls execution settings.	Keep
Reuse `TableScanBuilder` and inherit from the ordinary data-file task hierarchy	Would bring data-file scan contracts into an inspection API whose supported tables need neither data-file splits nor delete-file handling.	Drop
Require Java-compatible task JSON and cross-process task execution	Adds a wire-format and row-representation contract beyond the current local inspection scope.	Drop
Store intermediate rows as generic `StructLike` values or table-specific structs in a central variant	Duplicates data already present in metadata and makes each new table extend a shared row registry. Partition values and readable metrics also have schema-dependent structure.	Drop
Store owned Arrow arrays in the task, using ownership wrappers or deep copies for repeated reads	Couples task lifetime to execution buffers and complicates reuse. A reader can build independent batches from the retained source instead.	Drop
Retain shared ownership of the captured `TableMetadata` for snapshots/history tasks	Avoids a second row representation. Built scans retain their configuration and captured metadata across builder changes and normal table refreshes; mutable aliases must not modify the retained metadata.	Keep
Produce one reusable task for snapshots/history	Splitting snapshots/history adds scheduling overhead without useful I/O parallelism. Each reader maintains independent consumption state.	Keep
Materialize every Arrow batch when opening a task	Allocates the full output before consumption and performs unnecessary work when reading stops early.	Drop
Generate Arrow batches lazily in the reader	Limits producer-owned Arrow output to the batch being built and avoids converting unread rows. Source metadata and auxiliary indexes remain retained.	Keep
Configure Arrow batch size in the scan builder	Batch boundaries are a reading concern and do not affect the planned work. Batch size belongs in `MetadataScanTaskReader::Options`.	Drop
Implement concrete scans and column conversion in their corresponding table files	Keeps table-specific schemas and behavior together. Common utilities handle batch construction, projection validation, and stream ownership.	Keep
Publish options, filters, time travel, executor controls, or concrete task classes before implementing them	Creates unsupported or ineffective public configuration and prematurely fixes extension contracts.	Drop
Expose an estimated row count on metadata tasks	Has no current scheduling or execution consumer; it can be introduced when a concrete use requires it.	Drop

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Make the scan surface a subset of the API proposed in apache#1066 so the two can converge: - MetadataTable::NewScan() is virtual and const, and ScanProjection() is const. - MetadataScanBuilder::Select() resolves names immediately with the current CaseSensitive() setting and records errors for Build(); Project() sets the output schema directly, and Build() rejects a null projection. - MetadataScan::ToArrowArrayStream() is virtual and const. - Projection validation rejects unknown columns, type changes, and projecting a nullable column as required, with the same messages as apache#1066. - Drop the time-travel options together with TimeTravelMetadataTable and SnapshotSelection, which apache#1066 also removes; nothing on main implements them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…Build() Drop the MetadataScan class: separating a built scan from the Arrow stream it reads belongs to the task-based design in apache#1066. Build() now validates the options and returns the stream directly, and each call returns a new stream. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

- Add Table::Inspect() and task-based metadata scans while retaining column selection, projection, case sensitivity, and time-travel builder configuration from the merged inspection API. - Separate task planning from reading with reusable static tasks backed by captured table metadata. - Preserve scan configuration and metadata across builder changes and table refreshes. - Produce Arrow batches lazily through MetadataScanTaskReader, with reader-controlled batch sizes and independent stream lifetimes. - Keep canonical schemas and row conversion in each metadata table implementation, sharing batch construction and projection validation. - Consolidate projected-column append helpers with lazy batch construction. - Preserve all merged metadata inspection test scenarios and cover planned projections, task reuse, stream lifetimes, batching, and vector<bool>.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

manuzhang · 2026-11-01T15:20:32Z

+  // Walk the current snapshot's ancestry; the insert check stops on cycles.
+  std::unordered_set<int64_t> ancestors;
+  for (auto current = metadata->current_snapshot_id;
+       current != kInvalidSnapshotId && ancestors.insert(current).second;) {


the current snapshot might not exist

Good catch! Now I validate every snapshot and stop when anything is wrong.

manuzhang · 2026-11-01T15:25:18Z

+        if (next_task_ == tasks_.size()) {
+          return std::nullopt;
+        }
+        ICEBERG_ASSIGN_OR_RAISE(current_, reader_->Open(*tasks_[next_task_++]));


could tasks contain null pointer?

It is unlikely but does not hurt to add a safety check.

Copilot

🟡 Changes recommended

The unqualified Schema constructor parameter conflicts with the Schema() member, causing GCC to reject the class and preventing metadata_scan.cc from compiling.

1 open issue

Qualify Schema type to fix GCC compilation error · New

_{Review effort: Balanced}

manuzhang · 2026-11-01T15:33:16Z

                                               std::make_shared<MockCatalog>()));
  }

+  static Result<ArrowArrayStream> BuildStream(MetadataScanBuilder& builder) {


This is the same as that in the base class.

This is an oversight from my side.

Copilot

🔵 Needs a closer look

The unqualified Schema constructor parameter conflicts with the later Schema() member, causing a GCC compilation error; it must be qualified as iceberg::Schema.

1 open issue

Qualify Schema type to fix GCC compilation error

_{Review effort: Balanced}

Copilot

🟢 Approval recommended

No active review issues block approval of the metadata inspection changes.

No open issues 🎉

1 resolved since last review

Qualify Schema type to fix GCC compilation error

_{Review effort: Balanced}

Copilot AI balanced review requested due to automatic review settings October 27, 2026 05:50

Copilot AI reviewed Oct 27, 2026

wgtmac force-pushed the metadata_table_api_refactor branch from a0fb6c8 to 346d3b7 Compare October 27, 2026 06:00

Copilot AI balanced review requested due to automatic review settings October 27, 2026 06:00

Copilot AI reviewed Oct 27, 2026