[puneetdixit200]

Describe the issue this Pull Request addresses

Closes #18758.

Summary and Changelog

Collect Lance column ranges as records are written, then pass them to the metadata index through HoodieWriteStat. This addresses the request to avoid reading newly written files again.

• Add an optional writer callback after row encoding and metadata population.
• Reuse the existing range collector for Spark and Flink inserts, merges, and bulk writes; copy mutable bounds before retaining them.
• Enable column stats and partition stats for Lance tables. Existing files use projected reads when an index is initialized or write statistics are unavailable.
• Handle null Flink field values and select the installed engine's Lance reader for initialization.

Impact

Lance tables can use the existing column-stats configuration and, when partitioned, the partition-stats index. The writer interface gains a default callback method. No new configuration keys or file-format changes are introduced.

Risk Level

medium

This enables metadata indexes that affect file and partition pruning. Tests cover populated and omitted metadata fields, nulls, reused input buffers, both index versions, and SQL inserts, updates, deletes, and compaction for partitioned and unpartitioned COW/MOR tables.

Documentation Update

Companion update to website/docs/metadata_indexing.md on asf-site: #20440

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

Validation

187 tests passed with no failures or skips on Java 17: common (94), Hadoop metadata utilities (19), collector/indexer (12), Spark (31), and Flink (31). Checkstyle, license checks, and git diff --check passed.

The local test classpath needed hadoop-client-runtime-3.3.5.jar alongside the resolved Hadoop client API. No dependency declarations were changed. In the commands below, HADOOP_TEST_RUNTIME points to that jar.

mvn -B -Punit-tests -pl hudi-spark-datasource/hudi-spark -am \
  '-Dtest=TestColumnStatsCollector,TestHoodieTableMetadataUtil,TestMetadataPartitionType,TestColumnStatsStreamingTracker,TestColumnStatsIndexer,TestHoodieSparkLanceWriter,TestHoodieRowCreateHandle,TestLanceDataSource#testSqlCommands' \
  -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -DwildcardSuites=abc \
  -Dspark3.5 -Dlance.skip.tests=false -Dmaven.test.additionalClasspath="$HADOOP_TEST_RUNTIME" test

mvn -B -Punit-tests -pl hudi-flink-datasource/hudi-flink -am \
  '-Dtest=TestHoodieRowDataLanceWriter,TestHoodieRowDataCreateHandle,TestHoodieRowDataLanceReader' \
  -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -DwildcardSuites=abc \
  -Dflink2.2 -Dlance.skip.tests=false -Dmaven.test.additionalClasspath="$HADOOP_TEST_RUNTIME" test

The full project test suite was not run.

[puneetdixit200]

Pushed e734a41b1 for the Azure failure on TestLanceDataSource.testSqlCommands.

Root cause: the functional test still asserted that Lance tables must not initialize column/partition stats metadata, but this branch now adds Lance column stats support and the common metadata test expects those partitions to be enabled where applicable.

Change:

• COLUMN_STATS is now asserted available for Lance tables.
• PARTITION_STATS availability is asserted to match whether the table is partitioned.

Local verification:

• git diff --check passed.
• I attempted focused Maven test-compile, but this Mac does not have a usable Hudi CI JDK: Maven on Java 25 fails in upstream hudi-io Lombok-generated members, and the installed Java 8 runtime is a JRE without tools.jar. The pushed CI run should give the real signal for the Spark functional test.

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the contribution! This PR adds Lance support to readColumnStatsFromMetadata and removes the gating on the COLUMN_STATS / PARTITION_STATS metadata partitions for Lance tables. The implementation correctly handles null/empty column lists, missing columns, and nested fields, and it follows the same record-iteration pattern used for log files. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. A few naming and simplification suggestions below.

cc @yihua

[voonhous]

@rahil-c Can you please take a look at this?

[puneetdixit200]

@hudi-bot run azure

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! The PR enables Lance metadata column stats and partition stats by implementing LanceUtils.readColumnStatsFromMetadata via a record-iteration approach (similar to the log file path), routing .lance files in readColumnRangeMetadataFrom, and removing the gates in MetadataPartitionType. One architectural question worth surfacing in the inline comments about the performance trade-off of reading all records. Please take a look at the inline comment, and this should be ready for a Hudi committer or PMC member to take it from here. A couple of small readability suggestions below.

[hudi-agent]

🤖 This path iterates every record in the file to compute stats. Since the Lance writer doesn't populate writeStat.columnStats, this full-file re-read will be triggered for every newly-written Lance base file on every commit (and for partition-stats aggregation on top of that). For Parquet, the equivalent path is cheap because it reads block-level statistics from the footer. @yihua have you considered (a) populating column stats on the writer side in HoodieBaseLanceWriter like Parquet does, or (b) exposing Lance's native per-fragment column statistics via the Java binding? Either would avoid the per-commit full-scan. Is the intent to land this read-based path now and optimize in a follow-up?

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

[puneetdixit200]

This patch keeps the read-based path so Lance column and partition stats can be generated now. The writer-side/native Lance stats path looks like larger follow-up work: the current Lance writer path only writes file data plus bloom/vector metadata, and I do not see an existing HoodieWriteStat column-stats hook to reuse there.

[puneetdixit200]

@hudi-bot run azure

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for working on this! The PR enables column/partition stats for Lance base files by implementing readColumnStatsFromMetadata via a projected record scan and removing the Lance gate from MetadataPartitionType. The control flow, resource cleanup, and downstream contract (one entry per targetFields element, never null) all check out. No new issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.

cc @yihua

[hudi-bot]

CI report:

• 8c17e31 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

you just read the files again, a more proper way is to collect the column stats during the write, we might need to add a hook in base file write to trigger the column stats collection.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.81818% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.78%. Comparing base (163a15a) to head (8c17e31).
⚠️ Report is 632 commits behind head on master.

Files with missing lines	Patch %	Lines
...n/java/org/apache/hudi/common/util/LanceUtils.java	77.77%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #18901   +/-   ##
=========================================
  Coverage     68.77%   68.78%           
- Complexity    29087    29096    +9     
=========================================
  Files          2517     2517           
  Lines        139822   139850   +28     
  Branches      17209    17209           
=========================================
+ Hits          96169    96190   +21     
- Misses        35880    35888    +8     
+ Partials       7773     7772    -1     

Flag	Coverage Δ
common-and-other-modules	44.24% <6.06%> (-0.01%)	⬇️
hadoop-mr-java-client	44.72% <6.06%> (-0.03%)	⬇️
spark-client-hadoop-common	48.02% <6.06%> (-0.03%)	⬇️
spark-java-tests	49.31% <81.81%> (+<0.01%)	⬆️
spark-scala-tests	45.20% <6.06%> (-0.02%)	⬇️
utilities	37.27% <6.06%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../apache/hudi/metadata/HoodieTableMetadataUtil.java	82.48% <100.00%> (+0.05%)	⬆️
...rg/apache/hudi/metadata/MetadataPartitionType.java	82.95% <100.00%> (-0.20%)	⬇️
...n/java/org/apache/hudi/common/util/LanceUtils.java	54.54% <77.77%> (+13.36%)	⬆️

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.