[HarshilMital]

Which issue does this PR close?

Part of #7080.

Rationale for this change

The high-partition group only ran with CompressionCodec::None, but a wide shuffle runs with LZ4 by default. At high partition counts every block is tiny, so the per-block codec cost (a fresh LZ4 frame encoder, its header and end mark) is spread over very few rows and was never measured.

What changes are included in this PR?

Each partition count in shuffle_writer_high_partition (200, 2000 and 8000 hash partitions, end to end) now also runs with Lz4Frame, next to the existing None rows. The existing IDs are unchanged. sample_size(10), as the group already uses.

Results

Machine: 8 vCPU Arm Neoverse-V2 (aarch64), 31 GiB RAM, rustc 1.99.1
Commit: 174bbc6. Median with Criterion's 95% interval.

Reproduce from native/:

cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- 'shuffle_writer_high_partition' --save-baseline main

Partitions	compression=None	compression=Lz4Frame	Lz4Frame vs None
200	14.24 ms (14.24 ms – 14.25 ms)	6.87 ms (6.85 ms – 6.88 ms)	-52%
2000	21.03 ms (20.84 ms – 21.23 ms)	21.74 ms (21.69 ms – 21.80 ms)	+3%
8000	44.02 ms (43.90 ms – 44.20 ms)	46.63 ms (46.49 ms – 46.82 ms)	+6%

Both columns come from the same run. The codecs cross over as partitions grow. At 200 partitions (about 400 rows per partition) LZ4 halves the write, most likely because it sends far fewer bytes to the file. By 2000 they are nearly level (LZ4 3% slower), and at 8000 (about 10 rows per partition) LZ4 is 6% slower: the blocks are too small to compress well, so it mostly adds its per-block cost.

How are these changes tested?

This PR only adds benchmark code under native/shuffle/benches/; no production code changes.

• cargo fmt --all -- --check and cargo clippy -p datafusion-comet-shuffle --all-targets -- -D warnings pass.
• cargo bench -p datafusion-comet-shuffle --bench shuffle_writer --no-run builds.
• Every new case runs once in Criterion's test mode: cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- --test 'shuffle_writer_high_partition/.*compression=Lz4Frame'.

[andygrove]

Thanks for adding the compressed rows. Measuring Lz4Frame next to None at the same partition counts is a useful comparison, and the existing None benchmark IDs are preserved.

The comment above high_partition_group still says "compression is disabled to isolate it", but the group now runs Lz4Frame as well. The paragraph you added right below it describes the Lz4Frame rows, so the two now contradict each other. Could you reword the first one so it says the None rows isolate the per-partition write path and the Lz4Frame rows add the per-block codec cost on top?