Skip to content

fix(store): honor configured raft segment size - #3420

Merged
imbajin merged 3 commits into
apache:masterfrom
hugegraph:codex/store-raft-segment-size-20261027
Oct 28, 2026
Merged

imbajin merged 3 commits into
apache:masterfrom
hugegraph:codex/store-raft-segment-size-20261027

Conversation

@imbajin

@imbajin imbajin commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Before → After

Before: Store bound raft.maxSegmentFileSize but did not pass it to JRaft, so an explicit size had no effect.

After: pass the resolved segment size to log storage, retaining the effective 1 GiB default and legacy system-property fallback. Explicit Store configuration takes precedence. Document existing-file recovery, sparse files, and the encoded-entry size constraint.

Validation boundary

The source PR reports five real log-storage tests, including configuration precedence and reopening legacy segments with a 20 MiB entry, plus reactor/build checks. This is author-reported validation; it is not a new independent runtime or performance acceptance. Existing review follow-ups remain applicable.

Source PR: 🔗 hugegraph/hugegraph #341

Visual overview

Raft segment configuration and verified blank segment isolation

Verified-blank preallocated segments with incompatible sizes are atomically isolated before allocation; used segments remain unchanged. Unknown or truncated headers and symlinks fail closed. Isolated files still consume disk. This diagram is not power-loss, performance or RC acceptance.

- Pass resolved segment sizes to the JRaft log storage factory.
- Preserve the effective 1 GiB default and legacy property fallback.
- Test explicit sizes, backend selection and legacy segment recovery.
- Document per-segment limits and sparse disk accounting.
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.41096% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.47%. Comparing base (608cb1d) to head (1a38322).
⚠️ Report is 9 commits behind head on master.

Files with missing lines Patch % Lines
...he/hugegraph/store/raft/RaftSegmentLogStorage.java 89.55% 2 Missing and 5 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3420      +/-   ##
============================================
+ Coverage     57.27%   57.47%   +0.20%     
+ Complexity     6577     2447    -4130     
============================================
  Files           776      777       +1     
  Lines         68881    68952      +71     
  Branches       9431     9443      +12     
============================================
+ Hits          39449    39628     +179     
+ Misses        25804    25688     -116     
- Partials       3628     3636       +8     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø) Carriedforward from ceec45a
hstore 36.25% <ø> (+0.16%) ⬆️
pd 52.17% <ø> (+<0.01%) ⬆️ Carriedforward from ceec45a
server_hbase 42.18% <ø> (ø) Carriedforward from ceec45a
server_memory 55.98% <ø> (ø) Carriedforward from ceec45a
server_rocksdb 42.65% <ø> (ø) Carriedforward from ceec45a
store 34.14% <90.41%> (+1.00%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Isolate only validated incompatible blank allocations under the RocksDB directory lock.
Sync the directory before JRaft can reuse numeric filenames, preserving nonempty logs.
Release failed initialization resources and make repeated failure cleanup safe.
Cover native grow/shrink recovery, unknown headers and symlinks; document retained files.
MrJs133
MrJs133 previously approved these changes Oct 28, 2026
@bitflicker64

bitflicker64 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

HG review summary: round R10-contract-check (head ceec45a)

Score: 9.0 / 10. Two Minor issues are confirmed. Nothing blocking was found.

A report counts only if this round's tool records show that the reviewer loaded the gh-review skill and read AGENTS.md and hugegraph-store/AGENTS.md. Qualified lanes: all 4 (Codex, Claude, Copilot, AGY pool A); R10 is complete. Codex, Copilot and AGY found nothing new. None of them checked apache/hugegraph-doc, so they do not contradict finding 1. AGY treats the fail-closed behavior as a design choice and does not show that JRaft cannot recover, so finding 2 stands. R10 replaces the R9 assessment below; R9 scored 9.5 with only finding 2.

1. (Minor, new in R10) Website docs are stale, and the PR doesn't link a paired doc PR.

  • This PR makes raft.maxSegmentFileSize take effect, sets its default to 1 GiB, and adds it to the shipped application.yml.
  • apache/hugegraph-doc still lists the key as default 67108864 (64 MB) and "not set" in shipped config. See content/en/docs/quickstart/hugegraph/hugegraph-hstore.md:192 and content/cn/.../hugegraph-hstore.md:190.
  • Root AGENTS.md says that user-visible configuration changes must link a paired apache/hugegraph-doc PR, and that a follow-up issue alone does not satisfy the rule. The PR body links none, and a doc-repo PR search for maxSegmentFileSize finds nothing.
  • Before this PR the value was ignored, so the stale 64 MB was harmless. Now an operator who copies it gets real 64 MiB segments.

2. (Minor, carried from R9 and re-verified) The new pre-pass fails closed on a crash-truncated trailing segment that JRaft itself recovers.

  • RaftSegmentLogStorage.normalizeBlankSegments() runs before super.onInitLoaded().
  • isBlank() throws on a short header (RaftSegmentLogStorage.java:147) or bad magic (:155), so init fails for that partition.
  • JRaft 1.3.14 processCorruptedHeaderFiles renames such a last-sequence file to .corrupted and continues. In the base commit, segment storage was the default and recovered this way.
  • A standalone repro rerun this round against the head blob gave plain JRaft init=true and this class init=false.
  • The deployment guide documents fail-closed on purpose. Consider leaving a short or corrupt header on the highest-sequence file to JRaft.

Checked, not defects:

  • The effective default stays at 1 GiB, because JRaft's default reads the same system property.
  • AppConfig → HgStoreNodeService → PartitionEngine.createLogStorage → builder wiring is correct.
  • The reset path skips re-normalization.
  • The new test runs in the reuseForks=false core execution.
  • No new dependencies.

Verification limits: all four lanes reviewed source statically; one also ran the repro. None ran PartitionLogStorageTest, Maven, a cluster, or fault injection. Native sessions may have resumed, so this round is not a blind benchmark.

Automated multi-lane review (HugeGraph review squad); read-only, no approval.

Retain upstream learner-promotion retry changes.
Keep both log-storage and membership regressions enabled.
Preserve configured segment sizes and verified blank isolation.
@imbajin
imbajin merged commit 33f7422 into apache:master Oct 28, 2026
15 checks passed
@imbajin
imbajin deleted the codex/store-raft-segment-size-20261027 branch October 28, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants