Skip to content

fix(store): refresh recovery shard metadata - #3417

Merged
imbajin merged 4 commits into
apache:masterfrom
hugegraph:codex/store-recovery-shard-metadata-20261027
Oct 28, 2026
Merged

imbajin merged 4 commits into
apache:masterfrom
hugegraph:codex/store-recovery-shard-metadata-20261027

Conversation

@imbajin

@imbajin imbajin commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Before → After

Before: stale persisted SHARDGROUP metadata could omit a migrated Store and exclude its partition before JRaft restored the durable configuration.

After: validate PD membership and persisted graph partitions, then refresh derived metadata once per group only when its engine is absent. Preserve running engines and durable Raft configuration; missing, invalid, or unavailable PD metadata blocks recovery and retains local data.

Validation boundary

The source PR reports 19 tests with no failures or skips on Java 17, including a real JRaft node reopening a persisted one-voter configuration without replacing it with a different PD bootstrap. This does not claim independent multi-node migration/restart acceptance.

Source PR: 🔗 hugegraph/hugegraph #344

Visual overview

Store startup recovery and authoritative Raft membership

Direct PD partition NOT_FOUND skips deleted-graph recovery while retaining local data; surviving graphs still recover. Other metadata failures remain fail-closed. Durable Raft configuration stays authoritative, and empty-log bootstrap preserves learner roles. No independent multi-node migration or RC acceptance is claimed.

- Validate persisted groups against direct PD membership before startup recovery.
- Refresh derived shard metadata only before its engine starts, retaining durable Raft configuration.
- Cover persisted configuration recovery and refusal guards, and document restart checks.
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.54054% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.57%. Comparing base (608cb1d) to head (5711436).
⚠️ Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
.../apache/hugegraph/store/meta/PartitionManager.java 93.84% 1 Missing and 3 partials ⚠️
.../java/org/apache/hugegraph/pd/client/PDClient.java 83.33% 1 Missing ⚠️
...g/apache/hugegraph/store/pd/DefaultPdProvider.java 50.00% 0 Missing and 1 partial ⚠️
...java/org/apache/hugegraph/store/pd/PdProvider.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3417      +/-   ##
============================================
+ Coverage     57.27%   57.57%   +0.30%     
+ Complexity     6577     2479    -4098     
============================================
  Files           776      776              
  Lines         68881    68962      +81     
  Branches       9431     9453      +22     
============================================
+ Hits          39449    39708     +259     
+ Misses        25804    25613     -191     
- Partials       3628     3641      +13     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø) Carriedforward from f005ef6
hstore 36.11% <ø> (+0.02%) ⬆️
pd 52.34% <50.00%> (+0.18%) ⬆️
server_hbase 42.18% <ø> (ø) Carriedforward from f005ef6
server_memory 55.98% <ø> (ø) Carriedforward from f005ef6
server_rocksdb 42.65% <ø> (ø) Carriedforward from f005ef6
store 35.00% <90.54%> (+1.85%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The direct PD shard group refresh, the running-engine guard and the role-aware /learner peers match the PR goal, and the earlier learner finding is fixed at this head. Startup now exits the Store when PD no longer has a persisted graph partition, where the base skipped that partition.

Preserve direct PD lookup errors and distinguish graph absence from failed RPCs.
Retain skipped graph data and paths while recovering remaining valid graphs.
Verify native learner startup and await leader callback publication in recovery tests.
MrJs133
MrJs133 previously approved these changes Oct 28, 2026
Merge ASF master a185f1d, including the learner retry fix.
Keep recovery and learner test includes and operation guidance.
Validate the combined Store recovery and membership regressions.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: Recovery validates PD membership and persisted partitions before refreshing derived SHARDGROUP metadata, keeps durable Raft configuration and running engines authoritative, and both earlier findings (learner bootstrap, dropped-graph NOT_FOUND) are fixed with tests that run in CI.

@imbajin
imbajin merged commit a47478d into apache:master Oct 28, 2026
17 checks passed
@imbajin
imbajin deleted the codex/store-recovery-shard-metadata-20261027 branch October 28, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants