Repository navigation
Conversation
isValidNode compared only the number of attachments on a node with the number of networks expected, so a node attached to the wrong networks passed. Compare the attachments' network IDs instead, and fail on a nil node. When watchNode and watchNetwork timed out, they passed a nil object to the validation function, or called a nil validation function, so a test which should have failed panicked instead. Validate the last object seen, if there was one, then fail the test. Signed-off-by: Cory Snider <csnider@mirantis.com>
The allocator gives each node a load balancer attachment on the ingress network when it allocates the node: at init, and on node and task events. Nodes were not revisited when an ingress network was created after the allocator had started, so a node got an attachment on it only when a later event reallocated the node. Until then, the daemon on a node running no tasks did not set up the ingress network, and its routing mesh refused connections to ingress-mode published ports. This happened when the ingress network was removed and a new one created, and when the first ingress network was created after the allocator started. Allocate all nodes once a new ingress network has been committed. Add tests for both cases to the allocator test suite. Also run them in swarmkit against a mock network allocator, as the full suite needs a network allocator provider from outside swarmkit. Signed-off-by: Cory Snider <csnider@mirantis.com>
If the allocator failed to allocate an ingress network created after it had started, it did not record the network as the ingress network, even once a retry allocated it. Until the allocator restarted, it failed to allocate services publishing ingress-mode ports with "ingress network is missing", without queueing them for a retry, and gave nodes no attachment on the ingress network. Record a new ingress network before allocating it, as init does, so that tasks are attached to it while its allocation is retried, and allocate all nodes once a retry commits it. While the ingress network is recorded but not allocated, hold services publishing ingress-mode ports for a retry. The network allocator drops a virtual IP on a network it has not allocated, so such a service was committed without a virtual IP on the ingress network and was not retried, leaving its published ports with no load balancer backends. This could already happen when the allocator failed to allocate the ingress network at init. If the allocator failed to allocate the ingress network at the start of init and succeeded later in init, it attached nodes to its copy of the network from before allocation, which has no driver state or IPAM configuration, both of which the daemon on each node dereferences to set up the ingress network. Record the copy of an ingress network whose allocation is committed. This also records an ingress network created while init runs, if there was none when init started. Test these cases against a mock network allocator which fails to allocate the ingress network once, and which drops virtual IPs as the real network allocator does. Signed-off-by: Cory Snider <csnider@mirantis.com>
corhere
requested review from
crazy-max and
dperny
and
a balanced review from Copilot
November 4, 2026 19:39
There was a problem hiding this comment.
🟡 Changes recommended
The startup guard can skip reserving existing services’ VIPs on healthy overlay networks when ingress allocation fails, allowing fresh allocations to reuse those addresses and cause conflicts.
1 open issue
What changed in this PR
This PR fixes ingress network replacement in the manager allocator so existing nodes—including idle nodes—receive attachments and can participate in the routing mesh.
Changes:
- Attach existing nodes after a new ingress network is committed, including successful retries.
- Track the allocated ingress network state and defer ingress-dependent services until allocation succeeds.
- Add regression tests for ingress creation, replacement, and retries, with a mock network allocator and stronger attachment checks.
| File | Description |
|---|---|
manager/allocator/network.go |
Updates ingress tracking, service gating, and node attachment allocation. |
manager/allocator/network_test.go |
Tests failed ingress allocation retries during runtime and initialization. |
manager/allocator/mock_networkallocator_test.go |
Adds a mock allocator with configurable allocation failures. |
manager/allocator/allocator_test_suite.go |
Adds ingress creation and replacement tests and improves validation helpers. |
Review effort: Balanced
Comment on lines
+1193
to
+1195
| if !nc.nwkAllocator.IsAllocated(nc.ingressNetwork) { | ||
| nc.unallocatedServices[s.ID] = s | ||
| return fmt.Errorf("ingress network %s is not allocated", nc.ingressNetwork.ID) |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #3311 +/- ##
==========================================
+ Coverage 14.71% 15.53% +0.81%
==========================================
Files 198 199 +1
Lines 93037 93154 +117
==========================================
+ Hits 13693 14467 +774
+ Misses 77997 77237 -760
- Partials 1347 1450 +103 🚀 New features to boost your workflow:
|
corhere
marked this pull request as draft
November 6, 2026 16:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

- What I did
When an ingress network is created while the allocator is running, the allocator does not attach the nodes it has already allocated to it. A node gets an attachment only when a later event reallocates it. Until then, a node running no tasks does not set up the ingress network. Its routing mesh refuses connections to ingress-mode published ports. This happens after
docker network rm ingressanddocker network create --driver overlay --ingress <name>.If allocating a new ingress network fails, the allocator does not record it as the ingress network, even after a retry allocates it. Until the allocator restarts, services publishing ingress-mode ports fail to allocate and nodes get no attachment on it.
If allocating the ingress network fails at the start of init and succeeds later in init, the allocator attaches nodes to its copy of the network from before allocation. That copy does not have a driver state or IPAM configuration. The daemon on each node dereferences both to set up the ingress network.
- How I did it
Allocate all nodes once a new ingress network is committed, from its create event or from a retry. Record a new ingress network before allocating it, as init does, and hold services publishing ingress-mode ports until it is allocated. Record the copy of an ingress network whose allocation is committed.
- How to test it
go test ./manager/allocator/runs the new tests against a mock network allocator. The new suite tests also run against moby's network allocator: vendor this branch into moby and rungo test -run TestAllocator ./daemon/libnetwork/cnmallocator/.- Description for the changelog
Fix the routing mesh on nodes running no tasks after the ingress network is replaced.
Created with: Claude Code (Opus 5.5)