Skip to content

manager/allocator: attach nodes to a new ingress network - #3311

Draft
corhere wants to merge 3 commits into
moby:masterfrom
corhere:fix-ingress-node-attachments
Draft

corhere wants to merge 3 commits into
moby:masterfrom
corhere:fix-ingress-node-attachments

Conversation

@corhere

@corhere corhere commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

- What I did

When an ingress network is created while the allocator is running, the allocator does not attach the nodes it has already allocated to it. A node gets an attachment only when a later event reallocates it. Until then, a node running no tasks does not set up the ingress network. Its routing mesh refuses connections to ingress-mode published ports. This happens after docker network rm ingress and docker network create --driver overlay --ingress <name>.

If allocating a new ingress network fails, the allocator does not record it as the ingress network, even after a retry allocates it. Until the allocator restarts, services publishing ingress-mode ports fail to allocate and nodes get no attachment on it.

If allocating the ingress network fails at the start of init and succeeds later in init, the allocator attaches nodes to its copy of the network from before allocation. That copy does not have a driver state or IPAM configuration. The daemon on each node dereferences both to set up the ingress network.

- How I did it

Allocate all nodes once a new ingress network is committed, from its create event or from a retry. Record a new ingress network before allocating it, as init does, and hold services publishing ingress-mode ports until it is allocated. Record the copy of an ingress network whose allocation is committed.

- How to test it

go test ./manager/allocator/ runs the new tests against a mock network allocator. The new suite tests also run against moby's network allocator: vendor this branch into moby and run go test -run TestAllocator ./daemon/libnetwork/cnmallocator/.

- Description for the changelog

Fix the routing mesh on nodes running no tasks after the ingress network is replaced.

Created with: Claude Code (Opus 5.5)

isValidNode compared only the number of attachments on a node with the
number of networks expected, so a node attached to the wrong networks
passed. Compare the attachments' network IDs instead, and fail on a nil
node.

When watchNode and watchNetwork timed out, they passed a nil object to
the validation function, or called a nil validation function, so a test
which should have failed panicked instead. Validate the last object
seen, if there was one, then fail the test.

Signed-off-by: Cory Snider <csnider@mirantis.com>
The allocator gives each node a load balancer attachment on the ingress
network when it allocates the node: at init, and on node and task
events. Nodes were not revisited when an ingress network was created
after the allocator had started, so a node got an attachment on it only
when a later event reallocated the node. Until then, the daemon on a
node running no tasks did not set up the ingress network, and its
routing mesh refused connections to ingress-mode published ports. This
happened when the ingress network was removed and a new one created, and
when the first ingress network was created after the allocator started.

Allocate all nodes once a new ingress network has been committed.

Add tests for both cases to the allocator test suite. Also run them in
swarmkit against a mock network allocator, as the full suite needs a
network allocator provider from outside swarmkit.

Signed-off-by: Cory Snider <csnider@mirantis.com>
If the allocator failed to allocate an ingress network created after it
had started, it did not record the network as the ingress network, even
once a retry allocated it. Until the allocator restarted, it failed to
allocate services publishing ingress-mode ports with "ingress network
is missing", without queueing them for a retry, and gave nodes no
attachment on the ingress network. Record a new ingress network before
allocating it, as init does, so that tasks are attached to it while its
allocation is retried, and allocate all nodes once a retry commits it.

While the ingress network is recorded but not allocated, hold services
publishing ingress-mode ports for a retry. The network allocator drops a
virtual IP on a network it has not allocated, so such a service was
committed without a virtual IP on the ingress network and was not
retried, leaving its published ports with no load balancer backends.
This could already happen when the allocator failed to allocate the
ingress network at init.

If the allocator failed to allocate the ingress network at the start of
init and succeeded later in init, it attached nodes to its copy of the
network from before allocation, which has no driver state or IPAM
configuration, both of which the daemon on each node dereferences to set
up the ingress network. Record the copy of an ingress network whose
allocation is committed. This also records an ingress network created
while init runs, if there was none when init started.

Test these cases against a mock network allocator which fails to
allocate the ingress network once, and which drops virtual IPs as the
real network allocator does.

Signed-off-by: Cory Snider <csnider@mirantis.com>
@corhere
corhere requested review from crazy-max and dperny and a balanced review from Copilot November 4, 2026 19:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The startup guard can skip reserving existing services’ VIPs on healthy overlay networks when ingress allocation fails, allowing fresh allocations to reuse those addresses and cause conflicts.

1 open issue
What changed in this PR

This PR fixes ingress network replacement in the manager allocator so existing nodes—including idle nodes—receive attachments and can participate in the routing mesh.

Changes:

  • Attach existing nodes after a new ingress network is committed, including successful retries.
  • Track the allocated ingress network state and defer ingress-dependent services until allocation succeeds.
  • Add regression tests for ingress creation, replacement, and retries, with a mock network allocator and stronger attachment checks.
File Description
manager/​allocator/​network.go Updates ingress tracking, service gating, and node attachment allocation.
manager/​allocator/​network_test.go Tests failed ingress allocation retries during runtime and initialization.
manager/​allocator/​mock_networkallocator_test.go Adds a mock allocator with configurable allocation failures.
manager/​allocator/​allocator_test_suite.go Adds ingress creation and replacement tests and improves validation helpers.

Review effort: Balanced

Comment on lines +1193 to +1195
if !nc.nwkAllocator.IsAllocated(nc.ingressNetwork) {
nc.unallocatedServices[s.ID] = s
return fmt.Errorf("ingress network %s is not allocated", nc.ingressNetwork.ID)
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.29032% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 15.53%. Comparing base (323aa0b) to head (3236681).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3311      +/-   ##
==========================================
+ Coverage   14.71%   15.53%   +0.81%     
==========================================
  Files         198      199       +1     
  Lines       93037    93154     +117     
==========================================
+ Hits        13693    14467     +774     
+ Misses      77997    77237     -760     
- Partials     1347     1450     +103     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants