atunnel, atenet-router and the egress gateway limit kernel send queues behind slow readers by chuangw6 · Pull Request #3320 · agent-substrate/substrate

Chuang Wang (chuangw6) · 2026-10-27T17:05:00Z

Part of #3133; design and data in docs/network-socket-memory.md (#3319).

atunnel, atenet-router and the egress gateway set TCP_NOTSENT_LOWAT to 16 KiB on their TCP sockets. A write then blocks once 16 KiB is waiting to be sent, so data behind a slow reader stays upstream instead of piling up in the proxy's kernel send queue, which grows to 4 MiB per socket on GKE nodes and counts toward the container's memory limit. One commit per component:

atunnel: --atunnel-tcp-notsent-lowat (default 16384), --atunnel-tcp-send-buffer and --atunnel-tcp-receive-buffer (default 0), on every connection it accepts or dials.
atenet-router: --router-tcp-notsent-lowat (default 16384), --router-tcp-send-buffer and --router-tcp-receive-buffer (default 0), as Envoy socket options on the TLS listeners and the clusters that dial atunnel.
egress gateway: TCP_NOTSENT_LOWAT 16 KiB on the egress listener and the four forward proxy clusters, in atenet-egress.yaml. A test checks they keep it.

0 leaves the kernel default. Fixed buffer sizes stay off: they cut throughput by 85 to 98% with a few milliseconds of round trip.

Kernel socket memory per stalled connection, 100 connections, median of 3:

Who stops reading	Off	Lowat 16 KiB
actor, from a destination, through atunnel	753 KiB	62 KiB
client, from an actor, through atunnel	737 KiB	201 KiB
client, from an actor, through the router	981 KiB	57 KiB
destination, from an actor, through the gateway (one tunnel)	1420 KiB	48 KiB

Throughput stayed within run-to-run noise.

atunnel sets the option on each accepted connection, because Go listens with MPTCP where the kernel enables it, and an MPTCP listener doesn't pass the option to the TCP connections it accepts. Envoy listens with plain TCP, so the router's client sockets and the gateway's sockets from atunnel inherit it from the listening socket.

Tests pass
Docs updated

Bowei Du (bowei) · 2026-10-27T23:37:27Z

+	tcp linuxsys.TCPBufferOptions
+}
+
+func (l tunedListener) Accept() (net.Conn, error) {


Use point as the receiver: l* tunedListener, this copies the struct on every method invocation, which can be surprising.

Bowei Du (bowei) · 2026-10-27T23:38:35Z

+// tunedListener sets its TCP options on every connection it accepts. A
+// connection the options can't be set on is closed, and the next one
+// accepted.
+type tunedListener struct {


listenerWithOptions

Bowei Du (bowei) · 2026-10-27T23:39:01Z

 	fs.Float64Var(&cfg.ParkedRequest.RetryJitter, "parked-request-retry-jitter", ingress.DefaultParkedRequestRetryJitter, "Random fraction in [0, 1) added to each retry delay to de-synchronize parked requests")
 	fs.IntVar(&cfg.ExtProcMaxRequests, "extproc-max-requests", 0, "Circuit-breaker max_requests for Envoy's ext_proc cluster; 0 (the default) derives it as twice --parked-request-max (minimum 1024). Explicit values must be >= --parked-request-max: every parked request holds one slot for its full wait, and the excess is fast-path headroom")

+	bindSocketOptionsFlags(fs, &cfg.TCP)


for consistency in naming, this should be called bindTCPSocketOptionsFlags

Lior Lieberman (LiorLieberman)

Nice!!

Lior Lieberman (LiorLieberman) · 2026-10-27T21:05:32Z

+        # the kernel. See docs/network-socket-memory.md.
+        upstream_bind_config:
+          socket_options:
+          - { description: TCP_NOTSENT_LOWAT, level: 6, name: 25, int_value: 16384, state: STATE_PREBIND }


should we make the format nicer here??

Lior Lieberman (LiorLieberman) · 2026-10-27T21:05:35Z

+        # TCP_NOTSENT_LOWAT on the sockets from atunnel, as on the forward
+        # proxy clusters below. See docs/network-socket-memory.md.
+        socket_options:
+        - { description: TCP_NOTSENT_LOWAT, level: 6, name: 25, int_value: 16384, state: STATE_PREBIND }


what is that for?

Lior Lieberman (LiorLieberman) · 2026-10-27T21:07:48Z

 	xdsSrv.SetTraceRootSamplingPercent(traceRootSamplingPercent)

 	xdsSrv.SetRouteTimeout(s.cfg.RouteTimeout)
+	xdsSrv.SetTCPBufferOptions(s.cfg.TCP)


is this the only place we should do it? in atenet-egress.yaml looks like you add it in a few different places.

Yes for the router. SetTCPSocketOptions hands the options to the xDS server, which puts them on both TLS listeners (https, connect-tls) and both actor clusters (actor_connect, actor_original_dst). The gateway's config is a static manifest, so each listener and cluster carries its own block there.

Lior Lieberman (LiorLieberman) · 2026-10-28T00:21:18Z

+`TCP_NOTSENT_LOWAT` only on kernels that copy it: GKE's COS 6.12 does, some
+other kernels don't. Internal listeners and clusters have no kernel sockets


Nit - are there kernels that dont copy it? accepted sockets inherit NOTSENT_LOWAT on every kernel afaict but not sure.

I think same applies in socketoptions.go and tunnel.go (where you talk about tunedListener).

If there is a kernel that does not support lets just say what it is

Also dont need to mention GKE imo

You're right, plain TCP always inherits it. I thought otherwise because an earlier CI run failed: on the Ubuntu runner, the accepted socket didn't have the listener's value.

The actual cause was MPTCP: Go listens with MPTCP by default when the kernel enables it, and MPTCP doesn't pass the option on (repro: https://github.com/chuangw6/substrate/actions/runs/40614204349). Envoy listens with plain TCP, so this only affected atunnel, which now sets it on each connection.

A slow reader lets a socket's send queue grow to tcp_wmem's maximum, 4 MiB on GKE nodes. That memory is charged to atunnel's cgroup but is not in its heap. With TCP_NOTSENT_LOWAT at 16 KiB, a write blocks once 16 KiB wait unsent, and the backlog stays upstream under flow control. In the atunnel benchmark this cut socket memory behind a stalled actor from 753 to 62 KiB per connection, and behind a stalled router-side client from 737 to 201 KiB, with throughput within run-to-run noise. Add --atunnel-tcp-notsent-lowat, default 16384, set on every TCP connection atunnel holds: each connection its ingress listeners and egress listeners accept, and its dials to actors and to the gateway. Accepted connections get the option themselves: Go listens with MPTCP where the kernel enables it, and an MPTCP listener does not pass TCP_NOTSENT_LOWAT to the TCP connections it accepts. -1 leaves the kernel default. --atunnel-tcp-send-buffer and --atunnel-tcp-receive-buffer set fixed buffer sizes, off by default (-1): they cap a connection at the buffer size per round trip. Signed-off-by: Chuang Wang <chuangw@google.com>

A client that stops reading lets each of its sockets queue up to tcp_wmem's maximum, 4 MiB on GKE nodes, charged to the Envoy container but missing from Envoy's statistics. With TCP_NOTSENT_LOWAT at 16 KiB, the backlog stays in Envoy's buffers and upstream. In the Envoy benchmark this cut socket memory behind a stalled client from 981 to 57 KiB per connection, with throughput within run-to-run noise. Add --router-tcp-notsent-lowat, default 16384, set as an Envoy socket option on the TLS listeners and on the clusters that dial atunnel. -1 leaves the kernel default. --router-tcp-send-buffer and --router-tcp-receive-buffer set fixed buffer sizes, off by default (-1): they cap a connection at the buffer size per round trip. Signed-off-by: Chuang Wang <chuangw@google.com>

An upload to a destination that reads slowly can put the tunnel's 1 MiB HTTP/2 stream window into the send queue of the gateway's socket to the destination, charged to the gateway's container. Set TCP_NOTSENT_LOWAT to 16 KiB on the egress listener and on the four dynamic forward proxy clusters, so the backlog stays behind Envoy's buffer limit and the tunnel's flow control. In the Envoy benchmark a stalled tunnel alone on its connection dropped from 1420 to 48 KiB of socket memory, with throughput within run-to-run noise. Fixed buffer sizes are left out: a fixed SO_RCVBUF cut a tunnel's throughput by 90% at 10 ms round trip. Signed-off-by: Chuang Wang <chuangw@google.com>

Chuang Wang (chuangw6) force-pushed the sockmem-atunnel branch 2 times, most recently from 47a2b2d to 14102f5 Compare October 27, 2026 17:17

Bowei Du (bowei) added area/network area/node kind/feature An enhancement / feature request or implementation labels Oct 27, 2026

Chuang Wang (chuangw6) force-pushed the sockmem-atunnel branch from 14102f5 to ac3a2d4 Compare October 27, 2026 19:56

Chuang Wang (chuangw6) changed the title ~~atunnel limits kernel send queues behind slow readers~~ atunnel, atenet-router and the egress gateway limit kernel send queues behind slow readers Oct 27, 2026

This was referenced Oct 27, 2026

atenet-router limits kernel send queues to slow clients #3321

Closed

The egress gateway limits kernel send queues to slow destinations #3322

Closed

Chuang Wang (chuangw6) requested review from Lior Lieberman (LiorLieberman) and Bowei Du (bowei) October 27, 2026 20:55

Bowei Du (bowei) reviewed Oct 27, 2026

View reviewed changes

Chuang Wang (chuangw6) force-pushed the sockmem-atunnel branch from bf1a3df to 2070fc9 Compare October 27, 2026 22:25

Chuang Wang (chuangw6) requested a review from Bowei Du (bowei) October 27, 2026 23:13

Bowei Du (bowei) reviewed Oct 27, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) previously approved these changes Oct 28, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) mentioned this pull request Oct 28, 2026

benchmarking: measure kernel socket memory behind slow readers in the atunnel and Envoy benchmarks #3319

Open

2 tasks done

Chuang Wang (chuangw6) added 3 commits October 27, 2026 18:27

Chuang Wang (chuangw6) dismissed Lior Lieberman (LiorLieberman)’s stale review via e21ec6a October 28, 2026 01:27

Chuang Wang (chuangw6) force-pushed the sockmem-atunnel branch from 2070fc9 to e21ec6a Compare October 28, 2026 01:27

		`TCP_NOTSENT_LOWAT` only on kernels that copy it: GKE's COS 6.12 does, some
		other kernels don't. Internal listeners and clusters have no kernel sockets

Conversation

Chuang Wang (chuangw6) commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Chuang Wang (chuangw6) Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Chuang Wang (chuangw6) commented Oct 27, 2026 •

edited

Loading

Chuang Wang (chuangw6) Oct 28, 2026 •

edited

Loading