Skip to content

benchmarking: measure kernel socket memory behind slow readers in the atunnel and Envoy benchmarks - #3319

Open
Chuang Wang (chuangw6) wants to merge 1 commit into
agent-substrate:mainfrom
chuangw6:sockmem-bench
Open

Chuang Wang (chuangw6) wants to merge 1 commit into
agent-substrate:mainfrom
chuangw6:sockmem-bench

Conversation

@chuangw6

@chuangw6 Chuang Wang (chuangw6) commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #3133. When a reader stalls, the data atunnel, atenet-router and the egress gateway relay waits in kernel socket buffers. That memory counts toward the proxy's memory limit, but no heap profile shows it. This adds a way to measure it, and a design doc with the results.

  • benchmarking/atunnel gets four slow-reader scenarios (--scenarios slow), on the egress and CONNECT paths in both directions. They report kernel socket memory per connection. It also gets throughput scenarios for egress download and CONNECT, and new run-docker.sh options: --network-delay and --tunnel-flag. The containers get a GKE node's TCP buffer limits.
  • benchmarking/envoy-sockmem is a new benchmark that does the same for an Envoy configured like the router or the egress gateway.
  • docs/network-socket-memory.md explains the problem and the options, and gives the results and the defaults.

Kernel socket memory per stalled connection, 100 connections, median of 3:

Who stops reading Off TCP_NOTSENT_LOWAT 16 KiB
actor, from a destination, through atunnel 753 KiB 62 KiB
client, from an actor, through the router 981 KiB 57 KiB
destination, from an actor, through the gateway (one tunnel) 1420 KiB 48 KiB

Three follow-up PRs set the option in atunnel, the router and the egress gateway. Not measured on GKE yet; the numbers come from Docker with GKE's TCP buffer limits.

  • Tests pass
  • Docs updated

@chuangw6
Chuang Wang (chuangw6) force-pushed the sockmem-bench branch 4 times, most recently from 1d185e1 to ad9949b Compare October 27, 2026 19:10
@bowei Bowei Du (bowei) added area/benchmarking area/network kind/feature An enhancement / feature request or implementation labels Oct 27, 2026
@chuangw6 Chuang Wang (chuangw6) changed the title Measure kernel socket memory behind slow readers in the atunnel and Envoy benchmarks benchmarking: measure kernel socket memory behind slow readers in the atunnel and Envoy benchmarks Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think LGTM - lets get #3320 first

Add slow-peer scenarios to the atunnel benchmark: connections whose reader
stalls after reading 4 MiB, in both directions on the egress and CONNECT
paths. The report reads the cgroup's memory.stat sock, the socket buffers
no heap profile shows, per held connection. Also add throughput scenarios
for egress download and CONNECT, and run-docker.sh options for added
round trip time (--network-delay) and atunnel flags (--tunnel-flag). The
containers and actor sandboxes get a GKE node's TCP buffer limits.

Add benchmarking/envoy-sockmem, which measures the same for an Envoy
configured like atenet-router or the egress gateway, with per-socket
options set from flags.

docs/network-socket-memory.md has the results and the defaults they lead
to.

Signed-off-by: Chuang Wang <chuangw@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/benchmarking area/network kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants