Skip to content

Disable TCP keepalive on the handoff server's mock connections - #145

Open
franz1981 wants to merge 1 commit into
masterfrom
handoff-mock-no-keepalive
Open

franz1981 wants to merge 1 commit into
masterfrom
handoff-mock-no-keepalive

Conversation

@franz1981

Copy link
Copy Markdown
Owner

HandoffHttpServer opens one Apache HttpClient 5.6.1 connection to the mock per client connection (~10K at 10K connections). HttpClient's default SocketConfig enables TCP keepalive (5 s idle, 5 s interval, 3 probes). At 50K req/s most of these connections are idle, and after any load-generator stall (e.g. a wrk2 GC pause) thousands of them go idle at the same moment, so their probes fire as one batch: 5.6-8K probes within 100 ms, every ~5 s.

On loopback every packet is queued on the sending CPU's backlog (net.core.netdev_max_backlog, 1000 by default), so each batch overflows it:

  • 2-4.6K probes dropped per round (kfree_skb reason CPU_BACKLOG, matching the /proc/net/softnet_stat dropped deltas);
  • connections whose probes are dropped three times in a row are aborted by the server's kernel (RST reason TCP_KEEPALIVE_TIMEOUT, TcpExtTCPAbortOnTimeout); the mock logs Connection reset by peer;
  • the same backlog also drops server responses to the load generator, which then wait for a retransmit (>= 200 ms).

Unchanged server, 4 carriers, 10K connections, fixed 50K req/s: keepalive aborts in 3 of 4 runs (149, 1102, 517, 0); per-second p99 in the probe-batch seconds +15-20 ms.

This disables TCP keepalive on those connections; on loopback it has no purpose, and HttpClient still validates idle connections before reuse. With it, 2 runs: 0 aborts, 0 backlog drops, 0 resets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant