Skip to content

Nightly stress test: skip the warm-up when checking the Master RSS slope - #70683

Open
charzl wants to merge 1 commit into
saltstack:3008.xfrom
charzl:ci/stress-test-skip-warmup-3008
Open

charzl wants to merge 1 commit into
saltstack:3008.xfrom
charzl:ci/stress-test-skip-warmup-3008

Conversation

@charzl

@charzl charzl commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

The nightly stress test failed on the Master RSS slope (244 KB/sec, limit 10 KB/sec). analyze_stats.py fitted the last 30m, which is the whole default run, so the start-up ramp-up of RSS was counted as a leak.

  • The slope window now starts 10 minutes after the stress starts, and is at least the last 10 minutes. The workflow records the start time in STRESS_START_EPOCH. The warm-up can be changed with STRESS_WARMUP_SECONDS.
  • The shortest duration option is now 20m (10m warm-up + 10m of data); it was 10m.
  • The Master RSS slope limit is 15 KB/sec (was 10 KB/sec). Master RSS is not flat: it moves by a few hundred MB about every 6 minutes, so the fitted slope keeps some noise. Two 30m runs measured about 13 and 7 KB/sec.
  • The counter-reset check in check_salt_prometheus_metrics now uses the real run length instead of a fixed 30m.

What issues does this PR fix or reference?

Fixes #70677

Notes

  • The two 30m runs were done on a fork, on top of master. analyze_stats.py and nightly-stress-test.yml are identical on master and 3008.x, so the same code applies here. I have not run this branch itself.
  • A 20m run is judged on 10 minutes of data. Use 30m or longer for a reliable leak check.

Merge requirements satisfied?

CI only: no docs or tests.

  • Docs
  • Changelog
  • Tests written/updated

Commits signed with GPG?

No

The slope was computed over the last 30m, which is the whole default run,
so the start-up ramp-up of RSS looked like a leak.

- analyze_stats.py: use the data after the 10m warm-up (at least the last
  10m), based on the real run length from STRESS_START_EPOCH.  The
  counter-reset check uses the real run length too.  Master RSS limit is
  15 KB/sec (was 10): RSS oscillates by a few hundred MB every ~6
  minutes, so the fit keeps some noise.
- workflow: record STRESS_START_EPOCH; shortest duration is now 20m
  (10m warm-up + 10m of data)
- changelog entry

This branch was previously deployed

1 inactive deployment
ci — 11e809b3 Deployed Nov 4, 2026 by charzl via Build Onedir Packages / RPM (x86_64) #28441
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants