[potiuk]

Most test jobs on the self-hosted RunsOn runners can't use the CPUs they get. Jobs that run with --run-in-parallel (DB, lowest-dependencies and special tests) start one serial pytest per test type, so a job never uses more CPUs than it has test types. Logs from canary run 40517104168 show this:

• Postgres core DB job (8 CPUs): runs 3 test types (Core, Other, Serialization). Serialization finishes in 2 min, Other in about 21 min and Core in 23 min, so for most of the job only 2 of the 8 CPUs are busy.
• Lowest-dependencies provider jobs (32 CPUs): about 20 providers per job, so at most about 60% of the CPUs can be used.
• Integration-core and system tests: one pytest each, on 32-CPU / 128 GB runners.

The core test types are split into two matrix jobs to keep the Non-DB xdist collection inside the 16 GB of a public runner. On RunsOn that job gets 128 GB. The DB halves each still last as long as their slowest test type, so on RunsOn the split only doubles the job count. It also ran the single system test once per slice.

Changes, for RunsOn only (public runners are unchanged):

• Core test types run in one matrix job instead of 2.
• Lowest-dependencies provider tests run in 3 jobs instead of 5, which is about one provider per CPU of the 32-CPU runner.
• Integration-core and system tests move to the 8+16 / 32+64 size class, the same as the provider integration tests.
• Provider tests run one provider per test type, in one matrix job on the 32-CPU size class, instead of the long-running providers plus one group of all the others. The group's single serial pytest left the other CPUs idle. The Non-DB (xdist) jobs read a new providers-xdist-test-types-list-as-strings-in-json output that never splits the providers, since xdist runs them all in one pytest anyway.
• The longest-running providers (amazon, google, standard) are split into several test types such as Providers[amazon:hooks], with Providers[amazon:other] running the rest of the provider, so a provider job does not wait on one long serial pytest. Lowest-dependencies jobs on RunsOn are sized at about 40 test types each: 3 jobs for all providers, one for a change touching a few.
• Self-hosted runners split Docker's default address range into /24 subnets before the tests start. Each parallel test type creates its own compose network, and the default pools fit only about 30, fewer than the large runners run at once.
• Jobs with one process (end-to-end tests against one Airflow stack, a Gradle test run) or a minute or two of work move to the 8+16 or 4+8 size classes, and on RunsOn the Helm unit tests run all their packages in one job per Kubernetes version instead of one job per package.
• The RunsOn cost summary step also charts each job's CPU and memory use (metrics: cpu,memory), so the size classes can be set from measurements.
• Self-hosted runners raise fs.aio-max-nr, which a few dozen parallel MySQL servers exhaust.
• The provider compatibility tests keep the grouped provider test types, since each of their test types installs an Airflow version first; the per-provider list is a separate providers-split-test-types-list-as-strings-in-json output.

The selective-checks doc covers the new sizing and slicing rules, and a test covers the slice counts on both runner types.

Checks run locally: the selective-checks test suite (passes), the fast static checks (pass, except the docker and network hooks the sandbox blocks), and a roborev review (no findings). This PR has the canary and full tests needed labels so the full matrix runs on RunsOn and the new job durations can be compared with the canary run above.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 5

Generated-by: Claude Opus 5 following the guidelines

🤖 Generated with Claude Code

[github-actions]

The re-run of the jobs stopped by EC2 Spot interruptions finished: attempt 2 passed.

[github-actions]

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-4-test