[v3-4-test] Sync recent CI changes from main

.github/fork-ci-gate.toml

@@ -16,6 +16,11 @@
 # under the License.
 
 # Fork CI gate (AIP-120). See dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md
+#
+# This file is part of the gate. A pull request whose CI runs in the contributor's fork may only
+# add, change or remove its author's own [pilot] entry here; the fork CI monitor closes it if it
+# changes anything else, because that change would decide where its own CI runs. See
+# "Gate protection" in the document above.
 
 # Master switch. When false the gate job lets every pull request run project CI, except
 # pull requests labelled `use fork ci` and pull requests of [pilot] participants. Push

.github/workflows/auto-approve-workflow-runs.yml

@@ -19,7 +19,8 @@
 # Approves the waiting workflow runs of external contributors' PRs to main when the change is safe to
 # run without a maintainer's look, and labels the others "need workflow approval". A PR whose CI runs
 # in the contributor's fork (AIP-120) is approved unless it changes CI definition files, and then gets
-# a comment saying why its runs wait. The rules are in
+# a comment saying why its runs wait. PRs from the repositories of trusted stakeholders (Amazon, Google,
+# Astronomer) are always approved. The rules are in
 # dev/breeze/src/airflow_breeze/utils/workflow_run_approval.py and documented in
 # contributing-docs/05_pull_requests.rst.
 #

.github/workflows/basic-tests.yml

@@ -332,6 +332,17 @@ jobs:
           GITHUB_TOKEN: ${{ github.token }}
           SKIP: ${{ inputs.skip-prek-hooks }}
           COLUMNS: "202"
+      - name: "Fail if static checks modified tracked files"
+        # prek does not report files that a nested-project hook modifies outside its own project,
+        # so check the tree independently.
+        shell: bash
+        run: |
+          if ! git diff --quiet; then
+            echo "::error::Static checks passed but modified tracked files. Commit the changes below."
+            git diff --stat
+            git diff | head -n 500 || true
+            exit 1
+          fi
 
   test-git-clone-on-windows:
     timeout-minutes: 5

.github/workflows/ci-amd.yml

@@ -164,7 +164,9 @@ jobs:
   gate:
     # Fork CI gate (AIP-120): decides whether project CI runs for this event. Configuration and
     # code come from the base branch so a pull request cannot change its own eligibility. See
-    # dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.
+    # dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md. A pull request whose CI runs
+    # in the contributor's fork and changes this job, or how build-info depends on it, is closed
+    # by the fork CI monitor ("Gate protection" in that document).
     name: "Fork CI gate"
     runs-on: ["ubuntu-slim"]
     timeout-minutes: 5

.github/workflows/ci-image-checks.yml

@@ -162,6 +162,17 @@ jobs:
           PR_NUMBER: ${{ inputs.newsfragment-pr-number }}
           DEFAULT_BRANCH: ${{ inputs.branch }}
           RUFF_FORMAT: "github"
+      - name: "Fail if static checks modified tracked files"
+        # prek does not report files that a nested-project hook modifies outside its own project
+        # (e.g. `breeze k8s setup-env` relocking uv.lock), so check the tree independently.
+        shell: bash
+        run: |
+          if ! git diff --quiet; then
+            echo "::error::Static checks passed but modified tracked files. Commit the changes below."
+            git diff --stat
+            git diff | head -n 500 || true
+            exit 1
+          fi
       - name: "Show prek log on failure"
         run: cat ~/.cache/prek/prek.log || true
         if: failure()

.github/workflows/finalize-tests.yml

@@ -111,10 +111,12 @@ jobs:
       - name: Checkout ${{ steps.constraints-branch.outputs.branch }}
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         with:
+          # Forks do not carry the constraints branches - diff against the canonical ones
+          repository: apache/airflow
           path: "constraints"
           ref: ${{ steps.constraints-branch.outputs.branch }}
-          persist-credentials: true
-          fetch-depth: 0
+          persist-credentials: false
+          fetch-depth: 1
       - name: "Download constraints from the constraints generated by build CI image"
         uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333  # v8.0.2
         with:

.github/workflows/fork-ci-monitor.yml

@@ -21,6 +21,9 @@
 # those pull requests, and deletes their project CI runs that only skipped every job. It never checks
 # out or runs pull request code. See
 # dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md.
+#
+# Part of the fork CI gate: a pull request whose CI runs in the contributor's fork and changes
+# this file is closed by this monitor ("Gate protection" in the document above).
 name: "Fork CI monitor"
 on:  # yamllint disable-line rule:truthy
   schedule:

contributing-docs/05_pull_requests.rst

@@ -275,6 +275,10 @@ automatically unless the PR changes files that define CI: anything under ``.gith
 ``Dockerfile*``, or a ``.pre-commit-config.yaml`` in any directory. Then the runs wait for a maintainer and a comment on the PR explains
 why. CI in your fork runs either way.
 
+The runs of PRs opened from the repositories of trusted stakeholders (``astronomer/airflow``,
+``aws-mwaa/upstream-to-airflow`` and ``VladaZakharova/airflow``) are always approved, so none of
+the rules above apply to them.
+
 Each new push is checked again, because approval applies to one commit. Only runs started in the
 last 24 hours are considered, so older waiting runs stay with maintainers. If your PR does not
 qualify, it gets the ``need workflow approval`` label and a maintainer approves the runs after

contributing-docs/18_contribution_workflow.rst

@@ -147,15 +147,17 @@ Set up your checkout and fork once:
 2. If fork CI applies to you, enable the ``Tests (AMD)`` workflow in your fork: today that means
    you joined the pilot or a maintainer labelled your pull request ``use fork ci``; once fork CI is
    switched on for everyone, it means you are not a committer, a collaborator or a member of an
-   exempted organisation. GitHub disables workflows in new forks. Otherwise keep it disabled, so
-   your pushes do not run CI twice:
+   exempted organisation. GitHub disables workflows in new forks. Sync your fork's ``main`` with
+   ``apache/airflow`` first, so it has the current version of the workflow:
 
    .. code-block:: bash
 
+      gh repo sync <you>/airflow --branch main
       gh workflow enable ci-amd.yml -R <you>/airflow
 
-   If the workflow is missing, your fork's ``main`` is older than it; sync it first with
-   ``gh repo sync <you>/airflow --branch main``.
+   Keep the workflow enabled for as long as you have open pull requests. If it gets disabled, from
+   your fork's Actions tab or by GitHub after 60 days without activity in your fork, the fork CI
+   monitor comments on your pull requests with the commands to turn it back on.
 
 3. Check everything at once. ``breeze ci audit`` prints one row per check and the command that fixes
    anything missing; ``--fix`` runs those commands after asking:
@@ -184,7 +186,10 @@ at 00:00 UTC:
    "your-login" = YYYY-MM-DDT00:00:00Z
 
 Every pull request you open after that instant uses fork CI; your earlier pull requests keep project
-CI. To leave the pilot, open a pull request that removes the line. A pilot entry gates your pull
+CI. To leave the pilot, open a pull request that removes the line. Your own ``[pilot]`` line is
+the only part of the fork CI gate a pull request using fork CI may change: one that changes the
+gate's code, the rest of its configuration or the ``gate`` job of ``ci-amd.yml`` is closed with an
+explanation. A pilot entry gates your pull
 requests whatever your role, so a committer in the pilot also enables the workflow;
 ``breeze ci audit`` tells you which applies to you. The design is described in
 `Fork CI for external contributors <../dev/breeze/doc/ci/08_fork_ci_for_external_contributors.md>`__.

contributing-docs/25_maintainer_pr_triage.md

@@ -250,7 +250,7 @@ The triage workflow uses the following labels and states:
 ## Pull requests gated to fork CI
 
 Pull requests labelled `fork ci required` run their CI in the contributor's fork. The scheduled
-fork CI reconciler (`breeze ci fork-ci-reconcile`, run by `fork-ci-monitor.yml`) owns four things
+fork CI reconciler (`breeze ci fork-ci-reconcile`, run by `fork-ci-monitor.yml`) owns five things
 for them, and the triage skill must treat those pull requests as out of scope for each:
 
 - **Draft state.** The reconciler keeps the pull request in draft until the fork run for its
@@ -261,6 +261,10 @@ for them, and the triage skill must treat those pull requests as out of scope fo
 - **`needs project ci`.** The reconciler adds it when the fork run is green but the pull request
   changes CI-defining files, and removes it once the state is anything else. These pull requests
   stay in draft and wait for a maintainer to review the CI change and add `use project ci`.
+- **Closing for changing the gate.** The reconciler closes a pull request that changes the fork
+  CI gate (its code, configuration other than the author's own `[pilot]` entry, the `gate` job
+  of `ci-amd.yml` or the monitor), and reports it on Slack. Add `use project ci` to accept such
+  a change.
 - **Closing after inactivity.** The reconciler closes a pull request whose fork CI has not run for
   `close_after_days`.
 

dev/breeze/doc/08_ci_tasks.rst

@@ -168,8 +168,8 @@ Pull requests from external contributors can run their CI in the contributor's f
 ``breeze ci audit`` checks, read-only, whether your checkout and fork are set up for that: ``gh``
 installed and authenticated, the ``upstream`` remote pointing at ``apache/airflow`` and ``origin`` at your
 fork, whether you are a committer, collaborator, member of an exempted organisation, pilot participant or
-external contributor, whether the ``Tests (AMD)`` workflow in your fork is enabled (or disabled, for those
-whose pull requests run project CI), how far your fork's ``main`` is behind, and the branch you are on.
+external contributor, whether the ``Tests (AMD)`` workflow in your fork is enabled (when your pull requests
+use fork CI), how far your fork's ``main`` is behind, and the branch you are on.
 Each failing check prints the command that fixes it; ``--fix`` runs those commands after asking.
 
 These are all available flags of ``audit`` command:
@@ -207,7 +207,8 @@ the waiting workflow runs of external contributors' PRs to ``main`` whose CI job
 than 310 job-minutes, that change at most 100 lines of source code and that do not touch CI definition files, and labels the other PRs ``need workflow approval``. PRs whose CI runs in
 the contributor's fork (AIP-120) only run the fork CI gate in ``apache/airflow``, so their runs are
 approved whatever their size unless they change CI definition files; such a PR also gets a comment
-explaining that its CI change needs a maintainer. For every PR it also
+explaining that its CI change needs a maintainer. The runs of PRs from the repositories of trusted
+stakeholders (Amazon, Google, Astronomer) are always approved. For every PR it also
 estimates the CI jobs selective checks will run and how long they take, and writes them to the job
 summary. Use ``--report-only`` to see the decisions without approving runs or changing labels. The rules
 are described in `CI workflows <ci/05_workflows.md>`_.

dev/breeze/doc/ci/05_workflows.md

@@ -128,6 +128,15 @@ those CI-defining files. Such a PR gets the `need workflow approval` label and o
 the author that the CI change needs a maintainer; see
 [Approving gated runs](08_fork_ci_for_external_contributors.md#approving-gated-runs).
 
+The runs of PRs whose head repository belongs to a trusted stakeholder (`astronomer/airflow` for
+Astronomer, `aws-mwaa/upstream-to-airflow` for Amazon, `VladaZakharova/airflow` for Google; listed in
+`TRUSTED_STAKEHOLDER_REPOSITORIES`) are approved whatever the PR changes or targets: only people with
+write access to those repositories can push the PR's commits. The repositories are matched by their
+numeric id, so a repository that later takes over one of these names is not trusted. None of the
+rules above apply to them, and of the rules below only the 24-hour window does: their runs are
+approved even when several open PRs share the head commit, or when the runs started before the PR
+was opened, retargeted or relabelled.
+
 Only runs created in the last 24 hours (`--max-run-age-hours`) are considered, so the workflow never
 approves a backlog of older waiting runs at once; those stay with maintainers. A commit that is the
 head of several open PRs is never approved automatically, because its runs cannot be told apart by