Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,11 +287,33 @@ Keep the command and required run ID on one line. This pins an eligible complete
`run-sweep.yml` PR run, including failed or cancelled runs with usable results. The
run may come from any PR; its commit need not be part of this one.

Native video discovery requires both `video-sweep-manifest` and a bundle named
`h3-results-<run>-<attempt>-<64-hex recipe fingerprint>`. Discovery only identifies
candidate artifacts. Reuse authorization, automatic source selection, merge selection and
staging all download the retained video files and validate them against the trusted
run ID, attempt and head before admission, including mixed LLM/video runs. The same
check is available offline:

```bash
python -m infx.results.video_artifacts --artifacts-dir artifacts \
--run-id "$RUN_ID" --run-attempt "$RUN_ATTEMPT" --head-sha "$HEAD_SHA"
```

The validator requires the full selected video matrix, frozen measured request
population, native power verification and matching media/file hashes. Partial runs,
missing cells and stale attempts fail. The generic `validate_reusable_sweep_artifacts`
command also checks native video when its selected-matrix manifest is present.
Executed checkout identity remains separate from the workflow's PR head. Retained
artifact validation does not rerun media decoding or establish perceptual quality.
Each downloaded ZIP is limited to 256 MiB compressed and 1 GiB expanded; larger
bundles fail closed.

The legacy `/reuse-sweep-run <run_id>` remains equivalent. Bare `/reuse-sweep-run`
selects the latest successful eligible run automatically; bare `/use` is rejected.
Both names share authorization, validation, and reactions.

Source validation checks identity and artifacts, not full-matrix coverage.
For LLM and agentic results, source validation checks identity and artifacts, not
full-matrix coverage. Native video additionally requires its complete selected matrix.
Acceptance does not
certify a green full sweep. Verify coverage and pin the run ID when a full sweep
is required by the review process.
Expand Down
69 changes: 53 additions & 16 deletions inferencex-e2e/infx/klaud/github.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@
from __future__ import annotations

import base64
import hashlib
import io
import json
import os
import re
import stat
import subprocess
import time
import zipfile
Expand Down Expand Up @@ -154,35 +157,69 @@ def file_at(repository: str, head: str, path: str) -> bytes:
return base64.b64decode(result["content"])


def download_json(repository: str, artifact: dict, destination: Path) -> None:
"""Write the JSON members of ``artifact`` under ``destination``, raising VerificationError.

Archives over 256 MiB, unsafe paths and duplicates are rejected. Artifact code never runs.
"""
def download_files(
repository: str,
artifact: dict,
destination: Path,
*,
token: str | None = None,
json_only: bool = False,
) -> None:
"""Read bounded artifact files without trusting ZIP paths or executing content."""
limit = 256 * 1024 * 1024
if artifact["expired"] or not 0 < artifact["size_in_bytes"] <= limit:
raise VerificationError("Artifact unavailable or too large")
archive = subprocess.check_output(
[
"gh",
"api",
f"repos/{repository}/actions/artifacts/{int(artifact['id'])}/zip",
],
["gh", "api", f"repos/{repository}/actions/artifacts/{int(artifact['id'])}/zip"],
env={**os.environ, "GH_TOKEN": token} if token is not None else None,
timeout=60,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) On a slower link, video reuse or merge-with-reuse can fail even though each bundle is within the documented 256 MiB limit. Every gh api .../zip download in download_files is killed after a fixed timeout=60. That fixed limit was sized for small JSON artifacts. A 256 MiB bundle needs about 4.3 MiB/s to finish in time. A timeout raises TimeoutExpired, which validate_video_run re-raises as RuntimeError, so the run is rejected. Fix: scale the timeout with size_in_bytes, or use a generous floor for non-JSON downloads, while keeping the short limit for the small JSON downloads through download_json that klaud relies on.

Why this was flagged

Condition: a bundle whose download takes longer than 60 s, meaning a large bundle or a link slower than about 36 Mbit/s. This is most likely for merge_with_reuse, which maintainers run locally (uv run ... merge_with_reuse <pr>). merge_with_reuse.find_eligible_run (merge_with_reuse.py:279) calls sweep_runs.validate_video_run, which calls download_files(..., token=token) for each selected artifact (sweep_runs.py:116). At github.py:69-73, subprocess.check_output runs gh api with timeout=60. subprocess.TimeoutExpired is a SubprocessError, so sweep_runs.py:121 catches it and raises RuntimeError "invalid native video artifacts". merge_with_reuse then aborts before posting the reuse comment, and reuse/stage paths reject the run. The size check at github.py:67 accepts up to 256 MiB, and the README says only larger bundles fail. On base, this 60 s limit applied only to klaud's small JSON artifacts, and video runs were never downloaded.

Verification: In inferencex-e2e/infx/klaud/github.py, download_files accepts any archive with 0 < artifact["size_in_bytes"] <= limit, where limit is 256 MiB. It then runs gh api .../zip with a fixed 60 s timeout that does not depend on size. validate_video_run re-raises TimeoutExpired as RuntimeError, and at merge_with_reuse.find_eligible_run (line 279) this aborts the whole selection.

)
if len(archive) > limit:
raise VerificationError("Artifact archive too large")
if artifact.get("digest") and artifact["digest"] != (
"sha256:" + hashlib.sha256(archive).hexdigest()
):
raise VerificationError("Artifact digest mismatch")
try:
with zipfile.ZipFile(io.BytesIO(archive)) as source:
members = [item for item in source.infolist() if item.filename.endswith(".json")]
if not members or sum(item.file_size for item in members) > limit:
raise VerificationError("Invalid artifact JSON size")
members = [
item
for item in source.infolist()
if not item.is_dir() and (not json_only or item.filename.endswith(".json"))
]
expanded_limit = limit if json_only else 4 * limit
if (
not members
or len(members) > 4000
or sum(item.file_size for item in members) > expanded_limit
):
raise VerificationError("Invalid artifact expanded size")
paths = set()
for member in members:
path = PurePosixPath(member.filename)
if path.is_absolute() or ".." in path.parts or "\\" in member.filename:
raise VerificationError("Invalid artifact path")
target = destination.joinpath(*path.parts)
mode = stat.S_IFMT(member.external_attr >> 16)
if (
path.is_absolute()
or ".." in path.parts
or "\\" in member.filename
or not path.parts
or mode not in {0, stat.S_IFREG}
or path in paths
or any(parent in paths for parent in path.parents)
or any(path in previous.parents for previous in paths)
):
raise VerificationError("Invalid or duplicate artifact path")
paths.add(path)
for member in members:
target = destination.joinpath(*PurePosixPath(member.filename).parts)
if target.exists():
raise VerificationError("Duplicate artifact path")
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(source.read(member))
except zipfile.BadZipFile as error:
raise VerificationError("Invalid artifact archive") from error


def download_json(repository: str, artifact: dict, destination: Path) -> None:
"""Read bounded JSON only; never execute artifact code."""
download_files(repository, artifact, destination, json_only=True)
11 changes: 7 additions & 4 deletions inferencex-e2e/infx/matrix/revision.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,14 @@ def module_path(self) -> str:
FINGERPRINTER = "infx.matrix.fingerprint"
CONFIG_DIRS = ("configs", ".github/configs")
NESTED_PROJECT = "inferencex-e2e/"
VIDEO_SPECS = "experimental/video-generation/specs"
SNAPSHOT_PATHS = (
"infx",
"utils/matrix_logic",
*CONFIG_DIRS,
"benchmarks/single_node/srt-slurm-recipes",
"benchmarks/multi_node/srt-slurm-recipes",
VIDEO_SPECS,
)
INHERITED_ENV = ("PATH", "HOME", "LANG", "TMPDIR")
_FORK_REPLY = "infx-fork-exit "
Expand Down Expand Up @@ -328,11 +330,12 @@ def snapshot(ref: str) -> Iterator[Revision]:
for entry in listing.split(b"\0")[:-1]:
metadata, path = entry.split(b"\t", 1)
objects[os.fsdecode(path)] = metadata.split()[2]
if any(path.startswith(NESTED_PROJECT) for path in objects):
nested = any(path.startswith(NESTED_PROJECT) for path in objects)
if nested:
objects = {
path.removeprefix(NESTED_PROJECT): oid
path: oid
for path, oid in objects.items()
if path.startswith(NESTED_PROJECT)
if path.startswith((NESTED_PROJECT, VIDEO_SPECS + "/"))
}
blobs = io.BytesIO(
subprocess.run(
Expand All @@ -354,7 +357,7 @@ def snapshot(ref: str) -> Iterator[Revision]:
destination = Path(temp_dir, path)
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_bytes(content)
revision = Revision(Path(temp_dir))
revision = Revision(Path(temp_dir) / NESTED_PROJECT if nested else Path(temp_dir))
inputs = [*revision.master_configs, revision.runner_config]
if missing := [
Path(path).relative_to(revision.root).as_posix()
Expand Down
Loading
Loading