[ovr]

Check List

• Tests have been run in packages where changes have been made if available
• Linter has been run for changed code
• Tests for the changes have been added if not covered yet
• Docs have been added / updated if required

Description of Changes Made

Rebuilds the cubejs/rust-cross cross-compilation images and repoints every workflow, Cross.toml and the bake file at the new 08092026 tag. The aarch64 image was failing to build because zlib.net only serves the latest release, so the pinned 1.3.1 tarball 404s — it now fetches zlib 1.3.2 from GitHub releases. The x86_64-unknown-linux-gnu image moves from debian bullseye to bookworm (glibc 2.31 → 2.36, python 3.9 → 3.11), replacing the deprecated apt-key/add-apt-repository pair with a keyring drop-in plus an explicit sources list entry. Note this raises the minimum glibc for the published x86_64-unknown-linux-gnu binaries to 2.36. Also fixes stale bits in cross/README.md: the publish snippet referenced a non-existent x86_64-unknown-linux-musl-python bake target, and the glibc/clang notes were out of date.

[claude]

Claude finished @ovr's task in 3m 29s —— View job

---

Review complete — 1 high, 2 medium, 3 low

Tag bump is mechanically clean and complete (no 31072025 left anywhere). Main thing to confirm before merge: the bullseye→bookworm move raises the published x86_64 glibc floor to 2.36, which drops RHEL 9 / Amazon Linux 2023 (2.34) and Ubuntu 22.04 (2.35) for the standalone release artifacts.

Full review — findings, verification notes, and task list

Tasks

• Read PR diff against origin/master
• Verify tag bump is complete and consistent across all 12 files
• Review the two changed Dockerfiles (apt keyring, base image, zlib source)
• Check docker-bake.hcl targets against the README publish snippet
• Trace glibc floor impact through the publish/release pipeline
• Check for stale review threads to resolve (none exist)
• Post inline comments

Findings

#	Sev	Where	Issue
1	High	cross/x86_64-unknown-linux-gnu.Dockerfile:2	glibc floor 2.31 → 2.36 for published x86_64 binaries
2	Medium	cross/x86_64-unknown-linux-gnu.Dockerfile:8-9	Key in trusted.gpg.d/ is trusted for all repos; use signed-by=
3	Medium	.github/workflows/push.yml:242 and siblings	rust-cache key omits image tag → bullseye-built native deps restored into bookworm
4	Low	cross/README.md:15-19	Bake snippet omits the two bare (non--python) targets that CI actually pulls
5	Low	cross/aarch64-unknown-linux-gnu.Dockerfile:11	Still uses deprecated apt-key/add-apt-repository, now inconsistent with the x86 image
6	Low	cross/aarch64-unknown-linux-gnu.Dockerfile:41-51	No checksum on tarballs; openssl.org fetch has the same "only latest is served" rot as the zlib 404 this PR fixes

On #1 — the official Cube image is node:24.18.0-trixie-slim, so container users are unaffected. The exposure is cubestored-x86_64-unknown-linux-gnu and native-linux-x64-glibc-*.tar.gz, attached to releases and consumed outside this repo. Three specifics in the inline comment: it contradicts the repo's own "use minimal possible glibc" policy stated in both cross/README.md and publish.yml:598; arm64 stays at 2.31 so the two arches now have different support matrices; and it's a user-visible change riding along in a chore(ci): commit that fixes an unrelated aarch64 404. If intentional, it wants a release note. If not, pinning an archived LLVM suite on bullseye keeps 2.31.

What's good

• The zlib fix targets the right root cause. GitHub release URLs are immutable, so this class of 404 won't recur for zlib — unlike zlib.net, which only serves current.
• x86_64-unknown-linux-musl-python really was a phantom bake target; verified against docker-bake.hcl.
• ${LLVM_VERSION} brace-quoting made uniform on the clang- package.
• Tag bump verified complete — grep for 31072025 returns nothing, and every remaining rust-cross reference is either 08092026 or a ${CROSS_VERSION} interpolation.

Not verified

• Whether zlib-1.3.2 exists upstream. Network access was unavailable in this run (WebFetch not permitted), so I could not confirm the release or its asset. Everything rests on it — please confirm the aarch64 image actually built.
• Whether the 08092026 tags are pushed to Docker Hub. Same reason. All 12 files now point at tags I couldn't confirm are published; note that the image builds are manual and no CI job exercises these Dockerfiles, so a green CI run only proves the tags resolve, not that the Dockerfiles here match what was pushed.
• Whether apt.llvm.org still carries LLVM 18 for bookworm. Old suites do get pruned; a successful local build would settle it.

· branch [`fix-cross-image-build-date-tag`](https://github.com/cube-js/cube/tree/fix-cross-image-build-date-tag)

[claude]

Compatibility: this raises the glibc floor for published x86_64 artifacts from 2.31 to 2.36.

The image built here is what produces cubestored-x86_64-unknown-linux-gnu and the native-linux-x64-glibc-*.tar.gz release assets (.github/workflows/publish.yml:95, :600). Bookworm's glibc 2.36 becomes the hard minimum for anyone running those binaries outside the official Cube image (which is node:24-trixie-slim, so unaffected).

Hosts that worked before and would now fail with GLIBC_2.34/2.35 not found-style errors:

Distro	glibc
RHEL/Rocky/Alma 9, Amazon Linux 2023	2.34
Ubuntu 22.04 LTS	2.35
Debian 11, Ubuntu 20.04	2.31

Three things worth reconciling:

1. It contradicts this repo's own stated policy, both in cross/README.md ("Don't use modern unix*, which ship newest libc") and the # Please use minimal possible version of ubuntu, because it produces constraint on glibc comment sitting directly above the ubuntu-22.04 runner pin in publish.yml:598 — that runner is glibc 2.35, i.e. lower than what the container now emits.
2. It's asymmetric: aarch64-unknown-linux-gnu stays on focal/2.31, so arm64 users get a materially wider support matrix than x64 users for the same release.
3. The PR description frames the rebuild as fixing an aarch64 zlib 404; the bullseye→bookworm move is a separate, user-visible change riding along. If it's needed because apt.llvm.org dropped the bullseye/LLVM-18 suite, that's a fine reason — please say so, and if the floor bump is intentional it should land in release notes rather than only in a chore(ci): commit.

If the floor bump isn't intentional, bookworm's libc6 isn't the only option — staying on bullseye with a pinned/archived LLVM apt suite, or building against an older sysroot, keeps 2.31.

[ovr]

@claude It's okey, we cannot still stay on this outdated image too long.

[claude]

Claude finished @ovr's task in 33s —— View job

---

Acknowledged — glibc floor bump accepted

Understood, and that's a reasonable call: bullseye is EOL for standard support and pinning an archived LLVM suite to hold 2.31 would just move the maintenance cost somewhere less visible. Treating finding #1 as decided, not open — no further action requested from me on it.

Two small things so the decision doesn't get lost:

• Already documented. cross/README.md:15-17 (added in this PR) states the 2.36 / 2.31 split explicitly, so the floor is recorded next to the images themselves. The only gap left is a user-facing release note for the standalone cubestored-x86_64-unknown-linux-gnu and native-linux-x64-glibc-*.tar.gz artifacts — worth a line so RHEL 9 / AL2023 / Ubuntu 22.04 users hit a changelog entry rather than a GLIBC_2.36 not found.
• Asymmetry is now the interesting part. With x86_64 at 2.36 and aarch64 still on focal/2.31, the arches have different support matrices. Not a problem for this PR — just flagging that the same "don't stay too long" logic applies to the aarch64 image next time round, and bringing it to bookworm too would collapse the matrix back to one number.

Remaining open findings are all non-blocking:

#	Sev	Where	Note
3	Medium	.github/workflows/push.yml:242 + siblings	rust-cache key omits the image tag — first post-merge run restores bullseye-built native deps into bookworm. I can't push this one (App permissions block .github/workflows).
4	Low	cross/README.md:23-25	Bake snippet omits the two bare (non--python) targets that Cross.toml and CI actually pull.
6	Low	cross/aarch64-unknown-linux-gnu.Dockerfile:42,51	No tarball checksums; the openssl.org fetch has the same "only latest is served" rot as the zlib 404 this PR fixes.

Happy to push #4 (and the apt-key → keyring alignment on the aarch64 image) if you want them in this PR — say the word. Otherwise this looks good to merge once you've confirmed the 08092026 tags are on Docker Hub; I still can't verify that from here, and no CI job exercises these Dockerfiles, so a green run only proves the tags resolve.

--- · branch fix-cross-image-build-date-tag

[claude]

Good fix — GitHub release URLs are immutable, so this won't rot the way zlib.net/zlib-<pinned>.tar.gz does when upstream cuts a new release. Worth keeping in mind for the openssl.org fetch two blocks down (line 51), which has the same "only current release is served" failure mode; 1.1.1w is EOL and already moved to the old/ path once.

Two smaller things while this line is being touched:

• No integrity check on the tarball. Everything here is HTTPS-to-a-trusted-host so it's not a live hole, but a sha256sum -c against the pinned version would make the pin actually load-bearing. Pre-existing pattern across all three fetches, so fine to skip if you'd rather not expand scope.
• ENV ZLIB_VERSION (vs ARG) leaves the value baked into the runtime environment of every build that uses this image. Also pre-existing.

[claude]

Unrelated to the tag bump itself, but this bump is the first time it can bite: the Swatinem/rust-cache key a few lines down (shared-key: cubestore-release, key: ubuntu-24.04) doesn't include the cross-image tag, so the first run after this merges will restore a target/ populated inside the bullseye image into the bookworm container. Pure-Rust artifacts are keyed on rustc version and fine, but cc/cmake-built native deps (openssl, rocksdb, zlib) cached under target/*/build/ were compiled against glibc 2.31 headers and won't be rebuilt just because the container changed.

Usually this surfaces as a confusing link error rather than a silently bad binary, but it's cheap to avoid by folding the image tag into the cache key — e.g. key: ubuntu-24.04-08092026. Same applies to the sibling jobs in rust-cubestore.yml / rust-cubesql.yml / master.yml.

Note I can't push this change myself — GitHub App permissions don't allow modifying files under .github/workflows.

[claude]

Nice catch on the x86_64-unknown-linux-musl-python bake target below — confirmed there's no such target in docker-bake.hcl (only x86_64-unknown-linux-musl), so that snippet has been broken.

Two gaps left in this block:

• The bake list only covers x86_64-unknown-linux-gnu-python, aarch64-unknown-linux-gnu-python and x86_64-unknown-linux-musl. The two *-python targets inherits their base targets, but inherits copies config, it doesn't schedule the base as a separate pushed target — the base = "target:x86_64-unknown-linux-gnu" context builds it as an intermediate only. Yet Cross.toml and half the workflows pull the bare cubejs/rust-cross:x86_64-unknown-linux-gnu-08092026 / aarch64-... tags, which need their own --push. Worth adding the two bare targets to the snippet so the next person rebuilding doesn't ship a tag set that's missing exactly the images CI pulls most.
• Now that the two gnu images sit on different glibc floors, the "Verify versions" step only runs cc --version. A ldd --version line would make the documented 2.36/2.31 claim checkable rather than something a reader has to trust.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.21%. Comparing base (2c4fca5) to head (c56ced2).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11798      +/-   ##
==========================================
- Coverage   80.21%   80.21%   -0.01%     
==========================================
  Files         500      500              
  Lines      105649   105649              
  Branches     3885     3885              
==========================================
- Hits        84748    84747       -1     
- Misses      20352    20353       +1     
  Partials      549      549              

Flag	Coverage Δ
cube-backend	60.15% <ø> (ø)
cubesql	84.67% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude finished @ovr's task in 1m 7s —— View job

---

Incremental review of c56ced2 — keyring fix looks correct; 4 prior threads still open

chore(ci): scope the LLVM apt key to the LLVM repo correctly addresses the keyring finding, so I resolved that thread. No new issues introduced.

Full review — tasks, verification of the new commit, and status of prior findings

Tasks

• Diff c56ced2 against the previous head
• Verify the signed-by change is correct and complete
• Check whether the fix introduces new problems (path existence, armored-key support)
• List existing review threads and resolve mine that no longer apply
• Deduplicate against prior threads before posting anything new

The new commit

&& wget -O /etc/apt/keyrings/llvm-snapshot.asc https://apt.llvm.org/llvm-snapshot.gpg.key \
&& echo "deb [signed-by=/etc/apt/keyrings/llvm-snapshot.asc] https://apt.llvm.org/bookworm/ llvm-toolchain-bookworm-${LLVM_VERSION} main" > /etc/apt/sources.list.d/llvm.list \

This is the right form. Three things I checked:

• Scoping actually works. The key now validates only the llvm.list entry; deb.debian.org continues to be verified by the Debian archive keyring alone. That was the whole point of the finding.
• .asc extension matches the payload. llvm-snapshot.gpg.key is ASCII-armored despite the name, and apt only accepts armored keyrings via signed-by when the file ends in .asc — a .gpg extension here would have produced a Signed-By parse failure. Correct as written.
• /etc/apt/keyrings is shipped by the apt package (since apt 2.4; bookworm carries 2.6), so the missing mkdir -p is fine. I could not run a container in this session to confirm empirically — docker run wasn't permitted — but a wrong assumption here fails loudly at wget, not silently, so a successful image build settles it.

Resolved thread: r3959136... (keyring scoping).

Prior findings — still open

#	Sev	Where	Status
1	High	cross/x86_64-unknown-linux-gnu.Dockerfile:2	Unaddressed — glibc floor 2.31 → 2.36 for published x86_64 artifacts
3	Medium	.github/workflows/push.yml:242 + siblings	Unaddressed — rust-cache key omits image tag
4	Low	cross/README.md:19	Unaddressed — bake snippet omits the two bare (non--python) targets CI pulls
6	Low	cross/aarch64-unknown-linux-gnu.Dockerfile:42	Unaddressed — no tarball checksums; openssl.org fetch has the same "only latest is served" rot

Re-affirmed these four rather than re-posting; the unresolved state carries the signal. #1 is the only one I'd treat as merge-blocking-ish — it needs a decision, not necessarily a code change: if raising the x86_64 glibc floor to 2.36 (dropping RHEL 9 / Amazon Linux 2023 at 2.34 and Ubuntu 22.04 at 2.35) is intentional, it wants a release note, since it currently rides along inside a chore(ci): commit.

One leftover from the now-resolved thread that the new commit didn't cover: aarch64-unknown-linux-gnu.Dockerfile:11 still uses apt-key add -, which is deprecated and now inconsistent with the x86 image. Not a live vulnerability — worth aligning while both images are being rebuilt. Fix this →

Still not verifiable from CI

Unchanged from my earlier review — no network access in this run, and no CI job exercises these Dockerfiles, so a green run only proves the tags resolve:

• Whether zlib-1.3.2 exists upstream with that release asset.
• Whether the 08092026 tags are pushed to Docker Hub and match the Dockerfiles here.
• Whether apt.llvm.org/bookworm still carries the LLVM 18 suite.

A successful local docker buildx bake of both gnu targets would close all three at once.

· branch [`fix-cross-image-build-date-tag`](https://github.com/cube-js/cube/tree/fix-cross-image-build-date-tag)