Skip to content

simplify build workflow and move cibuildwheel configuration to pyproject.toml - #1371

Open
zacharyburnett wants to merge 15 commits into
GalSim-developers:releases/2.8from
zacharyburnett:simplify_build_workflow
Open

zacharyburnett wants to merge 15 commits into
GalSim-developers:releases/2.8from
zacharyburnett:simplify_build_workflow

Conversation

@zacharyburnett

@zacharyburnett zacharyburnett commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

we can move these specific cibuildwheel configurations into pyproject.toml, and it will run them for each platform / arch

also, cibuildwheel can use emulation to build macOS Intel x86 wheels on the Apple Silicon platform

also also, we can run the wheels.yml workflow on every pull request to build wheels (with on: pull_request) and use an if conditional to only attempt publishing to PyPI on an actual release

@zacharyburnett
zacharyburnett marked this pull request as ready for review October 20, 2026 18:46
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch 2 times, most recently from b1f643b to 1843404 Compare October 20, 2026 18:52
@rmjarvis

Copy link
Copy Markdown
Member

Why? What was wrong with the previous version? This is a pretty big overhaul with no (apparent to me) benefit. I'll let you make your case though.

also also, we can run the wheels.yml workflow on every pull request to build wheels (with on: pull_request) and use an if conditional to only attempt publishing to PyPI on an actual release

This in particular seems purely wasteful. Why would we want to build wheels on every random PR?

@zacharyburnett

zacharyburnett commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Why? What was wrong with the previous version? This is a pretty big overhaul with no (apparent to me) benefit. I'll let you make your case though.

This reduces the amount of jobs actually running, and GitHub runners spun up, from 4 (glibc Linux, muslc Linux, macOS x86, macOS ARM, plus the Linux containers that cibuildwheel starts) to 2 (GitHub-supplied ubuntu-latest and macos-latest, plus manylinux and musllinux spun up by cibuildwheel) by letting cibuildwheel handle the build matrix and also build the macOS x86 wheel with emulation alongside the macOS ARM64 wheel. We also don't need to install fftw on the final publish job in order to build the source dist, because we can build it alongside the wheel in the build jobs

Additionally, moving the meaningful configuration to pyproject.toml makes it a lot more explicit on what is actually different from defaults, and makes it much easier to maintain. In my experience configuration should be in one place or it will diverge and people will forget to update it. For instance, setting requires_python once in the pyproject.toml is much better IMO than copying CIBW_SKIP: cp36* cp37* cp38* pp* four times buried in the workflow YAML; if GalSim ever needs new syntax features that aren't backported and needs to update the minimum Python, there's much less configuration change required

also also, we can run the wheels.yml workflow on every pull request to build wheels (with on: pull_request) and use an if conditional to only attempt publishing to PyPI on an actual release

This in particular seems purely wasteful. Why would we want to build wheels on every random PR?

I personally like to run a build job on every pull request so I know about build failures early, otherwise I'm blindsided by build failures when it comes to release time and have to scramble. I can take that out though if you prefer; there's always the workflow_dispatch that can be run manually

@rmjarvis

Copy link
Copy Markdown
Member

This reduces the amount of jobs actually running, and GitHub runners spun up, from 4 (glibc Linux, muslc Linux, macOS x86, macOS ARM, plus the Linux containers that cibuildwheel starts) to 2

This IMO is not particularly desirable. The reason for having 4 there was to reduce the human wait time for them all to finish building.

@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch 15 times, most recently from 0a0a325 to 4d2c51b Compare October 23, 2026 18:31

@rmjarvis rmjarvis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I seem some benefits to the simplification here. But I have some significant concerns with the current implementation.

Comment thread .github/workflows/wheels.yml Resolved
Comment thread .github/workflows/wheels.yml Resolved
Comment thread .github/workflows/wheels.yml Resolved
Comment thread .github/workflows/wheels.yml Resolved
Comment thread .github/workflows/wheels.yml Resolved
Comment thread pyproject.toml Resolved
Comment thread pyproject.toml Resolved
Comment thread .github/workflows/wheels.yml Resolved
Comment thread .github/workflows/wheels.yml Resolved
Comment thread .github/workflows/wheels.yml Resolved
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch 3 times, most recently from b730a25 to 467f4f9 Compare October 29, 2026 18:29
Comment thread .github/workflows/wheels.yml Resolved
@zacharyburnett

Copy link
Copy Markdown
Contributor Author

CI failing with data errors?

Connecting to drive.usercontent.google.com (drive.usercontent.google.com)|173.194.64.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2040 (2.0K) [text/html]
Saving to: ‘des_data.tar.gz’

     0K .                                                     100% 27.6M=0s

2026-10-29 19:05:54 (27.6 MB/s) - ‘des_data.tar.gz’ saved [2040/2040]


gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Error: Process completed with exit code 2.

https://github.com/GalSim-developers/GalSim/actions/runs/40936831285/job/123669477221?pr=1371#step:9:21

@zacharyburnett
zacharyburnett marked this pull request as draft October 30, 2026 13:01
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch 2 times, most recently from f00693b to 992cde4 Compare October 30, 2026 13:34
Comment thread .github/workflows/wheels.yml Resolved
@zacharyburnett
zacharyburnett marked this pull request as ready for review October 30, 2026 13:48
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from 6065d55 to 74345ad Compare October 30, 2026 17:42
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch from 74345ad to a3a2a2e Compare November 2, 2026 13:32
@zacharyburnett
zacharyburnett force-pushed the simplify_build_workflow branch 2 times, most recently from 42cc394 to eebbefe Compare November 2, 2026 13:44
@zacharyburnett

Copy link
Copy Markdown
Contributor Author

@rmjarvis I appreciate your patience as I iterated through this, thank you!

@zacharyburnett

zacharyburnett commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

one additional thing; since the version is explicitly defined and manually updated in _version.py, there is the possibility that running the wheels.yml workflow (even as it is currently, before this PR) in between releases could build and release a version of GalSim with unwanted commits; for instance, if you ran that workflow right now from the release/2.8 branch I'm pretty sure it would build wheels for Python 3.15 and then upload them to PyPI as 2.8.5, but those wheels would actually have more commits than the other 2.8.5 wheels

In other projects I manage we use setuptools_scm to automatically derive the version from the Git tags, and if it's in between tagged commits it would make a version like 2.8.5.dev2 (2 commits ahead of 2.8.5) or something similar. I can add a config for that if you'd like, or save it for another PR

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants