Skip to content

Python 3.13 and 3.14 in CI and published images - #3479

Open
fellhorn wants to merge 8 commits into
flyteorg:masterfrom
fellhorn:dennis/py314-ci
Open

Python 3.13 and 3.14 in CI and published images#3479
fellhorn wants to merge 8 commits into
flyteorg:masterfrom
fellhorn:dennis/py314-ci

Conversation

@fellhorn

@fellhorn fellhorn commented Oct 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Tracking issue

Closes flyteorg/flyte#6993 (together with #3478)

Why are the changes needed?

#3478 makes flytekit installable and importable on Python 3.14, but CI and the published images still stop at 3.12, so nothing guards 3.13/3.14 support. Enabling 3.13/3.14 in CI surfaced several plugin and dev-environment issues, most of them caused by dropping the protobuf<5 pin in #3478 (dependencies that were previously held back now resolve to newer, incompatible releases).

What changes were proposed in this pull request?

Python 3.13 support from #3465 (cherry-picked)

Python 3.14 on top:

  • ci: test and publish Python 3.14: nightly matrix 3.10–3.14, pull requests run 3.14; py3.14 images in build_image.yml / pythonpublish.yml; 3.14 classifier; tensorflow only installed below 3.14 (no 3.14 wheels yet) and the tensorflow extras tests skipped on 3.14; 3.14 plugin exclusions mirroring 3.13 plus flytekit-mlflow (needs tensorflow). The 3.13 flytekit-mlflow exclusion from Add Python 3.13 support #3465 is dropped, it only existed because of the protobuf<5 pin.
  • ci: adjust plugin matrix for Python 3.13 and 3.14: do not force numpy<2 for the onnx/sqlalchemy/pandera plugin jobs on 3.13+ (the pandas/pyarrow releases with 3.13+ wheels need numpy>=2); exclude greatexpectations and whylogs on 3.13+ (both require numpy<2), deck-standard on 3.14 (ydata-profiling requires <3.14) and dbt on 3.14 (the pinned dbt-core<1.8 breaks on 3.14).
  • ci: skip ydata-profiling in the dev image on Python 3.14 and ci: install pydantic explicitly in the dev image: ydata-profiling cannot be installed on 3.14 (numba/llvmlite have no matching wheels); it was also the only source of pydantic in the integration image, which the pydantic integration workflow needs.

Fixes for breakage on 3.14:

  • fix(omegaconf): read class annotations via inspect.get_annotations: on 3.14 class annotations are evaluated lazily and are no longer in the class __dict__, so every structured config failed with Key ... not found in type annotations.

How was this patch tested?

Full Build workflow in my fork, restricted to Python 3.13 and 3.14.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

@fellhorn
fellhorn marked this pull request as draft October 25, 2026 21:06
@fellhorn fellhorn mentioned this pull request Oct 25, 2026
Merged
2 of 3 tasks
@fellhorn
fellhorn force-pushed the dennis/py314-ci branch 2 times, most recently from 0074747 to 1126a5f Compare October 25, 2026 22:52
@pingsutw

pingsutw commented Nov 3, 2026

Copy link
Copy Markdown
Member

@fellhorn Is it ready for review?

@fellhorn
fellhorn marked this pull request as ready for review November 4, 2026 08:25
bochinski and others added 8 commits November 4, 2026 09:31
@bochinski @claude @fellhorn
Add Python 3.13 support
3013ce7 
- pyproject.toml: requires-python ">=3.10,<3.14", add the 3.13 classifier;
  regenerate uv.lock accordingly.
- PythonVersion enum and default image prefixes: add 3.13 in flytekit
  core, the sqlalchemy plugin and the openai batch plugin. Without it,
  DefaultImages.find_image_for() raises "(3, 13) is not a valid
  PythonVersion" on 3.13.
- CI (pythonbuild.yml): run 3.13 on pull requests and in the nightly
  matrix, enable the extras (tensorflow/pytorch) tests on 3.13, exclude
  pandas<2 and plugins whose dependencies do not support 3.13 yet.
- dev-requirements.in: tensorflow>=2.21 and torch>=2.6 on 3.13; lift the
  protobuf<5 dev pin on 3.13 only (tensorflow 2.21 needs protobuf>=6, and
  the issue behind the pin, flyteorg/flyte#5448, was fixed upstream).
- Publish py3.13 images in pythonpublish.yml and build_image.yml.
- pytorch extras: torch>=2.6 defaults torch.load(weights_only=True), which
  cannot load the whole nn.Module / checkpoint objects flytekit stores.
  Pass weights_only=False where the keyword exists. This is required for
  the extras tests on 3.13 and already fails in the nightly 3.11 run.

Closes flyteorg/flyte#6993

Rebased onto flyteorg#3477: its pytorch weights_only handling replaces the one from this commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
(cherry picked from commit 605e0b1)
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@bochinski @claude @fellhorn
Run the extras tests on Python 3.12
f392ef6 
The tensorflow and pytorch transformer tests were skipped on 3.12 because
neither library had 3.12 wheels when 3.12 support was added. Both upstream
issues have been closed since: tensorflow ships 3.12 wheels from 2.16
(tensorflow/tensorflow#62003) and pytorch from 2.2
(pytorch/pytorch#110436).

- dev-requirements.in: install tensorflow on >=3.12 and torch on >=3.11
  with a single line each instead of per-version lines. With the
  protobuf<5 pin that is kept on <3.13, 3.12 resolves to tensorflow 2.19
  and 3.13 to the latest release.
- pythonbuild.yml: drop the 3.12 skip of the extras step.

Effect on the resolved dev environments: 3.10, 3.11 and 3.13 are
unchanged. On 3.12 every job that installs dev-requirements.in now also
gets tensorflow 2.19 and torch 2.14 (including the CUDA wheels torch pulls
in on linux), and numpy is capped to 2.1.x by tensorflow 2.19.

Verified locally on 3.12 with tensorflow 2.19.1 / torch 2.14.0 /
protobuf 4.25.9: unit_test_extras 123 passed, unit_test 1970 + 25 passed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
(cherry picked from commit 51d5fa8)
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@bochinski @claude @fellhorn
CI: run only Python 3.13 on pull requests, fix 3.13 comments
42e4c65 
- Pull requests run a single interpreter version again (the convention
  since flyteorg#3380), now 3.13 instead of 3.12. The nightly matrix still covers
  3.10 to 3.13. This halves the pull request matrix and keeps the heavier
  3.12 dev environment (tensorflow/torch) out of pull request runs.
- Reword the comment on the 3.13 plugin exclusions: they mirror the 3.12
  list but have not been verified on 3.13, several of the underlying
  issues are closed by now.
- The tensorflow releases with 3.13 wheels require protobuf>=5, not >=6
  (2.20 needs 5.28, 2.21 needs 6.31).
- dev-requirements.in: give types-protobuf the same version split as
  protobuf, so the stubs match the runtime on 3.13.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: bochinski <30434375+bochinski@users.noreply.github.com>
(cherry picked from commit 750bea8)
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@fellhorn @claude
ci: test and publish Python 3.14
8694ddf 
- Nightly matrix covers 3.10-3.14, pull requests run 3.14 only.
- Exclude pandas<2 and the 3.13 plugin exclusions on 3.14, plus
  flytekit-mlflow (its tests need tensorflow, which has no 3.14 wheels).
- Drop the 3.13 flytekit-mlflow exclusion: it was only needed because of
  the protobuf<5 pin, which is gone.
- Build and publish py3.14 images.
- dev-requirements.in: install tensorflow only below 3.14 and skip the
  tensorflow extras tests on 3.14.
- Add the 3.14 classifier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@fellhorn @claude
ci: skip ydata-profiling in the dev image on Python 3.14
5a2197e 
ydata-profiling declares requires-python <3.14 and caps numba<0.63, which
has no 3.14 wheels; uv ignores the upper bound and fails building
llvmlite from source. The integration tests do not use it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@fellhorn @claude
fix(omegaconf): read class annotations via inspect.get_annotations
c430025 
On Python 3.14 class annotations are evaluated lazily and no longer live in
the class __dict__, so all_annotations() returned an empty ChainMap and every
structured config failed with "Key ... not found in type annotations".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@fellhorn @claude
ci: adjust plugin matrix for Python 3.13 and 3.14
c233565 
- Do not force numpy<2 for the onnx, sqlalchemy and pandera plugins on
  3.13+: the pandas/pyarrow releases with 3.13+ wheels require numpy>=2.
- Exclude greatexpectations and whylogs on 3.13+ (both require numpy<2),
  deck-standard on 3.14 (ydata-profiling requires <3.14) and dbt on 3.14
  (the pinned dbt-core<1.8 does not work on 3.14).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@fellhorn @claude
ci: install pydantic explicitly in the dev image
d041e6b 
The pydantic integration workflow only worked because ydata-profiling
pulled pydantic in, which is not installed on Python 3.14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Dennis Keck <26092524+fellhorn@users.noreply.github.com>
@fellhorn

fellhorn commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@fellhorn Is it ready for review?

@pingsutw Thanks for the ping and for merging #3478. Sorry, I missed the notification that the downstack PR got merged. Yes, this PR is now ready for review.

fellhorn
fellhorn commented Nov 4, 2026
View reviewed changes
Comment thread Dockerfile.dev
pygments \
scikit-learn \
ydata-profiling \
"ydata-profiling; python_version<'3.14'" \

@fellhorn fellhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ydata-profiling does not have 3.14 support yet and seems to not be needed for the core tests.

It was added to the image in #2766 for the Flyte Decks / deck-standard plugin, not for the core tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wild-endeavor wild-endeavor Awaiting requested review from wild-endeavor wild-endeavor is a code owner
@kumare3 kumare3 Awaiting requested review from kumare3 kumare3 is a code owner
@pingsutw pingsutw Awaiting requested review from pingsutw pingsutw is a code owner
@cosmicbboy cosmicbboy Awaiting requested review from cosmicbboy cosmicbboy is a code owner
@samhita-alla samhita-alla Awaiting requested review from samhita-alla samhita-alla is a code owner
@davidmirror-ops davidmirror-ops Awaiting requested review from davidmirror-ops davidmirror-ops is a code owner
@machichima machichima Awaiting requested review from machichima machichima is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Python 3.13 support to Flytekit

3 participants

@fellhorn @pingsutw @bochinski