Repository navigation
Test and document the on-premise packages on free-threaded Python - #243
Merged
Merged
Conversation
SwigData in the Device Detection and IP Intelligence on-premise packages no longer holds a lock for each read of its native results or for close(). A results instance is not thread-safe, the caller is responsible for thread-safe access, and the class docstrings say so, with what to do instead. specifications#190 adds this as the "Results instances" section of the thread safety page. The thread tests asserted that reads never overlap and that close() waits for a read, which is no longer promised. They now check that the docstring says the caller coordinates threads and that an instance holds no lock. Closing results and reading afterwards is still tested with real data by test_flowdata_close_releases_results in both packages.
The ElementData docstring says a finished instance is safe to read from several threads at once, as its values are Python's own managed data and do not change once written. It now also says that Element Data reading from a results instance, which holds unmanaged memory, is the exception. Multi-threaded access to a results instance is not supported, for performance reasons, so the caller gives each thread its own Flow Data or coordinates its own access. The documentation test checks both, as lines 3 to 16 of the thread safety page in specifications#190 frame it.
Both SwigData docstrings now say a results instance holds unmanaged memory, and that multi-threaded access to it is not supported, for performance reasons, as lines 3 to 16 of the thread safety page in specifications#190 frame it. The caller either gives each thread its own Flow Data or coordinates its own access. That wording replaces the quotation of the rule against locking. The documentation tests check the new wording.
A new workflow builds the Device Detection and IP Intelligence on-premise packages on free-threaded Python 3.13 and 3.14 on Ubuntu, and 3.14 on Windows, and runs their tests with the free Lite data files. The SWIG wrappers declare that the native modules run without the GIL only when SWIGPYTHON_NOGIL is defined, and it is not, so Python turns the GIL back on when it loads them. tests/test_free_threaded.py in each package checks, in a new interpreter, that the GIL is off before the native module loads, on after, and that Python warns about it. The tests skip on every other build of Python.
Each on-premise readme and SwigData docstring says that free-threaded Python is supported, that loading the native module turns the GIL back on with a RuntimeWarning, and that the package therefore gains nothing from free-threading. The readmes give the warning in full, and both say how to start with the GIL on and so without the warning, which is PYTHON_GIL=1 or -X gil=1.
The ElementData docstring now says an instance is written only by the Flow Element that provides it, or the Flow Elements that share it, and only while they process, and that the Flow Data holding it is used by one thread at a time. Element Data that reads from a results instance takes no lock to make those reads safe. Both SwigData docstrings now follow the results instance section of the thread safety page in specifications#190 at a90ba516. Reading a value writes into the results instance, and the engine, its results instances and the instance take no lock to make those reads safe for several threads. The documentation tests check the new words.
Both SwigData docstrings keep the free-threaded paragraph after the results instance paragraph, which now uses the words of specifications#190 at a90ba516.
Each job of free-threaded-python.yml downloads the free Lite data files from the Distributor on every run. The new Pester test reads the workflow and checks that one job, outside the matrix, downloads them only when the cache misses, and that every job that runs the tests restores them from that cache with the same key and paths, on Linux and Windows alike. It fails on the workflow as it stands. CI script tests now also run when the workflow changes.
The test started each step with an empty list and then added lines only when that list was not empty, which is never, so it found no step at all. It now checks for no list rather than an empty one. On the workflow as it stands it still fails six of its seven tests, now for the reasons it was written for.
Each of the three jobs downloaded the free Lite data files from the Distributor on every run. A first job now downloads them only when the cache has no copy for the current commit of the data submodule, which links them, and the cache keeps them. The test jobs need that job and restore the files from the cache, on Linux and Windows alike, failing on a miss rather than downloading. The workflow keeps no secrets, so a pull request from a fork runs it as before.
The readmes and SwigData docstrings said the packages run on the free-threaded builds of Python 3.13 and 3.14, and that Python turns the GIL back on when it loads the native module. They now name the builds the workflow tests (3.13 and 3.14 on Linux, 3.14 on Windows), say that the packages are published as source and so need a compiler for that Python, and say that the GIL comes back on for the whole process, so the rest of the program runs with it too.
The tests ran the import in a new interpreter with check=True, so a failed import raised CalledProcessError with only its exit status. They now assert that the exit status is 0 with the new interpreter's error output as the message, so a missing module or a crash on import shows its cause.
rc now holds the final results instance wording, the close paragraph in the Element Data docstring and the thread test docstrings, so the merge takes rc's side in elementdata.py and both test_swig_data_threads.py. In both swig_data.py it keeps rc's SwigData docstring and adds the free-threaded paragraph after it. Against rc the branch now changes only the free-threaded workflow, its Pester test, the CI script tests trigger, the two readmes, the two free-threaded tests and that paragraph.
The packages here now require each other at a minimum version, and one installed from the checkout is 0.0.0 without a version.txt, so pip refused the editable installs and the tests then failed on missing modules. The test job now fetches the release tags, checks out common-ci, and runs ci/write-package-versions.ps1 before it installs, as the main CI does through ci/setup-environment.ps1. A pip install that fails now stops the step. ci/free-threaded-python.Tests.ps1 checks all three.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
James Rosewell asked for free-threaded Python to be documented and tested. Neither on-premise package says what happens on a free-threaded build (Python 3.13t or 3.14t), and nothing tests one.
Both SWIG wrappers carry the declaration that a module runs without the GIL, but only when
SWIGPYTHON_NOGILis defined, whichswig -nogildoes and these wrappers were not generated with. It is athash_python_wrap.cxxlines 32974 to 32980 andipi_python_wrap.cxxlines 49165 to 49171, with the matchingPyUnstable_Module_SetGILcall at lines 33067 to 33071 and 49258 to 49262. Neither wrapper definesSWIG_PYTHON_THREADS, and neithersetup.pydefines either name. So on a free-threaded build Python turns the GIL back on when it loads each native module.What was confirmed
On Windows, with free-threaded Python 3.14.10 and both native modules built for it from this checkout's
cxxat 178371ac with MSVC, importing each module in a new interpreter gave the following.sys._is_gil_enabled()beforefiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModulefiftyone_ipintelligence_onpremise.IpIntelligenceEngineModuleWith
PYTHON_GIL=1, or with-X gil=1, the GIL was on before and after the import of each module and there was no warning. WithPYTHON_GIL=0it stayed off after the import, with no warning, which is the state the new tests refuse.The rule
The thread safety page of the specification, in specifications#190, says at its head a90ba516, lines 71 to 73 that
What changed
This draft is on top of #241, because the docstring paragraph it adds follows the paragraph #241 adds to the same
SwigDatadocstrings, and a paragraph added onrcthere would conflict with #241. It is opened against #241's branch and retargets torconce #241 merges.Three commits.
The changes, file by file:
.github/workflows/free-threaded-python.yml, runs on a pull request or a push torcormainthat changes either on-premise package, the pipeline packages they use,cxxor the workflow, and on demand. On free-threaded Python 3.13 and 3.14 on Ubuntu and 3.14 on Windows, it checks the build is free-threaded with the GIL off at start, fetches the free Lite data files with the data repository's own script, builds both native modules, and runs both packages' own tests with-rs, so a skipped test shows.tests/test_free_threaded.pyin each on-premise package imports the native module in a new interpreter, withPYTHON_GILremoved from its environment, and asserts thatsys._is_gil_enabled()was False before the import and is True after it, and that the import gave a RuntimeWarning naming the native module. So a change that lets the module load with the GIL still off fails. The tests skip on any other build of Python, so the existing builds are unchanged apart from two skipped tests in each package.PYTHON_GIL=1or run with-X gil=1to start with the GIL on and so without the warning, that the package has not been tested with the GIL kept off byPYTHON_GIL=0, and that the results of one request are still not safe to use from several threads at once.SwigDataclass docstring has a paragraph saying the same, after the paragraph from Results instances are not thread-safe, so reads take no lock #241.How it was verified
Only the two packages this changes were tested locally, at 759cf0d, on Windows with free-threaded Python 3.14.10, both native modules built for it, and the free Lite data files (
51Degrees-LiteV4.1.hash,51Degrees-LiteV41.ipiand the free ASN file). No licence key, resource key, data file variable orPYTHON_GILwas set.pytest -rs tests, 105 passed and 14 subtests passed, none skipped.pytest -rs tests, 80 passed and 13 subtests passed, none skipped.test_free_threaded.pyskips both of its tests in each package.The new workflow passed on this head, 759cf0d, in run 44154527035, on free-threaded Python 3.13.18 and 3.14.10 on Ubuntu and 3.14.10 on Windows, as it did on 424eda0 in run 44051575693. Each of the three jobs gave 105 passed for Device Detection and 80 passed for IP Intelligence, none skipped. The quick checks passed too, being UTM link lint (run 44154526939), environment variable names (run 44154526850) and only rc into main (run 44154527497).
This draft changes no test that shipped in the last release,
mainat 612aa28, and adds only a workflow, two test files, readme sections and docstring paragraphs to what #241 has. The released tests of the two on-premise packages therefore give the results #241 records for them, which are the same as onrc.This test guards behaviour the packages already have, so it has no red on
rc. To show it would fail, a throwaway branch at 41d3324 changed the GIL assertion in both packages to expect the GIL still off after the import, and ran this workflow on a push to that branch, in run 44051719311. All three jobs failed, each package withAssertionError: 'False False' != 'False True'attests/test_free_threaded.pyline 71, giving 1 failed and 104 passed for Device Detection and 1 failed and 79 passed for IP Intelligence. The throwaway branch ran this workflow itself, as the common-ci build runs on standard Python, where these tests skip. The branch is deleted.No full build was started for this draft.
What remains
rc.swig -nogiland showing the native code and the wrappers are safe without the GIL, which this draft does not attempt.Before you upgrade
RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'fiftyone_devicedetection_onpremise._DeviceDetectionHashEngineModule', which has not declared that it can run safely without the GIL., followed by Python's advice on overriding it.PYTHON_GIL=1or run Python with-X gil=1. The packages have not been tested with the GIL kept off byPYTHON_GIL=0.