Skip to content

Test and document the on-premise packages on free-threaded Python - #243

Merged
jwrosewell merged 14 commits into
rcfrom
remote/free-threaded-python
Nov 21, 2026
Merged

jwrosewell merged 14 commits into
rcfrom
remote/free-threaded-python

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem

James Rosewell asked for free-threaded Python to be documented and tested. Neither on-premise package says what happens on a free-threaded build (Python 3.13t or 3.14t), and nothing tests one.

Both SWIG wrappers carry the declaration that a module runs without the GIL, but only when SWIGPYTHON_NOGIL is defined, which swig -nogil does and these wrappers were not generated with. It is at hash_python_wrap.cxx lines 32974 to 32980 and ipi_python_wrap.cxx lines 49165 to 49171, with the matching PyUnstable_Module_SetGIL call at lines 33067 to 33071 and 49258 to 49262. Neither wrapper defines SWIG_PYTHON_THREADS, and neither setup.py defines either name. So on a free-threaded build Python turns the GIL back on when it loads each native module.

What was confirmed

On Windows, with free-threaded Python 3.14.10 and both native modules built for it from this checkout's cxx at 178371ac with MSVC, importing each module in a new interpreter gave the following.

Module sys._is_gil_enabled() before After Warning
fiftyone_devicedetection_onpremise.DeviceDetectionHashEngineModule False True RuntimeWarning, below
fiftyone_ipintelligence_onpremise.IpIntelligenceEngineModule False True RuntimeWarning, below
<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'fiftyone_devicedetection_onpremise._DeviceDetectionHashEngineModule', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
<frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'fiftyone_ipintelligence_onpremise._IpIntelligenceEngineModule', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

With PYTHON_GIL=1, or with -X gil=1, the GIL was on before and after the import of each module and there was no warning. With PYTHON_GIL=0 it stayed off after the import, with no warning, which is the state the new tests refuse.

The rule

The thread safety page of the specification, in specifications#190, says at its head a90ba516, lines 71 to 73 that

Where the data is held in unmanaged memory, as an on-premise Engine's results instance is, multi-threaded access is not supported, for performance. That is a hard rule in every language.

What changed

This draft is on top of #241, because the docstring paragraph it adds follows the paragraph #241 adds to the same SwigData docstrings, and a paragraph added on rc there would conflict with #241. It is opened against #241's branch and retargets to rc once #241 merges.

Three commits.

  1. 56b501e1 adds the workflow and the tests.
  2. 424eda0e adds the documentation.
  3. 759cf0db merges Results instances are not thread-safe, so reads take no lock #241's newest head, which uses the specification's words for results instances, and keeps both docstring paragraphs.

The changes, file by file:

  • A new workflow, .github/workflows/free-threaded-python.yml, runs on a pull request or a push to rc or main that changes either on-premise package, the pipeline packages they use, cxx or the workflow, and on demand. On free-threaded Python 3.13 and 3.14 on Ubuntu and 3.14 on Windows, it checks the build is free-threaded with the GIL off at start, fetches the free Lite data files with the data repository's own script, builds both native modules, and runs both packages' own tests with -rs, so a skipped test shows.
  • tests/test_free_threaded.py in each on-premise package imports the native module in a new interpreter, with PYTHON_GIL removed from its environment, and asserts that sys._is_gil_enabled() was False before the import and is True after it, and that the import gave a RuntimeWarning naming the native module. So a change that lets the module load with the GIL still off fails. The tests skip on any other build of Python, so the existing builds are unchanged apart from two skipped tests in each package.
  • Each on-premise package's readme has a new "Free-threaded Python" section, which says the package runs on the free-threaded builds of Python 3.13 and 3.14, gives the warning in full, and says the package gains nothing from free-threading because the native code runs with the GIL held, as on the standard build. It says to set PYTHON_GIL=1 or run with -X gil=1 to start with the GIL on and so without the warning, that the package has not been tested with the GIL kept off by PYTHON_GIL=0, and that the results of one request are still not safe to use from several threads at once.
  • Each SwigData class docstring has a paragraph saying the same, after the paragraph from Results instances are not thread-safe, so reads take no lock #241.

How it was verified

Only the two packages this changes were tested locally, at 759cf0d, on Windows with free-threaded Python 3.14.10, both native modules built for it, and the free Lite data files (51Degrees-LiteV4.1.hash, 51Degrees-LiteV41.ipi and the free ASN file). No licence key, resource key, data file variable or PYTHON_GIL was set.

  • Device Detection on-premise, pytest -rs tests, 105 passed and 14 subtests passed, none skipped.
  • IP Intelligence on-premise, pytest -rs tests, 80 passed and 13 subtests passed, none skipped.
  • On standard Python 3.14.5, test_free_threaded.py skips both of its tests in each package.

The new workflow passed on this head, 759cf0d, in run 44154527035, on free-threaded Python 3.13.18 and 3.14.10 on Ubuntu and 3.14.10 on Windows, as it did on 424eda0 in run 44051575693. Each of the three jobs gave 105 passed for Device Detection and 80 passed for IP Intelligence, none skipped. The quick checks passed too, being UTM link lint (run 44154526939), environment variable names (run 44154526850) and only rc into main (run 44154527497).

This draft changes no test that shipped in the last release, main at 612aa28, and adds only a workflow, two test files, readme sections and docstring paragraphs to what #241 has. The released tests of the two on-premise packages therefore give the results #241 records for them, which are the same as on rc.

This test guards behaviour the packages already have, so it has no red on rc. To show it would fail, a throwaway branch at 41d3324 changed the GIL assertion in both packages to expect the GIL still off after the import, and ran this workflow on a push to that branch, in run 44051719311. All three jobs failed, each package with AssertionError: 'False False' != 'False True' at tests/test_free_threaded.py line 71, giving 1 failed and 104 passed for Device Detection and 1 failed and 79 passed for IP Intelligence. The throwaway branch ran this workflow itself, as the common-ci build runs on standard Python, where these tests skip. The branch is deleted.

No full build was started for this draft.

What remains

  • Results instances are not thread-safe, so reads take no lock #241 has to merge first, and this draft then retargets to rc.
  • The packages do not declare free-threading support, so they gain nothing from it. Declaring it would mean generating the wrappers with swig -nogil and showing the native code and the wrappers are safe without the GIL, which this draft does not attempt.

Before you upgrade

  1. What changed. Nothing in either package's behaviour. This draft documents and tests what already happens on a free-threaded build of Python 3.13 or 3.14, where loading either native module turns the GIL back on and Python prints a RuntimeWarning, such as RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'fiftyone_devicedetection_onpremise._DeviceDetectionHashEngineModule', which has not declared that it can run safely without the GIL., followed by Python's advice on overriding it.
  2. Why. specifications#190 at a90ba516, lines 71 to 73 says "Where the data is held in unmanaged memory, as an on-premise Engine's results instance is, multi-threaded access is not supported, for performance. That is a hard rule in every language."
  3. What to change. Nothing is required. To start with the GIL on, and so without the warning, set the environment variable PYTHON_GIL=1 or run Python with -X gil=1. The packages have not been tested with the GIL kept off by PYTHON_GIL=0.

jwrosewell and others added 12 commits November 17, 2026 08:16
SwigData in the Device Detection and IP Intelligence on-premise packages
no longer holds a lock for each read of its native results or for
close(). A results instance is not thread-safe, the caller is responsible
for thread-safe access, and the class docstrings say so, with what to do
instead. specifications#190 adds this as the "Results instances" section
of the thread safety page.

The thread tests asserted that reads never overlap and that close() waits
for a read, which is no longer promised. They now check that the
docstring says the caller coordinates threads and that an instance holds
no lock. Closing results and reading afterwards is still tested with real
data by test_flowdata_close_releases_results in both packages.
The ElementData docstring says a finished instance is safe to read from
several threads at once, as its values are Python's own managed data and
do not change once written. It now also says that Element Data reading
from a results instance, which holds unmanaged memory, is the exception.
Multi-threaded access to a results instance is not supported, for
performance reasons, so the caller gives each thread its own Flow Data or
coordinates its own access. The documentation test checks both, as lines
3 to 16 of the thread safety page in specifications#190 frame it.
Both SwigData docstrings now say a results instance holds unmanaged
memory, and that multi-threaded access to it is not supported, for
performance reasons, as lines 3 to 16 of the thread safety page in
specifications#190 frame it. The caller either gives each thread its own
Flow Data or coordinates its own access. That wording replaces the
quotation of the rule against locking. The documentation tests check the
new wording.
A new workflow builds the Device Detection and IP Intelligence on-premise
packages on free-threaded Python 3.13 and 3.14 on Ubuntu, and 3.14 on
Windows, and runs their tests with the free Lite data files.

The SWIG wrappers declare that the native modules run without the GIL
only when SWIGPYTHON_NOGIL is defined, and it is not, so Python turns the
GIL back on when it loads them. tests/test_free_threaded.py in each
package checks, in a new interpreter, that the GIL is off before the
native module loads, on after, and that Python warns about it. The tests
skip on every other build of Python.
Each on-premise readme and SwigData docstring says that free-threaded
Python is supported, that loading the native module turns the GIL back
on with a RuntimeWarning, and that the package therefore gains nothing
from free-threading. The readmes give the warning in full, and both say
how to start with the GIL on and so without the warning, which is
PYTHON_GIL=1 or -X gil=1.
The ElementData docstring now says an instance is written only by the Flow
Element that provides it, or the Flow Elements that share it, and only
while they process, and that the Flow Data holding it is used by one
thread at a time. Element Data that reads from a results instance takes no
lock to make those reads safe.

Both SwigData docstrings now follow the results instance section of the
thread safety page in specifications#190 at a90ba516. Reading a value
writes into the results instance, and the engine, its results instances
and the instance take no lock to make those reads safe for several
threads. The documentation tests check the new words.
Both SwigData docstrings keep the free-threaded paragraph after the results instance paragraph, which now uses the words of specifications#190 at a90ba516.
Each job of free-threaded-python.yml downloads the free Lite data files
from the Distributor on every run. The new Pester test reads the workflow
and checks that one job, outside the matrix, downloads them only when the
cache misses, and that every job that runs the tests restores them from
that cache with the same key and paths, on Linux and Windows alike. It
fails on the workflow as it stands. CI script tests now also run when the
workflow changes.
The test started each step with an empty list and then added lines only
when that list was not empty, which is never, so it found no step at all.
It now checks for no list rather than an empty one. On the workflow as it
stands it still fails six of its seven tests, now for the reasons it was
written for.
Each of the three jobs downloaded the free Lite data files from the
Distributor on every run. A first job now downloads them only when the
cache has no copy for the current commit of the data submodule, which
links them, and the cache keeps them. The test jobs need that job and
restore the files from the cache, on Linux and Windows alike, failing on
a miss rather than downloading. The workflow keeps no secrets, so a pull
request from a fork runs it as before.
The readmes and SwigData docstrings said the packages run on the
free-threaded builds of Python 3.13 and 3.14, and that Python turns the
GIL back on when it loads the native module. They now name the builds the
workflow tests (3.13 and 3.14 on Linux, 3.14 on Windows), say that the
packages are published as source and so need a compiler for that Python,
and say that the GIL comes back on for the whole process, so the rest of
the program runs with it too.
The tests ran the import in a new interpreter with check=True, so a
failed import raised CalledProcessError with only its exit status. They
now assert that the exit status is 0 with the new interpreter's error
output as the message, so a missing module or a crash on import shows its
cause.
Base automatically changed from remote/results-not-thread-safe to rc November 21, 2026 16:43
rc now holds the final results instance wording, the close paragraph in
the Element Data docstring and the thread test docstrings, so the merge
takes rc's side in elementdata.py and both test_swig_data_threads.py. In
both swig_data.py it keeps rc's SwigData docstring and adds the
free-threaded paragraph after it. Against rc the branch now changes only
the free-threaded workflow, its Pester test, the CI script tests
trigger, the two readmes, the two free-threaded tests and that
paragraph.
The packages here now require each other at a minimum version, and one
installed from the checkout is 0.0.0 without a version.txt, so pip
refused the editable installs and the tests then failed on missing
modules. The test job now fetches the release tags, checks out
common-ci, and runs ci/write-package-versions.ps1 before it installs,
as the main CI does through ci/setup-environment.ps1. A pip install
that fails now stops the step.

ci/free-threaded-python.Tests.ps1 checks all three.
@jwrosewell
jwrosewell marked this pull request as ready for review November 21, 2026 19:18
@jwrosewell
jwrosewell merged commit 497078c into rc Nov 21, 2026
9 checks passed
@jwrosewell
jwrosewell deleted the remote/free-threaded-python branch November 21, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants