Skip to content

Specification conformance, integration tests that skip only what needs a key - #239

Merged
jwrosewell merged 6 commits into
rcfrom
remote/python-integration-skip
Nov 21, 2026
Merged

jwrosewell merged 6 commits into
rcfrom
remote/python-integration-skip

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The problem

On rc, each product has a ci/run-integration-tests.ps1, for Device Detection, IP Intelligence, location, and pipeline. If no Resource Key is passed, the script throws before it runs anything, with "No TestResourceKey was passed, so the cloud tests cannot call the cloud service." So a build without the key runs none of the integration tests, including the many that need no key at all.

The specification

No sentence in the specifications says what an integration step does without a key. The nearest are these, at 51Degrees/specifications d79bbe3b.

pipeline-specification/automated-testing.md lines 8 to 13:

The tests provided are almost all unit tests, though some unit tests also serve as integration tests - for example, requiring access to live cloud server instances to pass - however they are not distinguished as such in continuous integration.

device-detection-specification/required-examples.md lines 20 to 22:

Those tests MUST include a run of each example with no arguments or settings, as someone would run it from a fresh clone, so that the run uses the free data files.

This change marks the tests that need the live cloud service and a Resource Key as live, so they can be told apart. With no key, it runs every other test. It keeps the rule from #58 that a cloud test fails when it finds no key, so a key that's passed but never reaches a test is still caught.

The change

  • c6d02d4 and 6468491, the test. ci/run-integration-tests.Tests.ps1 runs each product's script the way common-ci does, through Invoke-ProductStep in ci/dispatch.ps1 and common-ci's own steps/run-script.ps1, from a folder laid out like a common-ci checkout. The only stand-in is common-ci's python/run-integration-tests.ps1, which runs no tests and records the packages and the PYTEST_ADDOPTS each call gets. It has eight cases.
    • With no key, it checks for all four products that tests run, that only the live tests are left out, and that the step says so for each product. It also checks that the Device Detection and IP Intelligence examples run, as do the Cloud Request Engine and its examples. The 51Did live tests and the Selenium contract against the cloud example are skipped with a message, and nothing is cloned for the contract.
    • With no key and options already in PYTEST_ADDOPTS, the selection is added to them and the caller's value is put back afterwards. 6468491 added this case.
    • With a key, nothing is left out.
  • 892c433 and d038213, the scripts.
    • The four product scripts no longer throw. With no key each prints a ::warning saying the tests that need a Resource Key are skipped, adds -m "not live" to whatever PYTEST_ADDOPTS holds for its run, and puts the old value back in finally. live is the marker that device-detection/ci/run-unit-tests.ps1 and pipeline/ci/run-unit-tests.ps1 already leave out with the same -m "not live".
    • Pipeline runs the Cloud Request Engine and its examples first. Then, only with a key, it runs 51Did on its own with -m "live and not tier", because the two selections can't share one run. The first run's .coverage is renamed .coverage.cloudrequestengine so the second run's combine reads it. With no key, 51Did has nothing to run, so it and its OWID copy are skipped with a message.
    • IP Intelligence reports the key's products only when there is a key.
    • Device Detection runs the Selenium contract against the cloud example only with a key, and against the on-premise example only with the TAC data file. It clones nothing when it can run neither.
  • 892c433 and 4c6f1e9, the markers. The live marker comes from pytest_collection_modifyitems hooks in five conftest.py files that aren't on main, so no released test file changes. Each hook runs first (tryfirst), so -m sees the marker.
    • device-detection/fiftyone_devicedetection_examples/conftest.py, new at the package root, marks tests/test_cloudexamples.py and tests/test_cloudgettingstartedweb.py.
    • ip-intelligence/fiftyone_ipintelligence_examples/conftest.py, new at the package root, marks the five test_cloud_*_console tests in tests/test_examples.py.
    • location/tests/conftest.py marks test_examples.py and test_properties51D.py.
    • pipeline/fiftyone_pipeline_cloudrequestengine/tests/conftest.py marks the CloudEngineTests class, test_HttpDataSetInException included, since that class needs a Resource Key in setUp.
    • pipeline/fiftyone_pipeline_cloudrequestengine/examples/tests/conftest.py marks the resource key case of helpers.KEYS. The License Key case still skips itself when no License Key is set.
  • Two modules build a cloud Pipeline when they're imported, location/tests/test_properties51D.py and the Device Detection examples' test_cloudgettingstartedweb.py. With no key that raises during collection. A pytest_ignore_collect hook leaves them out when the -m expression leaves out live tests. Every test in them is live, so the selection doesn't change. To read the -m expression the hook uses pytest's own parser, _pytest.mark.expression.Expression, which is private. In a plain run with no -m they're collected as on rc.
  • Four test files that aren't on main carry the marker themselves. They're test_cloudfailuretomatch.py and test_clouduseragentclienthintsweb.py in the Device Detection examples, test_cloudgettingstartedweb.py in the IP Intelligence examples, and location's test_web_example.py.
  • Each changed package's tox.ini passes PYTEST_ADDOPTS through, and two readme files say how to run the tests that need no key.
  • 2ec7915, the head, merges rc f9013a0. Three tox.ini files had conflicts, and the merge keeps both sides. location/tox.ini and pipeline/fiftyone_pipeline_cloudrequestengine/tox.ini pass _51DEGREES_RESOURCE_KEY, _51DEGREES_LICENSE_KEY, and PYTEST_ADDOPTS. The Cloud Request Engine examples' tox.ini passes the five _51DEGREES_ names it documents and PYTEST_ADDOPTS. None of the three passes resource_key or license_key now, as on rc.

Released tests

Against rc f9013a0, the draft changes 21 files. None of them is a file under a tests/ folder or a conftest.py that's on main 612aa28. The 11 files on main that the draft does change are the four ci/run-integration-tests.ps1 scripts, five tox.ini files, and two readme.md files. With a key the scripts set no -m (apart from the 51Did run, as on rc), so the hooks leave nothing out and every released test runs as before. The full build of d038213 below, which had a key, shows that.

The runs

These ran on the head, 2ec7915. Each was read from the API by the head's SHA, and each concluded with success.

Run 44692862618, "Temporary build of 2ec7915", is a full Build and Test of the head, also a success. It built eda393b, whose only parent is 2ec7915 and which adds nothing but its own workflow file. All 10 Build and Test jobs passed, on Ubuntu and Windows with Python 3.10 to 3.14. Nobody has read its logs yet, so whether it had a Resource Key, and its test counts, aren't known.

None of the full builds here had a macOS job.

Earlier runs, of d038213, the commit before the merge:

  • CI script tests run 44454410973 passed on ubuntu-latest and windows-latest.
  • CI script tests run 44009105422 is the red for the first test commit. It was started by hand on a temporary branch holding rc 2373d14 plus only the test file from c6d02d4. On both runners all six no-key cases failed with RuntimeException: run-integration-tests failed for: device-detection, ip-intelligence, location, pipeline, because on rc each script throws before calling the stand-in. The case with a key passed.
  • Run 44463424224 is a temporary full Build and Test of d038213 with a Resource Key, on a branch holding d038213 plus one workflow file. It passed in all 10 jobs. In the Ubuntu, Python 3.12 job the integration step gave these results.
    • Cloud Request Engine: 149 passed, 2 skipped. Its examples: 61 passed, 13 skipped.
    • 51Did: 6 passed.
    • Device Detection cloud package and examples: 29 and 73 passed. Each Selenium contract: 9 of 9.
    • Location: 18 passed. IP Intelligence examples: 56 passed.
    • Nothing was deselected outside the 51Did run.
  • CI script tests run 44481008127 is the red for the case 6468491 added. It was started by hand on a temporary branch holding 6468491 and one workflow commit, e62427c, that changes nothing but its own file. On ubuntu-latest and windows-latest that case failed, "adds the selection to the options, and puts them back", expecting -p no:cacheprovider -m "not live". The other 151 passed.

Run 44463403072 built 6468491 through Build and Test and passed. That says nothing about the new case, because Build and Test doesn't run the Pester tests. Only the CI script tests workflow does, and it starts on a pull request, a push to main or rc, or by hand. That run also had a Resource Key, so every script took the path with a key, and d038213 only changes the lines that run without one. Locally (Windows, Pester 6.2.0, common-ci main df25db41) the test file at 6468491 gives 7 passed and 1 failed. The new case expected -p no:cacheprovider -m "not live" and got -m "not live". At d038213 all eight pass.

Local runs with no key

Windows, Python 3.12.10, pytest 9.2.0, no key variables set, PYTEST_ADDOPTS='-m "not live"' as the scripts set it, at d038213.

Folder Result
Cloud Request Engine tests 145 passed, 2 skipped, 4 deselected
Cloud Request Engine examples tests 48 passed, 13 skipped, 13 deselected
location tests 13 passed, 3 deselected, test_properties51D.py not collected

The on-premise packages need a native build, which these runs didn't do.

What remains

  1. No CI run has yet run the real integration step with no key. The no-key path is shown only by the Pester test, with common-ci's test runner replaced by a stand-in, and by the local runs above.
  2. Tests left out with -m "not live" appear only as pytest's "deselected" count and in the warning, and not one by one in the published results.
  3. With no key and the TAC file present, the Selenium contract still runs against the on-premise example with PAID_RESOURCE_KEY empty. Whether that suite needs the key for the on-premise example isn't established.
  4. That no other test module builds a cloud Pipeline when it's imported was checked by reading the tests. No runner has confirmed it with no key.
  5. The hooks use collection_path and item.path, which arrived in pytest 7, while the five tox.ini files still allow pytest>=6. tox installs the newest pytest, so CI isn't affected, but the floor should be pytest>=7. If a later pytest changes the private Expression class, the hook fails at collection rather than picking the wrong tests.
  6. The new conftest.py comments say the marker is applied "so the released test files stay as they are". Saying "so the test files need no marker of their own" would make sense to someone reading the file cold.
  7. No macOS job has run for any head of this draft.
  8. The branch is now 3 commits behind rc b64b480. GitHub reports it mergeable with no conflict.

Before you upgrade

Nothing changes for callers of the packages, because no package code changes. For CI, a build with no Resource Key now runs every integration test that needs no key and warns that the others are skipped. Before, it failed at once.

One release note. location/ci/run-integration-tests.ps1 sits inside the location path of fiftyone_location in ci/packages.json, whose ignore list leaves out only **/tox.ini. So this change alone would make the release plan release fiftyone_location. rc already has other changes under location/ since its last release, 4.5.10, so the next release includes it either way. Adding ci/** to that ignore list would stop a later CI-only change releasing it.

…eed none

ci/run-integration-tests.ps1 stops before running anything when no
TestResourceKey is passed, for every product, because each product's own
script throws "No TestResourceKey was passed". Two macOS runs stopped there,
so none of the integration tests that need no key ran on them.

ci/run-integration-tests.Tests.ps1 runs the step as common-ci does, through
ci/dispatch.ps1 and common-ci's steps/run-script.ps1, for Device Detection,
IP Intelligence, location and pipeline, with a stand-in for common-ci's
python/run-integration-tests.ps1 that records what each product asked it to
test. With no TestResourceKey it expects:

- every product to run its tests, the step not failing;
- each run to leave out only the tests marked live, which need a resource
  key, through PYTEST_ADDOPTS;
- each product to say that the tests needing a resource key are skipped;
- the Device Detection cloud and examples packages and the IP Intelligence
  examples to be run;
- pipeline to run the cloud request engine and its examples, and to skip
  the 51Did package, whose integration run has only live tests, saying so;
- the Selenium contract against the cloud example to be skipped, saying so,
  with nothing cloned for it.

With a TestResourceKey, Device Detection and location run every test, the
live ones included. CI-1.
…ed none

Each product's ci/run-integration-tests.ps1 now runs when no
TestResourceKey is passed. It prints a warning that the tests needing a
resource key are skipped, sets PYTEST_ADDOPTS to -m "not live" so they are
left out, and runs every other integration test. With a key nothing is left
out, and a live test that finds no key still fails, as before.

- Device Detection, IP Intelligence and location set the selection around
  their one run. IP Intelligence reports what the key carries only when
  there is a key.
- Pipeline runs the cloud request engine and its examples first, then the
  51Did package on its own with its "live and not tier" selection, because
  both runs now see PYTEST_ADDOPTS. The first run's coverage is kept under
  a name the second run's combine reads. With no key the 51Did package has
  no test to run, so it is skipped, saying so, and the OWID source is not
  copied.
- Device Detection skips the Selenium contract against the cloud example
  when there is no key, saying so, and clones and sets up nothing when
  neither example can be run.
- The tox.ini of the Device Detection and IP Intelligence examples, location,
  the cloud request engine and its examples pass PYTEST_ADDOPTS.
- The tests that need a resource key are marked live: the Device Detection
  cloud example tests, the IP Intelligence cloud console and web tests,
  location's example, property and web example tests, the cloud request
  engine's CloudEngineTests and the pipeline examples' resource key cases.
  Each folder registers the marker in its conftest.py, and the four copies
  of the on-premise conftest.py stay the same.
- Two test modules built a cloud pipeline when they were collected, which
  raises with no key, so collecting them failed even when they were left
  out. The Device Detection cloud getting started web test now builds it in
  setUpClass and fails with the no key message, as the other web tests do,
  and location's property tests do the same.
- location's cloud example test passed without a key. It now fails, naming
  the variables, like the other live tests.
- The cloud request engine's test of the error a wrong key gets needs no
  key, so it moves out of CloudEngineTests into a class of its own.
- The licence key tests already skip themselves without a licence key, and
  are unchanged.

The readmes of the Device Detection and pipeline examples say how to run the
tests that need no key. CI-1.
@jwrosewell jwrosewell added the remote-conformance Brings the code into conformance with the specification label Nov 16, 2026
Eddie51D and others added 4 commits November 19, 2026 00:45
…d tests as they are

The tests that call the live cloud service are marked live by a
pytest_collection_modifyitems hook in a conftest.py that is not on main.
Every released test file is back to its content on rc.

A module that builds a cloud Pipeline when it is imported raises with no
resource key, so it is not collected when the live tests are left out.
These are location/tests/test_properties51D.py and the device detection
examples' tests/test_cloudgettingstartedweb.py. Every test in them is live,
so the selection is the same.

test_HttpDataSetInException stays in CloudEngineTests, whose setUp needs a
resource key, so it is live with the rest of that class.
…n to PYTEST_ADDOPTS

The four products' ci/run-integration-tests.ps1 replaced any options a
caller had set. They now add the selection to them, and put them back
after the run as before.
rc stopped passing the deprecated environment variable names in the
tox.ini files. The three files where this branch adds PYTEST_ADDOPTS
keep PYTEST_ADDOPTS and its comment, and drop the old names as rc does.
@jwrosewell
jwrosewell marked this pull request as ready for review November 21, 2026 16:32
@jwrosewell
jwrosewell merged commit 1952ff5 into rc Nov 21, 2026
5 checks passed
@jwrosewell
jwrosewell deleted the remote/python-integration-skip branch November 21, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Brings the code into conformance with the specification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants