Repository navigation
Specification conformance, integration tests that skip only what needs a key - #239
Merged
Merged
Conversation
…eed none ci/run-integration-tests.ps1 stops before running anything when no TestResourceKey is passed, for every product, because each product's own script throws "No TestResourceKey was passed". Two macOS runs stopped there, so none of the integration tests that need no key ran on them. ci/run-integration-tests.Tests.ps1 runs the step as common-ci does, through ci/dispatch.ps1 and common-ci's steps/run-script.ps1, for Device Detection, IP Intelligence, location and pipeline, with a stand-in for common-ci's python/run-integration-tests.ps1 that records what each product asked it to test. With no TestResourceKey it expects: - every product to run its tests, the step not failing; - each run to leave out only the tests marked live, which need a resource key, through PYTEST_ADDOPTS; - each product to say that the tests needing a resource key are skipped; - the Device Detection cloud and examples packages and the IP Intelligence examples to be run; - pipeline to run the cloud request engine and its examples, and to skip the 51Did package, whose integration run has only live tests, saying so; - the Selenium contract against the cloud example to be skipped, saying so, with nothing cloned for it. With a TestResourceKey, Device Detection and location run every test, the live ones included. CI-1.
…ed none Each product's ci/run-integration-tests.ps1 now runs when no TestResourceKey is passed. It prints a warning that the tests needing a resource key are skipped, sets PYTEST_ADDOPTS to -m "not live" so they are left out, and runs every other integration test. With a key nothing is left out, and a live test that finds no key still fails, as before. - Device Detection, IP Intelligence and location set the selection around their one run. IP Intelligence reports what the key carries only when there is a key. - Pipeline runs the cloud request engine and its examples first, then the 51Did package on its own with its "live and not tier" selection, because both runs now see PYTEST_ADDOPTS. The first run's coverage is kept under a name the second run's combine reads. With no key the 51Did package has no test to run, so it is skipped, saying so, and the OWID source is not copied. - Device Detection skips the Selenium contract against the cloud example when there is no key, saying so, and clones and sets up nothing when neither example can be run. - The tox.ini of the Device Detection and IP Intelligence examples, location, the cloud request engine and its examples pass PYTEST_ADDOPTS. - The tests that need a resource key are marked live: the Device Detection cloud example tests, the IP Intelligence cloud console and web tests, location's example, property and web example tests, the cloud request engine's CloudEngineTests and the pipeline examples' resource key cases. Each folder registers the marker in its conftest.py, and the four copies of the on-premise conftest.py stay the same. - Two test modules built a cloud pipeline when they were collected, which raises with no key, so collecting them failed even when they were left out. The Device Detection cloud getting started web test now builds it in setUpClass and fails with the no key message, as the other web tests do, and location's property tests do the same. - location's cloud example test passed without a key. It now fails, naming the variables, like the other live tests. - The cloud request engine's test of the error a wrong key gets needs no key, so it moves out of CloudEngineTests into a class of its own. - The licence key tests already skip themselves without a licence key, and are unchanged. The readmes of the Device Detection and pipeline examples say how to run the tests that need no key. CI-1.
…d tests as they are The tests that call the live cloud service are marked live by a pytest_collection_modifyitems hook in a conftest.py that is not on main. Every released test file is back to its content on rc. A module that builds a cloud Pipeline when it is imported raises with no resource key, so it is not collected when the live tests are left out. These are location/tests/test_properties51D.py and the device detection examples' tests/test_cloudgettingstartedweb.py. Every test in them is live, so the selection is the same. test_HttpDataSetInException stays in CloudEngineTests, whose setUp needs a resource key, so it is live with the rest of that class.
…aller set in PYTEST_ADDOPTS
…n to PYTEST_ADDOPTS The four products' ci/run-integration-tests.ps1 replaced any options a caller had set. They now add the selection to them, and put them back after the run as before.
rc stopped passing the deprecated environment variable names in the tox.ini files. The three files where this branch adds PYTEST_ADDOPTS keep PYTEST_ADDOPTS and its comment, and drop the old names as rc does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
On
rc, each product has aci/run-integration-tests.ps1, for Device Detection, IP Intelligence, location, and pipeline. If no Resource Key is passed, the script throws before it runs anything, with "No TestResourceKey was passed, so the cloud tests cannot call the cloud service." So a build without the key runs none of the integration tests, including the many that need no key at all.The specification
No sentence in the specifications says what an integration step does without a key. The nearest are these, at 51Degrees/specifications d79bbe3b.
pipeline-specification/automated-testing.md lines 8 to 13:
device-detection-specification/required-examples.md lines 20 to 22:
This change marks the tests that need the live cloud service and a Resource Key as
live, so they can be told apart. With no key, it runs every other test. It keeps the rule from #58 that a cloud test fails when it finds no key, so a key that's passed but never reaches a test is still caught.The change
ci/run-integration-tests.Tests.ps1runs each product's script the way common-ci does, throughInvoke-ProductStepinci/dispatch.ps1and common-ci's ownsteps/run-script.ps1, from a folder laid out like a common-ci checkout. The only stand-in is common-ci'spython/run-integration-tests.ps1, which runs no tests and records the packages and thePYTEST_ADDOPTSeach call gets. It has eight cases.livetests are left out, and that the step says so for each product. It also checks that the Device Detection and IP Intelligence examples run, as do the Cloud Request Engine and its examples. The 51Did live tests and the Selenium contract against the cloud example are skipped with a message, and nothing is cloned for the contract.PYTEST_ADDOPTS, the selection is added to them and the caller's value is put back afterwards. 6468491 added this case.::warningsaying the tests that need a Resource Key are skipped, adds-m "not live"to whateverPYTEST_ADDOPTSholds for its run, and puts the old value back infinally.liveis the marker thatdevice-detection/ci/run-unit-tests.ps1andpipeline/ci/run-unit-tests.ps1already leave out with the same-m "not live".-m "live and not tier", because the two selections can't share one run. The first run's.coverageis renamed.coverage.cloudrequestengineso the second run's combine reads it. With no key, 51Did has nothing to run, so it and its OWID copy are skipped with a message.livemarker comes frompytest_collection_modifyitemshooks in fiveconftest.pyfiles that aren't onmain, so no released test file changes. Each hook runs first (tryfirst), so-msees the marker.device-detection/fiftyone_devicedetection_examples/conftest.py, new at the package root, markstests/test_cloudexamples.pyandtests/test_cloudgettingstartedweb.py.ip-intelligence/fiftyone_ipintelligence_examples/conftest.py, new at the package root, marks the fivetest_cloud_*_consoletests intests/test_examples.py.location/tests/conftest.pymarkstest_examples.pyandtest_properties51D.py.pipeline/fiftyone_pipeline_cloudrequestengine/tests/conftest.pymarks theCloudEngineTestsclass,test_HttpDataSetInExceptionincluded, since that class needs a Resource Key insetUp.pipeline/fiftyone_pipeline_cloudrequestengine/examples/tests/conftest.pymarks theresource keycase ofhelpers.KEYS. The License Key case still skips itself when no License Key is set.location/tests/test_properties51D.pyand the Device Detection examples'test_cloudgettingstartedweb.py. With no key that raises during collection. Apytest_ignore_collecthook leaves them out when the-mexpression leaves outlivetests. Every test in them is live, so the selection doesn't change. To read the-mexpression the hook uses pytest's own parser,_pytest.mark.expression.Expression, which is private. In a plain run with no-mthey're collected as onrc.maincarry the marker themselves. They'retest_cloudfailuretomatch.pyandtest_clouduseragentclienthintsweb.pyin the Device Detection examples,test_cloudgettingstartedweb.pyin the IP Intelligence examples, and location'stest_web_example.py.tox.inipassesPYTEST_ADDOPTSthrough, and two readme files say how to run the tests that need no key.rcf9013a0. Threetox.inifiles had conflicts, and the merge keeps both sides.location/tox.iniandpipeline/fiftyone_pipeline_cloudrequestengine/tox.inipass_51DEGREES_RESOURCE_KEY,_51DEGREES_LICENSE_KEY, andPYTEST_ADDOPTS. The Cloud Request Engine examples'tox.inipasses the five_51DEGREES_names it documents andPYTEST_ADDOPTS. None of the three passesresource_keyorlicense_keynow, as onrc.Released tests
Against
rcf9013a0, the draft changes 21 files. None of them is a file under atests/folder or aconftest.pythat's onmain612aa28. The 11 files onmainthat the draft does change are the fourci/run-integration-tests.ps1scripts, fivetox.inifiles, and tworeadme.mdfiles. With a key the scripts set no-m(apart from the 51Did run, as onrc), so the hooks leave nothing out and every released test runs as before. The full build of d038213 below, which had a key, shows that.The runs
These ran on the head, 2ec7915. Each was read from the API by the head's SHA, and each concluded with success.
ubuntu-latestandwindows-latest. It runs everyci/*.Tests.ps1, so all eight cases above passed at the head.Run 44692862618, "Temporary build of 2ec7915", is a full Build and Test of the head, also a success. It built eda393b, whose only parent is 2ec7915 and which adds nothing but its own workflow file. All 10 Build and Test jobs passed, on Ubuntu and Windows with Python 3.10 to 3.14. Nobody has read its logs yet, so whether it had a Resource Key, and its test counts, aren't known.
None of the full builds here had a macOS job.
Earlier runs, of d038213, the commit before the merge:
ubuntu-latestandwindows-latest.rc2373d14 plus only the test file from c6d02d4. On both runners all six no-key cases failed withRuntimeException: run-integration-tests failed for: device-detection, ip-intelligence, location, pipeline, because onrceach script throws before calling the stand-in. The case with a key passed.ubuntu-latestandwindows-latestthat case failed, "adds the selection to the options, and puts them back", expecting-p no:cacheprovider -m "not live". The other 151 passed.Run 44463403072 built 6468491 through Build and Test and passed. That says nothing about the new case, because Build and Test doesn't run the Pester tests. Only the CI script tests workflow does, and it starts on a pull request, a push to
mainorrc, or by hand. That run also had a Resource Key, so every script took the path with a key, and d038213 only changes the lines that run without one. Locally (Windows, Pester 6.2.0, common-cimaindf25db41) the test file at 6468491 gives 7 passed and 1 failed. The new case expected-p no:cacheprovider -m "not live"and got-m "not live". At d038213 all eight pass.Local runs with no key
Windows, Python 3.12.10, pytest 9.2.0, no key variables set,
PYTEST_ADDOPTS='-m "not live"'as the scripts set it, at d038213.test_properties51D.pynot collectedThe on-premise packages need a native build, which these runs didn't do.
What remains
-m "not live"appear only as pytest's "deselected" count and in the warning, and not one by one in the published results.PAID_RESOURCE_KEYempty. Whether that suite needs the key for the on-premise example isn't established.collection_pathanditem.path, which arrived in pytest 7, while the fivetox.inifiles still allowpytest>=6. tox installs the newest pytest, so CI isn't affected, but the floor should bepytest>=7. If a later pytest changes the privateExpressionclass, the hook fails at collection rather than picking the wrong tests.conftest.pycomments say the marker is applied "so the released test files stay as they are". Saying "so the test files need no marker of their own" would make sense to someone reading the file cold.rcb64b480. GitHub reports it mergeable with no conflict.Before you upgrade
Nothing changes for callers of the packages, because no package code changes. For CI, a build with no Resource Key now runs every integration test that needs no key and warns that the others are skipped. Before, it failed at once.
One release note.
location/ci/run-integration-tests.ps1sits inside thelocationpath offiftyone_locationinci/packages.json, whose ignore list leaves out only**/tox.ini. So this change alone would make the release plan releasefiftyone_location.rcalready has other changes underlocation/since its last release, 4.5.10, so the next release includes it either way. Addingci/**to that ignore list would stop a later CI-only change releasing it.