Skip to content

Give the released missing property messages again, and share a parameter only when it starts with 51D_ or the Pipeline uses it - #246

Merged
ben51degrees merged 7 commits into
rcfrom
remote/released-tests-to-the-musts
Nov 19, 2026
Merged

ben51degrees merged 7 commits into
rcfrom
remote/released-tests-to-the-musts

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Some tests from the last release fail when they're run unchanged against rc. Each of those failures falls into one of three kinds, and the kind decides what this draft does about it.

  1. If a test fails only because a key, a module or a built package is missing, that isn't a behavior change, and the draft leaves it alone.
  2. If rc changed the behavior because a specification sentence says MUST, or describes the new behavior directly, the new behavior stays. rc's own test already checks it, and the change is listed under Before you upgrade.
  3. If the change rests on a SHOULD, or on no sentence at all, the draft goes back to what the release did. The test from the release passes again with no edits, and rc's copy of that test is restored to match.

That leaves two items in this draft.

  1. Missing property messages. The messages from #221 are the third kind, so the messages from the release come back. The MUST that a missing property raises an error stays, which means the on-premise Device Detection and IP Intelligence services now raise where the release returned nothing (Before you upgrade item 13).
  2. Query string and form parameters. #218 changed what a parameter list of None means. That no longer matters, because this draft adds a new usage sharing rule for parameters, and under that rule the list has no effect (Before you upgrade item 5).

The rule for query string and form parameters

specifications#191 has merged into the specification's rc, and every specification link here points at rc's commit d79bbe3b. The rule itself is in usage-sharing-element.md lines 72 to 78.

Query string and form parameters follow the same rule as cookies, with one addition. It MUST also return false, whatever the configuration and even when all Evidence is shared, for any key whose prefix is query., unless its field starts with 51D_ or the key is Evidence the Pipeline uses. A key is Evidence the Pipeline uses when a Flow Element in the Pipeline accepts it, as that Flow Element's list of keys or its filter function says.

Lines 100 to 104 cover the options for included query string parameters and for sharing all of them. Both "are deprecated and have no effect", and "setting it MUST NOT change which parameters are shared." Lines 81 to 91 apply the query. prefix to a GET query string and a POST form body alike. They also say a parameter no Flow Element in the Pipeline accepts, such as password, is never shared.

Problem

The tests folder of each package from the last release, main 612aa28, ran unchanged against rc 2373d14 (still the merge base) with no keys. Failures by package were core 1, engines 31, engines_fiftyone 8, cloud request engine 7, 51Did 8 plus 5 files not collected, cloud Device Detection 9, on-premise Device Detection 6, Device Detection examples 6 plus 1 not collected, on-premise IP Intelligence 1, IP Intelligence translation 1, IP Intelligence examples 4, and location 1 file not collected. Add a resource key, put cryptography and the OWID source on the path for 51Did the way CI installs them, and the only failures left are the behavior changes in the second table.

Released tests that fail only for their environment

Package Released tests Why they fail here Checked
fiftyone_pipeline_cloudrequestengine test_cloudengine.py CloudEngineTests test_HttpDataSetInException, test_cloud_engine, test_cloud_get_request_with_sequence_evidence, test_cloud_post_request_with_sequence_evidence No resource key Pass with one
fiftyone_devicedetection_cloud test_cloud.py test_basic_get_cloud, test_cloud_request_origin, test_engine_init_performance, test_missing_property_service_element_not_found, test_pipeline_builder_cloud_engine_init, test_properties_cloud No resource key Pass with one
fiftyone_devicedetection_cloud test_properties.py test_available_properties, test_value_types No resource key, then no tests/51Degrees.csv With a key, both skip locally because the CSV isn't there. CI fetches it, and both passed in the green run at the head
fiftyone_devicedetection_examples test_cloudexamples.py test_cloud_configurator_console, test_cloud_failuretomatch, test_cloud_getting_started_console, test_cloud_metadata_console, test_cloud_nativemodellookup_console, test_cloud_taclookup_console, and test_cloudgettingstartedweb.py, not collected No resource key Pass with one
fiftyone_ipintelligence_examples test_examples.py test_cloud_getallproperties_console, test_cloud_gettingstarted_console, test_cloud_metadata_console, test_cloud_mixed_console No resource key Pass with one
location test_properties51D.py, not collected No resource key Passes with one
fiftyone_pipeline_did test_creator_context_server.py, test_did_client.py, test_did_client_live.py, test_did_tier_live.py, test_fodid.py, not collected cryptography and owid weren't installed With both on the path and the keys the live tests read, all pass except the three in the next table. rc's own tests fail to collect the same way without them
fiftyone_pipeline_did test_live_cloud.py CopiesTests.test_every_copy_is_the_same_file, one failure for each of 8 folders It compares the release's copy of live_cloud.py with rc's copies in the other test folders The copies differ only in the configuration page link that #107 changed, so no package behavior is involved
fiftyone_ipintelligence_translation test_translation.py test_translations_are_packaged It looks in the folder of a built package, and the run used the source rc's copy, which is the same, fails the same way from the source

Released tests that fail because rc changed the behavior

Released tests What changed rc pull request Specification sentence MUST or not This draft
core test_setheader.py test_get_response_header_name_1_SetHeaderbrowserAccept_CH A Set Header name whose identifier starts with a lower-case letter gives its header name, where it raised #140 "[Identifier] is some string that relates to what this value is for. It MUST NOT have upper-case characters after the first character." set-headers-element.md lines 12 to 17 MUST Kept, as rc's test already checks the new behavior. Item 1
engines test_datafile_update_http.py UpdateFromMemoryTests testMemoryDataFileIsRefreshedFromTheBytes, testFileDataFileIsWrittenThenRefreshed, testFailedRefreshIsReportedAndLogged, and CallbackTests testACallbackErrorOnAnUpdateFromMemoryIsLoggedAndStopsTheCallbacks An update from memory reports nothing to the update complete callbacks, and a failure raises to the caller unlogged #169 "Two calls report nothing to onUpdateCallbacks, and give their result to the caller instead. ... an update from memory returns success or fails with an error." data-updates.md lines 379 to 383, and "A failure with a buffer passed in by a programmatic update from memory fails that call with an error, which is not logged or reported to onUpdateCallbacks" lines 321 to 323 Plain statement rc's tests already check this, so they stay. Item 2
engines UpdateFromMemoryTests.testAFailedWriteFromMemoryEndsTheUpdate, FileReplacementTests testFailedCopyNamesThePath, testNoTemporaryPathIsReported Data from memory that can't be written to the data file is a warning, and the engine is still refreshed from it #175 The errors table row "Writing data passed in from memory to the data file failed", at Warning, "None, the Engine is still refreshed" line 518 Plain statement No change, since rc's tests already cover it. Item 2
engines LoggingTests testAnyOtherStatusNamesTheCode, testConnectionFailureIsError, testFailedRefreshIsError, testIntegrityCheckFailureIsError, testRefusedDownloadIsError, testTheErrorEntryIsWrittenExactly, testTooManyRequestsIsError, DownloadedFileTests testAFailedTemporaryCopyIsAnyOtherError, testAFailedWriteEndsTheUpdateAndLeavesTheEngineAlone, CallbackTests testACallbackErrorOnAProgrammaticCheckIsLoggedAndStopsTheCallbacks, testAConnectionFailureKeepsPolling, LicenceKeyTests testEachFailedAnswerShowsOnlyTheMaskedKey, testFailedConnectionShowsOnlyTheMaskedKey, ContentTests testCorruptCompressedContentIsAnErrorAndCheckedAgainLater, testFailedMd5CheckOnStartUpIsNotRepeatedAtOnce, and test_licence_key_masking.py UpdateServiceLogTests test_a_callback_error_logs_only_the_masked_key, test_a_failed_request_logs_only_the_masked_key, test_no_message_holds_the_key, test_the_masked_url_is_in_the_message A programmatic check, update_data_file, raises its failure to the caller after reporting it, and logs nothing. Each test called it and read the log #188 "on fail, report the failure to onUpdateCallbacks, then pass the error to the caller without logging it" lines 356 and 357, "A programmatic check, the update on start-up and an update from memory pass the error to their caller instead of logging it." lines 484 to 486, and of a callback's error "On a programmatic check and the update on start-up it passes to the caller." lines 499 and 500 Plain statement Kept. rc's tests read the log from a polled check instead, which still logs. Item 3
engines test_datafile_update_service.py FileSystemWatcherTests testFileSystemWatcherFailedRefreshUsesTheTableMessage, testFileSystemWatcherKeepsWatchingAfterFailedRefresh, and the text of LoggingTests.testFailedRefreshIsError A refresh from the data file is tried 10 times, so the message says "after 10 tries" #220 "These messages MUST also be consistent across languages ... They are the messages and levels of the .NET data update service" lines 488 to 490, with the row "Refreshing the Engine from the data file still failed after 10 tries" line 516 MUST rc's tests expect "after 10 tries", and the draft leaves them. Item 4
engines_fiftyone test_usage.py test_default_shares_only_accepted_evidence and test_share_usage_accepts_no_evidence.py test_the_web_request_is_shared_by_the_same_rules A query string or form parameter is shared only when its name starts 51D_ or the Pipeline uses it, where #201 shared every one by default This draft The MUST of lines 72 to 78, quoted above MUST Both pass again as the release has them, and rc's copy of the first is restored from the release
engines_fiftyone test_usage.py test_share_all (its cookie part), test_share_usage_evidencekeyfilter.py test_share_all, and test_share_usage_accepts_no_evidence.py test_the_method_is_shared_and_the_path_and_cookie_header_are_not A cookie whose name doesn't start 51D_ is never shared, even when all evidence is #201 "It MUST also return false, whatever the configuration and even when all Evidence is shared, for any key whose prefix is cookie. and whose field does not start with 51D_." lines 66 to 70 MUST Kept, with rc's tests checking that such a cookie isn't shared. Item 6
engines_fiftyone test_usage.py test_empty_query_whitelist, test_parameter_names_with_dots and the parameter part of test_share_all, and test_share_usage_evidencekeyfilter.py test_field_is_everything_after_the_first_dot, test_lists_ignore_case, test_naming_a_parameter_does_not_share_the_query_string, test_never_shared_keys_match_whole_parts, test_no_parameter_list_shares_every_parameter and the parameter part of test_share_all query_whitelist and share_all_query_string_parameters no longer share a parameter, and query_whitelist=None no longer shares every one This draft, replacing what #218 did "setting it MUST NOT change which parameters are shared" lines 100 to 104 MUST rc's copies now follow the rule, and test_no_parameter_list_shares_every_parameter is renamed test_no_parameter_list_shares_only_51d_parameters. Item 5
cloud request engine test_credentials.py test_licence_key_discovers_properties_with_the_key_in_the_body The engine asks for the evidence keys when it's built, so the last request is that one rather than the properties request the test reads #147 "The Engine MUST attempt both discovery requests when it is built (in its builder or constructor)" cloud-request-engine.md lines 245 to 248 MUST No change. rc's test picks out the properties request by its address. Item 7
cloud request engine test_cloudresponse.py test_validate_error_handling_invalid_resourceKey One entry of the cloud service's errors list is the message, without the list's brackets and quotes #204 "If the top-level errors Property in the JSON response contains an entry, then it will need to be parsed and the text used as the message for the thrown error." lines 480 to 482. The SHOULD of line 478 is about whether to throw, which both releases do Plain statement Kept. rc's expected message is already the new one. Item 8
on-premise Device Detection test_devicedetection.py test_validate_data_true, test_validate_data_false The match metric Properties have the category "Device Metrics", where they had "Device metrics", so the tests' check of the category no longer skips them #145 "Metadata for these Properties MUST be added ... Category = "Device Metrics" (sic)" device-detection-on-premise.md lines 452 to 460 MUST rc's tests already check the new behavior, and the draft doesn't touch them. Item 9
on-premise Device Detection test_update_matched_useragent.py test_turned_off_leaves_user_agents_empty UserAgents is [] when update_matched_useragent is off, where it was [""] #168 UserAgents, default [], "Populated when the Update Matched User Agent configuration option is on" line 473 Plain statement Kept. rc's test expects []. Item 10
on-premise IP Intelligence test_ipintelligence.py test_missing_licence_key An engine builds with no licence key when nothing downloads, where it refused #136 "A License Key is needed only for the two mechanisms that download, which are polling and the update on start-up. The file system watcher needs no License Key." data-updates.md lines 114 to 116 Plain statement rc replaced the test with test_licence_key_rule.py, which stays. Item 11
51Did test_did_client.py RedeemTests test_a_factor_that_was_not_compared_is_still_none, test_redeemed_with_factors, test_to_dict_is_the_cloud_shape A null factor in a redeem answer is FactorResult.MISMATCH, where it was None #141 "A factor word a package does not know MUST be reported as a Mismatch. ... any other word is a Mismatch, so an unexpected value never reads as a factor that matched or as one that could not be compared." package-surface.md lines 117 to 121 MUST No change. rc's tests expect FactorResult.MISMATCH. Item 12
engines test_engines.py testRestrictedProperty, testRestrictedPropertyIgnoresCase, testMissingPropertyService, on-premise Device Detection test_devicedetection.py test_excluded_property, test_missing_property_service_not_found_anywhere, test_missing_property_service_not_found_in_current, cloud request engine test_credentials.py test_missing_property_message_names_both_causes, and, with a key, cloud Device Detection test_cloud.py test_missing_property_service_not_found_anywhere A missing property raises the missing properties table's message, with the reason decided in the table's order #221 "An implementation SHOULD give the same text" properties.md lines 133 to 137, and "An implementation SHOULD decide the reason in this order" lines 143 and 144 SHOULD The release's messages come back, and so do its versions of these tests

The change

  1. Missing property messages, 9c1b7fd. The missing property services of fiftyone_pipeline_engines, fiftyone_pipeline_cloudrequestengine, fiftyone_devicedetection_onpremise and fiftyone_ipintelligence_onpremise, and AspectData.get, go back to the release's code, so each raises the message it raised in the release. HardwareProfileData in fiftyone_devicedetection_cloud came in on rc and isn't on main, so it now uses the missing property service that cloud data had in the release, from clouddata.py. What stays from Give a missing property's reason in the specification's order #221 is the MUST that "the system MUST throw an error/exception with a message explaining why the Property is not available" (properties.md lines 116 to 118). In the release, the on-premise services returned nothing for a property the engine lists in a data file tier that holds it. They now raise for a property that isn't available, as rc does.
  2. A parameter list of None, c781d95, made query_whitelist=None mean what it meant in the release. After item 5 the list has no effect at all, so c781d95 no longer changes anything.
  3. Tests, f03482c, committed first and on their own. The eight missing property tests and the filter test are restored from the release. test_missing_property_reasons.py and test_cloudengine_missing_property.py are deleted, because every check in them is a message from the table, and the code they tested goes too. New tests cover the MUST that stays. test_missing_property_raises.py in both on-premise packages checks that a listed property of the data file's tier raises. It passes on rc and fails against the services as released, which return nothing. test_the_deprecated_option_changes_nothing checks that the option, given as False or True, changes nothing.
  4. Parameter tests, 28fcaba, also first and on their own. With parameters in a GET query string and in a POST form body, test_share_usage_rules.py checks that a 51D_ parameter and one a Flow Element of the Pipeline uses are shared. It also checks that password and other names never are, under any setting of share_all, query_whitelist or share_all_query_string_parameters, whether set in code or in a configuration file. The filter, list and usage tests that relied on those options now follow the rule, and rc's copy of test_default_shares_only_accepted_evidence is the release's version again.
  5. The parameter rule, 715e618. ShareUsageEvidenceKeyFilter now decides a query. key before it looks at share_all, whatever the configuration says. It shares the key only when the name starts 51D_ or a Flow Element of the Pipeline accepts it, and ShareUsage hands the filter the Flow Data's Pipeline so it can ask. The two options stay as deprecated names that do nothing. The JavaScript Builder needs care here. Its filter passes every query string parameter so it can send them back to its callback, but its specification lists only query.fod-js-object-name and query.fod-js-enable-cookies as accepted Evidence (javascript-builder.md lines 16 to 21). Usage sharing asks the JavaScript Builder about those two only, because otherwise every parameter of a web Pipeline, password included, would count as used.
  6. Test names, dc6e95e. test_no_parameter_list_shares_every_parameter now checks that a list of None, or an empty one, shares only 51D_ parameters, so it's renamed test_no_parameter_list_shares_only_51d_parameters. test_the_option_given_decides_whatever_the_list, which this draft added, becomes test_the_deprecated_option_changes_nothing, since that's what it checks.
  7. The deprecated option's default, 1504f3e. ShareUsageEvidenceKeyFilter defaults share_all_query_string_parameters to False again, as on rc, in place of the None that c781d95 gave it. ShareUsage always passes its own value, so what's shared doesn't change.

Red and green

Each run built the named commit with one temporary workflow commit on top, on Ubuntu and Windows with Python 3.10 to 3.14, which is ten jobs.

  • The missing property messages went red in run 44179246532, which built f03482c, the tests alone on rc 2373d14. Every job failed on the same seven restored tests and nothing else. Three are in test_engines.py ("3 failed, 117 passed"), three in test_devicedetection.py ("3 failed, 101 passed"), and the seventh is test_no_parameter_list_shares_every_parameter ("1 failed, 121 passed"). A unit stage failure ends the job, and the restored test_credentials.py test and the cloud test_cloud.py test belong to the integration stage, so they didn't run there. They're red in Local counts instead.
  • The parameter rule went red in run 44219564064, which built 28fcaba, the parameter tests alone on c781d95. Seven jobs reached the tests (Windows with Python 3.10 to 3.14, Ubuntu with 3.10 and 3.14). In each, the only failures were the usage sharing tests the rule changes, "26 failed, 107 passed" in fiftyone_pipeline_engines_fiftyone. The Ubuntu jobs for 3.11, 3.12 and 3.13 stopped before the tests when pip install failed.
  • The head is green. Run 44465779200 built 1504f3e and passed in all ten jobs, through the unit and the integration tests. Its Ubuntu job with Python 3.12 gave fiftyone_pipeline_engines_fiftyone 123 passed, fiftyone_pipeline_engines 120, the Cloud Request Engine 144 passed and 2 skipped, on-premise Device Detection 104 and on-premise IP Intelligence 79.
  • Locally, the new test_missing_property_raises.py in both on-premise packages fails with "Exception not raised" against each package's swig_data.py from the release, and passes on rc and on the head.

dc6e95e and 1504f3e don't add a test that fails first, since neither changes what's shared.

Local counts

These runs took each package's tests folder from main 612aa28, with no edits, and ran it against rc 2373d14 and against the draft, comparing failing tests by name. Runs with a key set a resource key, and the 51Did runs also had cryptography and the OWID source on the path. Apart from the filter tests the parameter rule changes (Before you upgrade item 5), nothing that passes on rc fails on the head.

Package Failing on rc, no key Failing on the head, no key Failing on rc, with a key Failing on the head, with a key Passing on the head, failing on rc
fiftyone_pipeline_core 1 1 - - none
fiftyone_pipeline_engines 31 28 - - testMissingPropertyService, testRestrictedProperty, testRestrictedPropertyIgnoresCase
fiftyone_pipeline_engines_fiftyone 8 10 - - test_default_shares_only_accepted_evidence, test_the_web_request_is_shared_by_the_same_rules
fiftyone_pipeline_cloudrequestengine 7 6 3 2 test_missing_property_message_names_both_causes
fiftyone_pipeline_translation 0 0 - - none
fiftyone_pipeline_did 8, and 5 files not collected 8, and 5 files not collected 11 11 none
fiftyone_devicedetection_cloud 9 9 1, and 2 skipped 0, and 2 skipped test_missing_property_service_not_found_anywhere, with a key
fiftyone_devicedetection_onpremise 6 3 - - test_excluded_property, test_missing_property_service_not_found_anywhere, test_missing_property_service_not_found_in_current
fiftyone_devicedetection_examples 6, and 1 file not collected 6, and 1 file not collected 0 0 none
fiftyone_ipintelligence_shared 0 0 - - none
fiftyone_ipintelligence_cloud 0 0 - 0 none
fiftyone_ipintelligence_onpremise 1 1 - - none
fiftyone_ipintelligence_translation 1 1 - - none
fiftyone_ipintelligence_examples 4 4 0 0 none
location 1 file not collected 1 file not collected 0 0 none

What still fails on the head is the environment failures and the MUST changes in the tables above.

Most rows date from a run before dc6e95e and 1504f3e, which only touch fiftyone_pipeline_engines_fiftyone. The four pipeline package rows were run again on Windows with Python 3.12.10 and no key, against rc 2373d14 and 1504f3e. They gave the counts in the table, with the same failing tests by name as before. At 1504f3e, the package's own tests gave 123 passed for fiftyone_pipeline_engines_fiftyone.

What remains

  1. One call is close. test_validate_error_handling_invalid_resourceKey depends on "the text used as the message" in lines 480 to 482. The draft reads that as a plain statement, because the SHOULD of line 478 is about throwing, not about the message. If it's read as part of that SHOULD instead, the release's message, brackets and quotes included, would come back for one entry.
  2. The table's missing property messages and their order can only come back as a change listed under Before you upgrade, because the release's tests match each message it gave.
  3. Usage sharing treats a parameter as used by the Pipeline only if it's among the keys a Flow Element's specification lists as accepted. The one place that matters is the JavaScript Builder, whose filter passes every query string parameter while javascript-builder.md lines 16 to 21 lists two. The merged rule says "as that Flow Element's list of keys or its filter function says", and that wording doesn't choose between the list and the filter. The next step is to ask for a sentence in the specification saying a Flow Element's listed Evidence decides. Until then, the JavaScript Builder's filter stays as it is.
  4. For each query. key, usage sharing asks each Flow Element's evidence key filter in turn. Each Flow Element keeps its filter, so the cost is small and the loop stays as it is.
  5. Two tests from the release still set share_all_query_string_parameters=False, a setting rc added that now does nothing. They're test_the_web_request_is_shared_by_the_same_rules in test_share_usage_accepts_no_evidence.py, whose comment about that option is no longer true, and test_empty_query_whitelist in test_usage.py. The first also passes as the release has it. Putting both back without the option would only change tests.

Before you upgrade

Every item here changes what the last release did, and each one is required by a MUST in the specification or by a sentence that describes the new behavior directly. Items 5 and 13 are new in this draft. The rest came in with the rc pull request named in parentheses, and the draft keeps them.

  1. Set Header names (Add the web elements only when the caller has not, and read every valid Set Header name #140). set-headers-element.md lines 12 to 17 only rule out upper-case characters after the first one. So SetHeaderElement.get_response_header_name now reads a name whose identifier starts with a lower-case letter, such as SetHeaderbrowserAccept-CH, as the header Accept-CH, where it used to raise. Code that relied on the error gets the header name instead.

  2. Updates from memory (Give the result of an update from memory to its caller alone #169 and Refresh the engine from the bytes given when an update from memory cannot write the data file #175). update_from_memory reports nothing to the update complete callbacks, and returns success or raises DataUpdateException without logging it. Data that can't be written to the data file is logged as a warning, and the engine is still refreshed from it. That's what data-updates.md lines 321 to 323, 379 to 383 and 518 ask for. Read the result update_from_memory returns, catch its error, and don't wait for a callback.

  3. Programmatic checks (Raise a failed programmatic update check to its caller #188). Under data-updates.md lines 356 and 357, 484 to 486, and 499 and 500, update_data_file reports a failed check to the update complete callbacks and then raises it to the caller, as DataUpdateException or as the error itself, logging nothing. An error a callback raises during the check reaches the caller too. Where code read the log or relied on the call returning, catch the error.

  4. Refresh tries (Try a refresh 10 times and wait for a watched file, as .NET does #220). A failed refresh from the data file is tried 10 times, 0.2 seconds apart, and the message says "after 10 tries", which is the message data-updates.md lines 488 to 490 and 516 call for. Code that matched "after 1 tries" should match "after 10 tries", and a failed refresh now takes about two seconds longer to report.

  5. Query string and form parameters (this draft). ShareUsage shares a query string or form parameter, from a GET or a POST alike, only when its name starts 51D_ or a Flow Element of the Pipeline accepts it as evidence. That holds whatever the configuration, even with share_all=True. query_whitelist and share_all_query_string_parameters are deprecated and do nothing, and query_whitelist=None no longer shares every parameter. The rule comes from usage-sharing-element.md lines 72 to 78, and lines 100 to 104 deprecate the two options. These tests from the release change:

    • test_usage.py: test_empty_query_whitelist, test_parameter_names_with_dots, and the parameter part of test_share_all, which expected <query Name="campaign">campaign value</query> to be shared.
    • test_share_usage_evidencekeyfilter.py: test_field_is_everything_after_the_first_dot, test_lists_ignore_case, test_naming_a_parameter_does_not_share_the_query_string, test_never_shared_keys_match_whole_parts, the parameter part of test_share_all, and test_no_parameter_list_shares_every_parameter, now test_no_parameter_list_shares_only_51d_parameters.

    If you want a parameter shared, its name has to start with 51D_ or it has to be evidence a Flow Element of the Pipeline accepts. Remove query_whitelist and share_all_query_string_parameters when convenient, since they do nothing.

  6. Cookies never shared (Share every query value by default, and no cookie but 51D_ ones #201). usage-sharing-element.md lines 66 to 70 rule out sharing a cookie whose name doesn't start 51D_, even with share_all=True, and no setting can share one now. The session cookie is still read for track_session.

  7. Discovery when built (Discover the cloud Properties and Evidence keys when the Cloud Request Engine is built #147). Constructing a CloudRequestEngine makes the accessibleProperties and evidencekeys requests, and a key the service refuses raises CloudRequestException from the constructor. cloud-request-engine.md lines 245 to 248 say the engine must attempt both discovery requests when it's built. Expect both requests, and the error, when the engine is built rather than on first use.

  8. Cloud error messages (Raise each entry of the cloud service's errors list as its own error #204). Following cloud-request-engine.md lines 480 to 482, one entry of the cloud service's errors list becomes the message as plain text, without the list's brackets and quotes. Code that matched '[" in the message should match the text alone.

  9. Match metric metadata (Give the match metric Properties the specification's metadata, and list each matched User-Agent once #145). The match metric Properties have the category Device Metrics and the component Metrics, where they had Device metrics and Match Metrics, because device-detection-on-premise.md lines 452 to 465 set their metadata that way. Compare against the new names.

  10. UserAgents when not recorded (Give UserAgents as an empty list when the option is off, and send no usage data from the web example test #168). With update_matched_useragent=False, UserAgents is [], where it was [""]. That's the default device-detection-on-premise.md line 473 gives it. Expect an empty list.

  11. Licence keys (Need a licence key only for the updates that download #136). data-updates.md lines 114 to 116 need a licence key only for the mechanisms that download. An on-premise Device Detection or IP Intelligence engine now builds with no licence key when nothing downloads, where it refused with "licence key is required". One with auto_update or update_on_start on, the Distributor's update URL and no key refuses to build, with a message naming those options. Give a key or turn those options off, and don't expect the refusal when nothing downloads.

  12. 51Did factors (Read a null 51Did factor as a Mismatch, and refuse a short Payload for its version #141). A null factor in a redeem answer is FactorResult.MISMATCH, and to_dict() writes "mismatch" for it, where it was None. package-surface.md lines 117 to 121 ask for a factor word the package doesn't know to be reported as a Mismatch. Code that tested a factor is None should test for FactorResult.MISMATCH.

  13. A listed on-premise property with no value (this draft). On-premise Device Detection and IP Intelligence data now raise for a property the engine lists in a data file tier that holds it, when the data has no value for it, where get returned None. properties.md lines 116 to 118 say a Property that isn't available must raise an error that explains why. Catch the error where code read None.

Restore the released form of the tests whose behaviour rests on a
specification SHOULD, being the missing property messages and a usage
sharing filter given None for its query string parameters:

- test_engines.py testRestrictedProperty, testRestrictedPropertyIgnoresCase
  and testMissingPropertyService
- the on-premise Device Detection test_devicedetection.py
  test_excluded_property, test_missing_property_service_not_found_anywhere
  and test_missing_property_service_not_found_in_current
- test_credentials.py test_missing_property_message_names_both_causes
- the cloud Device Detection test_cloud.py
  test_missing_property_service_not_found_anywhere
- test_share_usage_evidencekeyfilter.py
  test_no_parameter_list_shares_every_parameter

test_missing_property_reasons.py and test_cloudengine_missing_property.py
go, because each of their checks is a message of the missing properties
table, which the released messages replace.

New tests keep what a MUST requires. In both on-premise packages a
property the engine lists, in a data file tier that holds it, raises when
the data has no value for it. And share_all_query_string_parameters given
as False or True decides whatever the parameter list holds.
The properties specification asks that an implementation SHOULD give the
missing properties table's text and SHOULD decide the reason in the
table's order. As both are SHOULDs, each engine raises the message it
raised in the last release again, so the released tests pass unaltered.
That restores the missing property services of the engines, cloud and
on-premise packages and AspectData.get to their released form.

The one change that stays is the specification's MUST to raise for a
missing property. The on-premise Device Detection and IP Intelligence
services raise for a property the engine lists in a data file tier that
holds it, with the table's message for a property that is not available,
where they returned nothing.
… given

ShareUsageEvidenceKeyFilter given query_whitelist=None, with
share_all_query_string_parameters not given, shares every query string
and form parameter, as the released filter did, because no sentence of
the specification says how None is read. The option now defaults to None
for not given. Given as False, the specification's MUST holds, so only
parameters starting 51D_ and those on the list are shared, and given as
True every parameter is. The ShareUsage element always gives the option,
so it shares as before.
jwrosewell and others added 4 commits November 18, 2026 01:56
Query string and form parameters are shared only when their names start
with 51D_ or a Flow Element of the Pipeline accepts them as evidence, from
a GET query string and a POST form body alike, whatever the settings. The
deprecated query_whitelist and share_all_query_string_parameters options
share nothing more, and the JavaScript Builder counts only for the
parameters its specification lists.

The filter, list and usage tests that relied on those options, or on every
parameter being shared, are updated to the rule. The released test
test_default_shares_only_accepted_evidence goes back to its released form.
The usage sharing filter decides a query. key, which carries query string
and form parameters alike, before share_all and whatever the
configuration. It is shared when its name starts with 51D_ or a Flow
Element of the Pipeline accepts it as evidence, so a value such as a
password is never sent. The JavaScript Builder passes every parameter so
that it can send them back to its callback, so it is asked only about the
two parameters its specification lists.

query_whitelist and share_all_query_string_parameters are deprecated and
have no effect, and are still accepted so that existing code and
configuration build.
test_no_parameter_list_shares_every_parameter checks that a list of None,
or an empty one, shares only the parameters starting 51D_. It is now
test_no_parameter_list_shares_only_51d_parameters.

test_the_option_given_decides_whatever_the_list checks that the
deprecated share_all_query_string_parameters option, on or off, changes
nothing that is shared. It is now
test_the_deprecated_option_changes_nothing.

Only the names change, and both tests pass as before.
ShareUsageEvidenceKeyFilter took None for share_all_query_string_parameters
to mean that the option was not given, so that a parameter list of None
could share every parameter. The option now has no effect, so None means
nothing and the default is False again, as the filter had on rc. Which
evidence is shared does not change.
ben51degrees pushed a commit that referenced this pull request Nov 19, 2026
PackageTests and its helpers move from test_share_usage_rules.py to the
new test_share_usage_packages.py, unchanged. test_share_usage_rules.py is
now the same as on rc, so this branch no longer touches the file that
#246 rewrites for the 51D_ and used Evidence rule.
@jwrosewell jwrosewell changed the title Give the released behaviour again where no specification MUST changed it Give the released missing property messages again, and share a parameter only when it starts with 51D_ or the Pipeline uses it Nov 19, 2026
@ben51degrees
ben51degrees marked this pull request as ready for review November 19, 2026 20:15
@ben51degrees
ben51degrees merged commit 74123a9 into rc Nov 19, 2026
3 checks passed
@ben51degrees
ben51degrees deleted the remote/released-tests-to-the-musts branch November 19, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants