Repository navigation
Add the Metrics component and the profiles for a property value, and read Device Detection results by position with no lock - #174
Merged
Merged
Conversation
…device data device-detection-specification/pipeline-elements/device-detection-on-premise.md line 327 at 706de1f0: Aspect Data that reads its values from a ResultsHash instance MUST let only one thread at a time read from its results. Covers a match metric (DeviceId), a typed property read (IsMobile) and the public getValues. DD-18.
device-detection-specification/pipeline-elements/device-detection-on-premise.md line 327 at 706de1f0: Aspect Data that reads its values from a ResultsHash instance MUST let only one thread at a time read from its results. The lock in DeviceDataBaseOnPremise.tryGetValue covered only the typed getters. The check that the results have a property, the public getValues, the match metrics (which can walk every graph not yet walked) and close read the results with no lock. DeviceDataHashDefault now holds one lock of its own for each of them. DD-18.
…ves it device-detection-specification/pipeline-elements/device-detection-on-premise.md line 452 at 706de1f0: Metadata for these Properties MUST be added, as they will not be included in the metadata exposed by the native code. Checks the order, Category, Available With, Component, type and default of each metric, over a native engine that is a mock with no properties. The default of the list-valued UserAgents is left out, as how it is described is an open question. DD-20.
…e value device-detection-specification/pipeline-elements/device-detection-on-premise.md line 452 at 706de1f0: Metadata for these Properties MUST be added, as they will not be included in the metadata exposed by the native code. The table gives Component = "Metrics" for every match metric, where each had no component, and gives Method the type string, where it was marked as a list. The engine now makes a Metrics component once, holding the seven metrics, and each metric names it. The default of UserAgents is unchanged while how a list-valued metric is described is an open question. DD-20.
device-detection-specification/pipeline-elements/device-detection-on-premise.md line 464 at 706de1f0: Component = "Metrics" - This component MUST also be added to the list of components returned by the Engine. Over a native engine that is a mock with no components. DD-21.
device-detection-specification/pipeline-elements/device-detection-on-premise.md
line 464 at 706de1f0: Component = "Metrics" - This component MUST also be
added to the list of components returned by the Engine.
getComponents gave only the data file's components. It now gives the Metrics
component after them, so getComponent("Metrics") finds it too. The metadata
hash test no longer leaves out the metrics' component, which they now have,
and skips the default profile of Metrics, which has no profiles. DD-21.
… property device-detection-specification/pipeline-elements/device-detection-on-premise.md line 479 at 706de1f0: The Engine MUST be able to give the ids of the profiles that hold a given value of a given Property, such as a TAC or a native model name, through the native code's EngineHash::getProfileIdsForPropertyValue(). One test over a mock native engine. One over the free data file, skipped where the file is not there, checks the ids are ascending and distinct, a value in another case matches nothing, and indexing the property with the Indexed Properties option gives the same ids. DD-22.
…s option device-detection-specification/pipeline-elements/device-detection-on-premise.md line 479 at 706de1f0: The Engine MUST be able to give the ids of the profiles that hold a given value of a given Property, such as a TAC or a native model name, through the native code's EngineHash::getProfileIdsForPropertyValue(). The native function was in the SWIG wrapper, but the engine kept its native engine private and nothing called it. DeviceDetectionHashEngine now has getProfileIdsForPropertyValue, which copies the native ids and releases the native vector. The builder has setIndexedProperties, the Indexed Properties option of the configuration table, so a lookup of an indexed property does not read every profile. DD-22.
device-detection-specification/data-model.md line 10 at 706de1f0: All Engines MUST populate data objects that implement Aspect Data as defined in the Pipeline specification. Checks the data type each engine is declared with, and the classes made to hold each engine's data. DD-2.
device-detection-specification/data-model.md line 15 at 706de1f0: Each of these Engines MUST return data that is interface compatible with the others. Checks the on-premise and cloud data are both DeviceData with the same accessors and types, and the cloud hardware lookup gives DeviceData. DD-3.
…shed device-detection-specification/pipeline-elements/device-detection-on-premise.md line 86 at 706de1f0: This MUST be done at start-up and any time the data is refreshed. Over a native engine that is a mock whose keys change between refreshes, so a key only the new data has is accepted after the refresh. Also checks the keys are stored, not read from the native code for each use. DD-11.
…asked device-detection-specification/pipeline-elements/device-detection-on-premise.md line 152 at 706de1f0: The Element Data MUST read a Property value from ResultsHash only when that Property is requested, and MUST NOT copy Property values out of ResultsHash while the Engine processes the Flow Data. Processing, over a native engine that is a mock, makes no call to the results it stores. That test makes native evidence, so it needs the native library, though no data file. Storing the results reads nothing, and a request for IsMobile reads IsMobile and nothing else. DD-13.
device-detection-specification/pipeline-elements/device-detection-on-premise.md line 189 at 706de1f0: An implementation configured with no names MUST create the Engine with no names, and MUST NOT pass the data set's names itself. Reads the names held by the native configuration the engine was created with, which the property indexes cannot show, as they are the same either way. Needs the free data file, and is skipped where it is not there. DD-14.
device-detection-specification/packaging-and-structure.md line 16 at 706de1f0: Examples and tests SHOULD NOT be included in any published packages. Lists what each published package's main jar holds and looks for test helpers, tests and examples. DD-7.
…ed jar device-detection-specification/packaging-and-structure.md line 16 at 706de1f0: Examples and tests SHOULD NOT be included in any published packages. FileUtils and KeyUtils move from the shared module's main sources to its test sources, so the tests of the other modules still reach them through the test jar. The examples used them too, so the examples' DataFileHelper now has the data file names and the search for the best data file, and their KeyHelper reads and checks keys. DD-7.
…on only device-detection-specification/pipeline-elements/device-detection-on-premise.md line 205 at 706de1f0: Every read SHOULD use the position. Reading IsMobile, at position 1, makes one call to the native results, the read by position, and none by name. DD-15.
device-detection-specification/pipeline-elements/device-detection-on-premise.md line 205 at 706de1f0: Every read SHOULD use the position. The values were read by position, but each read first asked the native results whether they held the property, by name. A property the engine was created with has a position, so it is now taken as available without that call, and a property the data set lacks still reads as having no value. A name with no position, and results from more than one engine, still use the name. DD-15.
…once device-detection-specification/pipeline-elements/device-detection-on-premise.md line 339 at 706de1f0: As such, steps SHOULD be taken to minimize this as much as possible. The last of those steps stores in the engine the values that change only on a data refresh. Counts the calls to a mock native engine for the property metadata and for the evidence keys, before and after refreshes. DD-19.
…ed logic device-detection-specification/pipeline-elements/device-detection-cloud.md line 41 at 706de1f0: The majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic. The engine should give only the parsing of its part of the response. The shared logic passes over an empty response, whose error the Cloud Request Engine has reported, and refuses a Pipeline where that engine has not run. DD-9.
…e Detection device-detection-specification/pipeline-elements/device-detection-cloud.md line 41 at 706de1f0: The majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic. DeviceDetectionCloudEngine replaced the shared processEngine with its own, so an empty response, after an error the Cloud Request Engine had reported, threw a JSON error, and a Pipeline where that engine had not run was not refused. The engine now gives only processCloudEngine, which parses its part of the response, and the shared processEngine does the rest. Two test files that called the engine's own processEngine now go through the shared one, or call processCloudEngine with the response. DD-9.
…logic device-detection-specification/pipeline-elements/hardware-profile-lookup-cloud.md line 56 at 706de1f0: The majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic. The engine gives no processing of its own. Each profile in its part of a canned response becomes DeviceData with the values and no-value reasons the response gives, and an empty response gives no profiles. DD-28.
rc already moved the example tests and the example data file helper off the packages' test helpers, in #175, so the merge keeps rc's version of those and of the TacCloud key check. The two engine files keep this draft's version, which adds the lock around every read of the native results and only takes the position shortcut with one set of results. The device detection cloud request failed test processes through the engine, as the IP Intelligence one does, now that the engine leaves processing to the shared logic.
…e examples' KeyHelper a second time
…ng else of the kind
# Conflicts: # device-detection/device-detection.cloud/src/main/java/fiftyone/devicedetection/cloud/flowelements/DeviceDetectionCloudEngine.java
…ts, now the engine leaves processing to the shared logic
…ared logic checks
The Results and threads section of the on-premise Device Detection page says Aspect Data that reads its values from a ResultsHash instance MUST NOT lock it, and that the caller is responsible for thread-safe access. The thread safety page says the same of every results instance. ResultsNotLockedTests reads the same data from two threads and expects both inside the native results at once. It replaces ResultsOneThreadAtATimeTests, which expected the opposite from an earlier text of that section.
The Results and threads section of the on-premise Device Detection page says Aspect Data that reads its values from a ResultsHash instance MUST NOT lock it, is not safe to use from several threads at once, and leaves thread-safe access to the caller. DeviceDataHashDefault no longer holds a lock for its reads of the native results or for close, and its documentation says an instance is not safe to use from several threads at once. The read by position, which needs no native lookup of the name when there is one results instance, is unchanged.
The cloud Device Detection page says the majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic. A SHOULD does not change a released test, and moving the engine onto the shared processEngine needed MissingPropertyHandlingTests and ProductMissingTests changed. The engine's processEngine on rc already passes over an empty response, marks its data when the Cloud Request Engine has recorded a failed request, and refuses a Pipeline with no Cloud Request Engine, so the move gained nothing the specification requires. DeviceDetectionCloudEngine, CloudRequestFailedTests, EmptyMetadataTests, MissingPropertyHandlingTests and ProductMissingTests are rc's again, and SharedCloudProcessingTests, which held the engine to the shared logic, goes. This undoes 5c440dc, 74985cd, c1e7424 and 96a04e4 and the merge resolution of CloudRequestFailedTests that followed them. HardwareProfileCloudEngineTests stays, as that engine already processes through the shared logic.
ben51degrees
marked this pull request as ready for review
November 19, 2026 17:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Head bfc08ce. Base
rc820a544 is merged in, and the head merges into today'srcwithout conflict.The problem
On-premise Device Detection gave no metadata of its own for the match metric Properties, did not list the Metrics component, could not give the profiles that hold a value of a Property, and looked a Property up by name in native code even where it had the position. Six requirements it already met had no test.
The specification
Links are pinned to specifications
rcd79bbe3b, and the quotations are from device-detection-on-premise.md unless another page is named.The match metric Properties carry metadata of their own. Lines 448 to 475
The Engine gives the profiles that hold a value. Lines 477 to 499
Every read uses the position. Line 205
No lock on the native results. Lines 319 to 332
Cloud processing. device-detection-cloud.md line 41 says the majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic. A SHOULD does not change a released test, so cloud Device Detection keeps its own processing (see Changes at this head).
What changed
Metricscomponent the table gives,Methodis a string rather than a list, andMetricsfollows the data file's components ingetComponents()(56b1acf, 4f5ab52).DeviceDetectionHashEngine.getProfileIdsForPropertyValuecalls the native function and keeps no index of its own, and the builder'ssetIndexedPropertiesgives the Indexed Properties option, default none (a657a4f).DeviceDataHashDefaultreads its native results with no lock, as onrc, and its documentation now says an instance is not safe to use from several threads at once and the caller is responsible for any locking (bcac3b8).ResultsNotLockedTestsholds that (4bb1bbe).HardwareProfileCloudEngineTestsholds that the hardware profile cloud engine processes through the shared logic (a4f3217).FileUtilsandKeyUtils, which stay in the jar as onrc(4c15cff, e8c2559, 60d509f, 91b52ae).Before you upgrade
getComponents()ends with a Metrics component. ItsgetDefaultProfile()is null, because the match metrics have no profile, so code that reads every component's default profile has to allow for null. Code that counts the components sees one more. The specification requires this (line 464, "This component MUST also be added to the list of components returned by the Engine"), and the released test helperMetaDataHasherHashchanges with it, hashing the metrics' component and passing over a null default profile.Methodis a string, where it was marked as a list (line 452, "Metadata for these Properties MUST be added").DeviceDetectionHashEngine.getProfileIdsForPropertyValueandDeviceDetectionHashEngineBuilder.setIndexedProperties. Nothing existing changes.Tests
Red, run 43870189689 on
tmp/red-174(workflow commit 2fed1d9), one job for each test commit before its fix. The four that still apply all failed.MatchMetricMetaDataTests, 2 of 5 fail, "MatchedNodes has no component" and "Method is a string, not a list"MetricsComponentTests, all three fail, "No component named Metrics"getProfileIdsForPropertyValueandsetIndexedProperties, which only the fix addsReadsByPositionTests.Read_OfANamedProperty_IsByPositionOnlyfailsThe run's two other jobs were for tests this draft no longer has,
ResultsOneThreadAtATimeTestsandSharedCloudProcessingTests.Red of the no-lock test, run 44451010539 on
tmp/fixconf-java-174-red(workflow commit c08bb88 on 4bb1bbe),ResultsNotLockedTestsat its test commit, before the fix. It failed as intended,Results_GetValues_NotLocked"One thread waited for the other to leave the native results for getValues, so the read took a lock expected:<2> but was:<1>".The six tests of item 5 cover behavior the code already had, so none has a red.
Green, run 44451020576 on
tmp/fixconf-java-174-green(workflow commit 97c19ca on bfc08ce), every build entry without macOS, with tests. It passed, all 9 build entries (Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21), with the two performance comparison jobs skipped.Base branch
rc820a544 is merged in with 12c0e79, and the head merges into today'srcwith no conflict. The earlier merges keptrc's side ofTacCloud, the examples'DataFileHelperand the example tests, which #175 had changed, and 88c03aa removes a second copy ofgetNamedKeyandisInvalidKeythat git merged into the examples'KeyHelperwithout a conflict, so this draft changes no file underexamples/.NoNamesTestsandProfilesForPropertyValueTeststake the free data file's path fromTestFileUtils(cdd4138), becausercbuilds with deprecations as errors.Changes at this head
Three commits follow the previous head, 91b52ae.
ResultsNotLockedTestsholds that a read of the native results throughgetValuestakes no lock. The on-premise Device Detection page says, at lines 327 to 330 at d79bbe3b, "Aspect Data that reads its values from aResultsHashinstance MUST NOT lock it, and is not safe to use from several threads at once". Red in run 44451010539, as Tests says.DeviceDataHashDefaultholds no lock for its reads of the native results or for close, its documentation says an instance is not safe to use from several threads at once and the caller is responsible for any locking, andResultsOneThreadAtATimeTests, which held the opposite, goes. The read by position is unchanged.rchas it. The cloud page says the majority of the processing SHOULD be handled by the shared Cloud Aspect Engine logic, which is not a requirement, and moving the engine onto the shared logic needed the releasedMissingPropertyHandlingTestsandProductMissingTestschanged.rc'sprocessEnginealready passes over an empty response, marks its data when the Cloud Request Engine recorded a failed request, and refuses a Pipeline with no Cloud Request Engine. SoDeviceDetectionCloudEngineand its tests arerc's again, andSharedCloudProcessingTestsgoes. This undoes 5c440dc, 74985cd, c1e7424 and 96a04e4.The green of the head is run 44451020576, as Tests says.
What remains
UserAgentsmatch metric keeps the defaultn/a, where the table gives[]. That is unchanged fromrc, and how a list-valued metric is described is an open question.rc'sDeviceDataBaseOnPremisestill holdsdataLockandgetLockaround its typed reads, which goes with the separate work to take every lock off results instances. That is also whyResultsNotLockedTestsreads throughgetValues, which does not pass through those locks.