Repository navigation
Give every Property its stored string, description and data tiers, accept relative paths, and check Element Data keys - #179
Merged
Merged
Conversation
… an enumeration by name (CF-8) Specification: pipeline-specification/features/pipeline-configuration.md line 50 at 51Degrees/specifications 706de1f0, "Configuration files MUST be human-readable and consistent between languages.", with lines 60 and 61 giving a list of strings as a comma-separated string and an enumeration value as the string name of the value.
…ation option by its name (CF-8) setProperties(String) split only at a comma with one white space character each side, so "IsMobile,BrowserName" became one property. It now splits at every comma and trims each name. The Pipeline builder now sets an option whose method takes an enumeration from the name of the value, without regard to case, where it needed a String overload. Specification: pipeline-specification/features/pipeline-configuration.md line 50 at 51Degrees/specifications 706de1f0, "Configuration files MUST be human-readable and consistent between languages."
Specification: pipeline-specification/features/access-to-results.md line 11 at 51Degrees/specifications 706de1f0, "String comparisons MUST be case insensitive."
The map that holds the Element Data in a FlowData compared key names
with their case, so flowData.get("Device") threw where the key was
"device". It now compares them with equalsIgnoreCase, which does not
depend on the default locale, for get, tryGet, getOrAdd, containsKey
and removal alike.
Specification: pipeline-specification/features/access-to-results.md
line 11 at 51Degrees/specifications 706de1f0, "String comparisons MUST
be case insensitive."
…isses PipelineOverhead_Cache timed 10,000 calls including the first, which misses the cache and sleeps for 0.2 sec, so 0.02 ms of the 0.1 ms permitted average was the process cost and not the overhead, and the test failed at 0.108 ms on a busy machine. The test now fills the cache with one call before timing, and checks that the cache answered every later call by counting the times the engine processed, which does not depend on time.
…hs (CF-1) An on-premise engine reads a data file given relative to the working directory, resolves a relative temporary folder against it, and the file system watcher loads a new file at a relative path. Specification: pipeline-specification/features/file-system-paths.md line 3 at 51Degrees/specifications 706de1f0, "Where file system paths are supplied as parameters, the Pipeline API MUST be able to work with relative paths."
… a Pipeline (CF-5) The file has the structure and names of the .NET sample configuration file, examples/device-detection/sample-configuration.json in 51degrees-dotnet, written as XML, including the Pipeline level SetAutoDisposeElements and SetSuppressProcessException. Specification: pipeline-specification/features/pipeline-configuration.md line 40 at 51Degrees/specifications 706de1f0, "The configuration file definition MUST be as similar as possible to configuration files for other languages."
…etAutoCloseElements (CF-5) The .NET sample configuration file sets the Pipeline option SetAutoDisposeElements. The Java Pipeline builder names the option setAutoCloseElements, so the same file failed with "The following builder parameters could not be processed". An AlternateName lets a configuration file give it by either name. Specification: pipeline-specification/features/pipeline-configuration.md line 40 at 51Degrees/specifications 706de1f0, "The configuration file definition MUST be as similar as possible to configuration files for other languages."
…-15) Each Property has a description and a list of the data tiers where it is present, and each Property available in the data file lists that data file's tier. Specification: pipeline-specification/features/properties.md line 101 at 51Degrees/specifications 706de1f0, "Aspect Engines are REQUIRED to populate some additional metadata values beyond those described for Flow Elements.", with lines 108 and 109 naming Description and Data tiers where present.
…sed value (PR-9) On-premise Device Data offers the stored string through a capability a caller can test for, returns it as stored, leaves the typed accessors as they were, and reports a missing Property as the typed accessor does. Specification: pipeline-specification/features/access-to-results.md line 66 at 51Degrees/specifications 706de1f0, "Where an Element Data instance holds Property values as strings and parses them into the type each Property declares, it SHOULD be possible to get the stored string for a Property as well as the parsed value."
…esValuesAsString (PR-9) On-premise Device Detection and IP Intelligence data hold Property values as strings but kept the stored string behind a protected method. A new interface, ProvidesValuesAsString, which a caller can test for, gives getStoredValueAsString. It reads the Property through the same route as the other accessors first, so a missing Property throws the same PropertyMissingException, then returns the string as stored. The typed accessors are unchanged. The protected getValueAsString keeps its name and access, so no subclass has to change. Specification: pipeline-specification/features/access-to-results.md line 66 at 51Degrees/specifications 706de1f0, "Where an Element Data instance holds Property values as strings and parses them into the type each Property declares, it SHOULD be possible to get the stored string for a Property as well as the parsed value."
…'s table (PR-17) The campaign parameters on each link are removed before comparing, as the specification lets an implementation add them. Specification: pipeline-specification/features/properties.md line 136 at 51Degrees/specifications 706de1f0, "An implementation SHOULD give the same text, and MAY add to each link the campaign parameters its own repository requires.", for the table at lines 123 to 131 and the rule at lines 134 and 135 that lists are joined with a comma and no space.
The upgrade message now says "license or data file", links the pricing page and ends with a full stop, and has a second form for when the data tiers are not known. The resource key messages link the configurator rather than a documentation page, and list the Properties joined with a comma and no space. The text for a failed cloud request is added. Each link keeps this repository's campaign parameters. Specification: pipeline-specification/features/properties.md line 136 at 51Degrees/specifications 706de1f0, "An implementation SHOULD give the same text, and MAY add to each link the campaign parameters its own repository requires."
…n's order (PR-18) Covers step 1, a failed cloud request naming the first Engine; step 2, the last Engine named when every Engine gives Unknown; step 3.1, the data tier compared ignoring case and checked only for on-premise Engines; step 3.2, a cloud Engine whose metadata is not loaded; and step 3.3, an on-premise typed accessor, inherited or not. Specification: pipeline-specification/features/properties.md line 143 at 51Degrees/specifications 706de1f0, "An implementation SHOULD decide the reason in this order, using the first that applies.", for the steps at lines 146 to 177.
…rder (PR-18) - Step 1: a new reason, CloudRequestFailed, when a cloud engine's request for this Flow Data failed, naming the first engine. The missing property service has a new overload that takes the Flow Data, whose default uses the old one, and Aspect Data passes its Flow Data. CloudAspectEngine has requestFailed(FlowData), which CloudAspectEngineBase answers from the Cloud Request Engine's data and errors for the Flow Data. - Step 3.1: the data file tier is compared ignoring case, and only for on-premise engines, so a listed cloud property is never given DataFileUpgradeRequired. - Step 3.2: a cloud engine that has not loaded its metadata gives CloudRequestFailed where its request for the property list failed, and Unknown otherwise, rather than ProductNotAccessible. - Step 3.3: an on-premise engine whose Element Data has a typed accessor for the property, inherited or not, gives DataFileUpgradeRequired with the message for unknown data tiers. The cloud engine mocks in the missing property tests now say they have loaded their metadata, as the reasons they check need. The test's list helper no longer uses varargs, which the build's -Werror refused. Specification: pipeline-specification/features/properties.md line 143 at 51Degrees/specifications 706de1f0, "An implementation SHOULD decide the reason in this order, using the first that applies."
…or hyphen (PR-1) Checks the keys of the JavaScript Builder, JSON Builder, Sequence, Set Headers and Usage Sharing elements, the Device Detection and IP Intelligence cloud engines and the Location cloud engine. The DigitalElement Location key, location_digitalelement, is left out because it is the cloud service's name for that product, which the engine reads its response and property list by. Specification: pipeline-specification/features/access-to-results.md line 7 at 51Degrees/specifications 706de1f0, "Strings SHOULD only contain alphanumeric characters, full stop or hyphen."
Checks the same elements and engines as the key format test, with every Location provider's key, the DigitalElement one included. Specification: pipeline-specification/features/access-to-results.md line 10 at 51Degrees/specifications 706de1f0, "Lower-case SHOULD be used."
…en reading configuration (CF-8) The change that set an enumeration option by the name of its value ran before the type's own parse method, so MissingTranslationBehavior's spellings such as "empty_string", and its message for a value it does not know, were refused. Name matching now applies only to an enumeration without a parse method. TranslationEngineConfigurationTests passes again (15 of 15). Specification: pipeline-specification/features/pipeline-configuration.md line 50 at 51Degrees/specifications 706de1f0, "Configuration files MUST be human-readable and consistent between languages."
rc already carries the missing Property messages and the order the reason is decided in, from #175, so the merge keeps rc's version of those six files and this draft's own implementation of them goes, along with its two test files for them, which rc covers with MissingPropertyMessageTests and MissingPropertyReasonTests.
…ed cloud request this draft no longer needs
…c took two lines out of
…er with -Pdev, as it does the other packages
This was referenced Nov 16, 2026
# Conflicts: # pipeline/pipeline.engines/src/main/java/fiftyone/pipeline/engines/flowelements/AspectEngineBuilderBase.java
rc 49bc431. One conflict, in the imports of PipelineBuilderBase. rc added the PipelineConfigurationException import for checkRunsLast, which refuses usage sharing anywhere but last, and this branch added the AlternateName import for the SetAutoDisposeElements name. Both imports are kept, with rc's checkRunsLast and closeMadeElements and this branch's AlternateName annotation on setAutoCloseElements, which merged without conflict.
getStoredValueAsString in the Device Detection and IP Intelligence on-premise Element Data read the value inside the getLock monitor. The thread safety page says an Engine, its results instances and the Element Data that reads from them MUST NOT take a lock to make those reads safe for several threads, and that the caller coordinates a shared instance. Both methods now read the value with no lock of their own, and their documentation says the instance is not safe to use from several threads at once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Head 6998871. Base
rc49bc431 is merged in.The problem
Seven places where the Pipeline did not do what the specification asks, or did it with nothing to hold it there. A configuration file could not set an enumeration option by the name of its value unless the enumeration had a parse method of its own. A configuration file using the name the other languages use for
setAutoCloseElementsfailed to build. Nothing showed that a data file and a temporary folder may be given as relative paths, that Element Data keys keep to the string key rules, or that every Hash engine Property carries a description and data tiers. A Property's stored string could not be read beside its parsed value. The Pipeline overhead test timed the one call its cache misses.The specification
Links are pinned to specifications
rcd79bbe3b, with the thread safety page at 52e9b047, the same text asrc.A configuration file gives a list as a comma-separated string and an enumeration by its name. pipeline-configuration.md lines 55 to 61
A configuration option has the name the code uses. Lines 40 to 43
A path may be relative. file-system-paths.md lines 3 to 5
String keys. access-to-results.md lines 5 to 11
The stored string. access-to-results.md lines 65 to 77 says it SHOULD be possible to get the stored string for a Property as well as the parsed value, and value-types.md lines 36 to 53 says how
No lock on a results instance. thread-safety.md, Results instances
Property metadata. properties.md lines 100 to 109 has Aspect Engines REQUIRED to populate Description and "Data tiers where present".
What changed
PipelineBuildersets an enumeration option from the name of its value, matched without regard to case. An enumeration with a parse method of its own is left to that method (2566fe8, 3195f5d). The comma split of a property list reachedrcthrough Split a property list given as one string at every comma #166, so this draft adds its tests (f9eb229) and the enumeration.setAutoCloseElementscarries@AlternateName("SetAutoDisposeElements"), so a configuration file written for .NET builds a Pipeline here too (3d978c3). The Java name still works.RelativePathTestsgives a data file and a temporary folder as relative paths and builds an engine from them (7002189). The code already did this, and nothing held it to it.ElementDataKeyFormatTestsin the JavaScript Builder, Device Detection cloud, IP Intelligence cloud and Location modules hold every key to alphanumeric characters, full stops and hyphens, in lower case (6ad767f, b157ff8).ElementDataKeyCaseTestsholds the case insensitive lookup (99ffe2e), whose code reachedrcthrough Find Element Data by its key whatever the case #167.ProvidesValuesAsString.getStoredValueAsStringreturns the string the data file holds, as anAspectPropertyValue<String>, from the Device Detection and IP Intelligence on-premise Element Data (61a505c). It reads the Property through the existing accessor first, so a missing Property is reported the same way, and it returns the native string, never one made from a parsed value. It reads the results instance with no lock of its own, and its documentation says the instance is not safe to use from several threads at once (6998871).PropertyMetaDataTestsholds every Hash engine Property to a description and data tiers (6c3c550).The missing Property messages and the order their reason is decided in (02a4d28, df84ae1, f9b48b7, 1c59f13) reached
rcthrough #175, and the merges keeprc's code and tests for them, so the net diff holds none of that work.Before you upgrade
SetAutoDisposeElementsis read forsetAutoCloseElementsin a configuration file. No name is removed. Nothing to change.ProvidesValuesAsString. The new public methodgetStoredValueAsString(String)takes no lock, so an instance shared between threads needs the caller's own coordination, as for every other accessor. A subclass that already declares a method of that name and argument with a different return type no longer compiles.PipelineOverheadTests(onmain) now fills the cache with one call before timing 10,000 calls, and asserts that the engine processed once. The call that misses the cache costs 0.2 seconds, 0.02 ms of the permitted average, so the test measured the engine's process cost and not the Pipeline's overhead. No specification sentence requires this, and no user has to change anything.Tests
Red, run 43870006060 on
tmp/red-179(workflow commit 3570edd on 464d294). Each job checks out one test commit before its fix, or breaks on purpose one rule a guard holds, and all four failed.ConfigurationValueTypesTestssetSpeedrefusedConfigurationFileLanguagesTestStoredValueAsStringTestsProvidesValuesAsString, which only the fix addsJavaScript_Builder,ElementDataKeyFormatTestsThe relative path, key case and Property metadata tests cover behavior the code already had, so none has a red. The lock removal in 6998871 has no red either, because
getStoredValueAsStringfirst reads the Property through the typed accessor, which still takes the older lock intryGetValuethat is onrcand goes with the lock removal work, so no test can tell the new method's own lock apart yet.Green, run 44450152079 on
tmp/fixconf-java-179-green(workflow commit 9a8e986 on 6998871), every build entry without macOS, with tests. It passed, all 9 build entries (Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21), with the two performance comparison jobs skipped.Base branch
rcis merged in four times, the latest being 49bc431. That merge had one conflict, in the imports ofPipelineBuilderBase, wherercaddedPipelineConfigurationExceptionfor its check that usage sharing runs last and this draft addedAlternateName. Both imports are kept, withrc'scheckRunsLastandcloseMadeElementsand this draft's annotation, which merged without conflict.The earlier merges keep
rc's side of work this draft repeated. The missing Property messages and reason order are #175's (a52f006 puts backrc'sMissingPropertyServiceTests, which an earlier merge had changed), the comma split is #166's and the key case lookup is #167's (9af6ee0 keepsrc'sTypedKeyMapBuilder). 3472227 movesPropertyMetaDataTestsoff the deprecatedFileUtils, becausercbuilds with deprecations as errors. 56c21f5 gives the IP Intelligence API web example adevprofile that uses this repository's JSON Builder, because with-Pdevit took the published one, which does not holdProvidesValuesAsString, and failed withClassNotFoundException. Built without-Pdevit still takes the published packages.Changes at the last head
59d11e4 merges
rcat 49bc431, keeping both sides' imports inPipelineBuilderBase. 6998871 takes the lock out ofgetStoredValueAsString, which has no red for the reason under Tests. The green is the run above, and Before you upgrade covers the head.What remains
getStoredValueAsStringhas no test of its own, and theSetAutoDisposeElementstest checks only that the name is accepted, not that the option took effect. Both are small follow ups.tryGetValueof both on-premise Element Data classes isrc's and goes with the separate lock removal work.