Skip to content

Give every Property its stored string, description and data tiers, accept relative paths, and check Element Data keys - #179

Merged
ben51degrees merged 28 commits into
rcfrom
remote/java-10
Nov 19, 2026
Merged

ben51degrees merged 28 commits into
rcfrom
remote/java-10

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Head 6998871. Base rc 49bc431 is merged in.

The problem

Seven places where the Pipeline did not do what the specification asks, or did it with nothing to hold it there. A configuration file could not set an enumeration option by the name of its value unless the enumeration had a parse method of its own. A configuration file using the name the other languages use for setAutoCloseElements failed to build. Nothing showed that a data file and a temporary folder may be given as relative paths, that Element Data keys keep to the string key rules, or that every Hash engine Property carries a description and data tiers. A Property's stored string could not be read beside its parsed value. The Pipeline overhead test timed the one call its cache misses.

The specification

Links are pinned to specifications rc d79bbe3b, with the thread safety page at 52e9b047, the same text as rc.

A configuration file gives a list as a comma-separated string and an enumeration by its name. pipeline-configuration.md lines 55 to 61

To date, the following types are supported in configuration files for all languages: ... list of strings (represented as a comma-separated string), enumeration value (represented as the string name of the value)

A configuration option has the name the code uses. Lines 40 to 43

The configuration file definition MUST be as similar as possible to configuration files for other languages. All configuration options SHOULD use the same names in the configuration file as would be used when configuring in code.

A path may be relative. file-system-paths.md lines 3 to 5

Where file system paths are supplied as parameters, the Pipeline API MUST be able to work with relative paths. A relative path is resolved against the working directory of the process.

String keys. access-to-results.md lines 5 to 11

Strings SHOULD only contain alphanumeric characters, full stop or hyphen. ... Lower-case SHOULD be used. String comparisons MUST be case insensitive.

The stored string. access-to-results.md lines 65 to 77 says it SHOULD be possible to get the stored string for a Property as well as the parsed value, and value-types.md lines 36 to 53 says how

It MUST expose them through a capability a caller can test for at run time ... The stored value MUST be returned as an Aspect Property Value of string ... Offering the stored value MUST NOT change what any existing accessor returns ... It MUST return the value as it was stored, and MUST NOT compose a string from a parsed value.

No lock on a results instance. thread-safety.md, Results instances

An Engine, its results instances and the Element Data that reads from them MUST NOT take a lock to make those reads safe for several threads.

Property metadata. properties.md lines 100 to 109 has Aspect Engines REQUIRED to populate Description and "Data tiers where present".

What changed

  1. Enumerations in a configuration file. PipelineBuilder sets an enumeration option from the name of its value, matched without regard to case. An enumeration with a parse method of its own is left to that method (2566fe8, 3195f5d). The comma split of a property list reached rc through Split a property list given as one string at every comma #166, so this draft adds its tests (f9eb229) and the enumeration.
  2. The name the other languages use. setAutoCloseElements carries @AlternateName("SetAutoDisposeElements"), so a configuration file written for .NET builds a Pipeline here too (3d978c3). The Java name still works.
  3. Relative paths. RelativePathTests gives a data file and a temporary folder as relative paths and builds an engine from them (7002189). The code already did this, and nothing held it to it.
  4. Element Data keys. ElementDataKeyFormatTests in the JavaScript Builder, Device Detection cloud, IP Intelligence cloud and Location modules hold every key to alphanumeric characters, full stops and hyphens, in lower case (6ad767f, b157ff8). ElementDataKeyCaseTests holds the case insensitive lookup (99ffe2e), whose code reached rc through Find Element Data by its key whatever the case #167.
  5. The stored string. ProvidesValuesAsString.getStoredValueAsString returns the string the data file holds, as an AspectPropertyValue<String>, from the Device Detection and IP Intelligence on-premise Element Data (61a505c). It reads the Property through the existing accessor first, so a missing Property is reported the same way, and it returns the native string, never one made from a parsed value. It reads the results instance with no lock of its own, and its documentation says the instance is not safe to use from several threads at once (6998871).
  6. Property metadata. PropertyMetaDataTests holds every Hash engine Property to a description and data tiers (6c3c550).
  7. The Pipeline overhead test. The cached Pipeline's overhead is timed after the one call that misses the cache (e1f17c3). See Before you upgrade item 4.

The missing Property messages and the order their reason is decided in (02a4d28, df84ae1, f9b48b7, 1c59f13) reached rc through #175, and the merges keep rc's code and tests for them, so the net diff holds none of that work.

Before you upgrade

  1. An enumeration option may be named in a configuration file. A value given as the name of one of the enumeration's values, in any case, now sets the option where it used to fail with "does not have the expected 'parse' method". An enumeration with its own parse method behaves as before. Nothing to change.
  2. SetAutoDisposeElements is read for setAutoCloseElements in a configuration file. No name is removed. Nothing to change.
  3. Device Detection and IP Intelligence on-premise Element Data implement ProvidesValuesAsString. The new public method getStoredValueAsString(String) takes no lock, so an instance shared between threads needs the caller's own coordination, as for every other accessor. A subclass that already declares a method of that name and argument with a different return type no longer compiles.
  4. A released test changes. PipelineOverheadTests (on main) now fills the cache with one call before timing 10,000 calls, and asserts that the engine processed once. The call that misses the cache costs 0.2 seconds, 0.02 ms of the permitted average, so the test measured the engine's process cost and not the Pipeline's overhead. No specification sentence requires this, and no user has to change anything.

Tests

Red, run 43870006060 on tmp/red-179 (workflow commit 3570edd on 464d294). Each job checks out one test commit before its fix, or breaks on purpose one rule a guard holds, and all four failed.

Job What it ran Result
A list and an enumeration in a configuration file test commit f9eb229, ConfigurationValueTypesTests 2 failures and 1 error, the list kept as one string ("expected:<[IsMobile, BrowserName]> but was:<[IsMobile,BrowserName]>") and setSpeed refused
The names the .NET sample uses test commit 0dd259f, ConfigurationFileLanguagesTest "The following builder parameters could not be processed: setautodisposeelements"
The stored string test commit 294688b, StoredValueAsStringTests Stops at test compile, "cannot find symbol" for ProvidesValuesAsString, which only the fix adds
An Element Data key with a capital and an underscore the JavaScript Builder's key changed to JavaScript_Builder, ElementDataKeyFormatTests Both rules fail on that key, so the guard is shown working by breaking it

The relative path, key case and Property metadata tests cover behavior the code already had, so none has a red. The lock removal in 6998871 has no red either, because getStoredValueAsString first reads the Property through the typed accessor, which still takes the older lock in tryGetValue that is on rc and goes with the lock removal work, so no test can tell the new method's own lock apart yet.

Green, run 44450152079 on tmp/fixconf-java-179-green (workflow commit 9a8e986 on 6998871), every build entry without macOS, with tests. It passed, all 9 build entries (Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21), with the two performance comparison jobs skipped.

Base branch

rc is merged in four times, the latest being 49bc431. That merge had one conflict, in the imports of PipelineBuilderBase, where rc added PipelineConfigurationException for its check that usage sharing runs last and this draft added AlternateName. Both imports are kept, with rc's checkRunsLast and closeMadeElements and this draft's annotation, which merged without conflict.

The earlier merges keep rc's side of work this draft repeated. The missing Property messages and reason order are #175's (a52f006 puts back rc's MissingPropertyServiceTests, which an earlier merge had changed), the comma split is #166's and the key case lookup is #167's (9af6ee0 keeps rc's TypedKeyMapBuilder). 3472227 moves PropertyMetaDataTests off the deprecated FileUtils, because rc builds with deprecations as errors. 56c21f5 gives the IP Intelligence API web example a dev profile that uses this repository's JSON Builder, because with -Pdev it took the published one, which does not hold ProvidesValuesAsString, and failed with ClassNotFoundException. Built without -Pdev it still takes the published packages.

Changes at the last head

59d11e4 merges rc at 49bc431, keeping both sides' imports in PipelineBuilderBase. 6998871 takes the lock out of getStoredValueAsString, which has no red for the reason under Tests. The green is the run above, and Before you upgrade covers the head.

What remains

  1. The IP Intelligence getStoredValueAsString has no test of its own, and the SetAutoDisposeElements test checks only that the name is accepted, not that the option took effect. Both are small follow ups.
  2. The older lock in tryGetValue of both on-premise Element Data classes is rc's and goes with the separate lock removal work.

… an enumeration by name (CF-8)

Specification: pipeline-specification/features/pipeline-configuration.md
line 50 at 51Degrees/specifications 706de1f0, "Configuration files MUST
be human-readable and consistent between languages.", with lines 60 and
61 giving a list of strings as a comma-separated string and an
enumeration value as the string name of the value.
…ation option by its name (CF-8)

setProperties(String) split only at a comma with one white space
character each side, so "IsMobile,BrowserName" became one property.
It now splits at every comma and trims each name. The Pipeline builder
now sets an option whose method takes an enumeration from the name of
the value, without regard to case, where it needed a String overload.

Specification: pipeline-specification/features/pipeline-configuration.md
line 50 at 51Degrees/specifications 706de1f0, "Configuration files MUST
be human-readable and consistent between languages."
Specification: pipeline-specification/features/access-to-results.md
line 11 at 51Degrees/specifications 706de1f0, "String comparisons MUST
be case insensitive."
The map that holds the Element Data in a FlowData compared key names
with their case, so flowData.get("Device") threw where the key was
"device". It now compares them with equalsIgnoreCase, which does not
depend on the default locale, for get, tryGet, getOrAdd, containsKey
and removal alike.

Specification: pipeline-specification/features/access-to-results.md
line 11 at 51Degrees/specifications 706de1f0, "String comparisons MUST
be case insensitive."
…isses

PipelineOverhead_Cache timed 10,000 calls including the first, which
misses the cache and sleeps for 0.2 sec, so 0.02 ms of the 0.1 ms
permitted average was the process cost and not the overhead, and the
test failed at 0.108 ms on a busy machine. The test now fills the cache
with one call before timing, and checks that the cache answered every
later call by counting the times the engine processed, which does not
depend on time.
…hs (CF-1)

An on-premise engine reads a data file given relative to the working
directory, resolves a relative temporary folder against it, and the
file system watcher loads a new file at a relative path.

Specification: pipeline-specification/features/file-system-paths.md
line 3 at 51Degrees/specifications 706de1f0, "Where file system paths
are supplied as parameters, the Pipeline API MUST be able to work with
relative paths."
… a Pipeline (CF-5)

The file has the structure and names of the .NET sample configuration
file, examples/device-detection/sample-configuration.json in
51degrees-dotnet, written as XML, including the Pipeline level
SetAutoDisposeElements and SetSuppressProcessException.

Specification: pipeline-specification/features/pipeline-configuration.md
line 40 at 51Degrees/specifications 706de1f0, "The configuration file
definition MUST be as similar as possible to configuration files for
other languages."
@jwrosewell jwrosewell added the remote-conformance Specification conformance work from the remote packs label Nov 15, 2026
…etAutoCloseElements (CF-5)

The .NET sample configuration file sets the Pipeline option
SetAutoDisposeElements. The Java Pipeline builder names the option
setAutoCloseElements, so the same file failed with "The following
builder parameters could not be processed". An AlternateName lets a
configuration file give it by either name.

Specification: pipeline-specification/features/pipeline-configuration.md
line 40 at 51Degrees/specifications 706de1f0, "The configuration file
definition MUST be as similar as possible to configuration files for
other languages."
…-15)

Each Property has a description and a list of the data tiers where it is
present, and each Property available in the data file lists that data
file's tier.

Specification: pipeline-specification/features/properties.md line 101
at 51Degrees/specifications 706de1f0, "Aspect Engines are REQUIRED to
populate some additional metadata values beyond those described for
Flow Elements.", with lines 108 and 109 naming Description and Data
tiers where present.
…sed value (PR-9)

On-premise Device Data offers the stored string through a capability a
caller can test for, returns it as stored, leaves the typed accessors
as they were, and reports a missing Property as the typed accessor
does.

Specification: pipeline-specification/features/access-to-results.md
line 66 at 51Degrees/specifications 706de1f0, "Where an Element Data
instance holds Property values as strings and parses them into the type
each Property declares, it SHOULD be possible to get the stored string
for a Property as well as the parsed value."
…esValuesAsString (PR-9)

On-premise Device Detection and IP Intelligence data hold Property
values as strings but kept the stored string behind a protected method.
A new interface, ProvidesValuesAsString, which a caller can test for,
gives getStoredValueAsString. It reads the Property through the same
route as the other accessors first, so a missing Property throws the
same PropertyMissingException, then returns the string as stored. The
typed accessors are unchanged. The protected getValueAsString keeps its
name and access, so no subclass has to change.

Specification: pipeline-specification/features/access-to-results.md
line 66 at 51Degrees/specifications 706de1f0, "Where an Element Data
instance holds Property values as strings and parses them into the type
each Property declares, it SHOULD be possible to get the stored string
for a Property as well as the parsed value."
…'s table (PR-17)

The campaign parameters on each link are removed before comparing, as
the specification lets an implementation add them.

Specification: pipeline-specification/features/properties.md line 136
at 51Degrees/specifications 706de1f0, "An implementation SHOULD give the
same text, and MAY add to each link the campaign parameters its own
repository requires.", for the table at lines 123 to 131 and the rule at
lines 134 and 135 that lists are joined with a comma and no space.
The upgrade message now says "license or data file", links the pricing
page and ends with a full stop, and has a second form for when the data
tiers are not known. The resource key messages link the configurator
rather than a documentation page, and list the Properties joined with a
comma and no space. The text for a failed cloud request is added. Each
link keeps this repository's campaign parameters.

Specification: pipeline-specification/features/properties.md line 136
at 51Degrees/specifications 706de1f0, "An implementation SHOULD give the
same text, and MAY add to each link the campaign parameters its own
repository requires."
…n's order (PR-18)

Covers step 1, a failed cloud request naming the first Engine; step 2,
the last Engine named when every Engine gives Unknown; step 3.1, the
data tier compared ignoring case and checked only for on-premise
Engines; step 3.2, a cloud Engine whose metadata is not loaded; and
step 3.3, an on-premise typed accessor, inherited or not.

Specification: pipeline-specification/features/properties.md line 143
at 51Degrees/specifications 706de1f0, "An implementation SHOULD decide
the reason in this order, using the first that applies.", for the steps
at lines 146 to 177.
…rder (PR-18)

- Step 1: a new reason, CloudRequestFailed, when a cloud engine's
  request for this Flow Data failed, naming the first engine. The
  missing property service has a new overload that takes the Flow Data,
  whose default uses the old one, and Aspect Data passes its Flow Data.
  CloudAspectEngine has requestFailed(FlowData), which
  CloudAspectEngineBase answers from the Cloud Request Engine's data
  and errors for the Flow Data.
- Step 3.1: the data file tier is compared ignoring case, and only for
  on-premise engines, so a listed cloud property is never given
  DataFileUpgradeRequired.
- Step 3.2: a cloud engine that has not loaded its metadata gives
  CloudRequestFailed where its request for the property list failed,
  and Unknown otherwise, rather than ProductNotAccessible.
- Step 3.3: an on-premise engine whose Element Data has a typed
  accessor for the property, inherited or not, gives
  DataFileUpgradeRequired with the message for unknown data tiers.

The cloud engine mocks in the missing property tests now say they have
loaded their metadata, as the reasons they check need. The test's list
helper no longer uses varargs, which the build's -Werror refused.

Specification: pipeline-specification/features/properties.md line 143
at 51Degrees/specifications 706de1f0, "An implementation SHOULD decide
the reason in this order, using the first that applies."
…or hyphen (PR-1)

Checks the keys of the JavaScript Builder, JSON Builder, Sequence, Set
Headers and Usage Sharing elements, the Device Detection and IP
Intelligence cloud engines and the Location cloud engine. The
DigitalElement Location key, location_digitalelement, is left out
because it is the cloud service's name for that product, which the
engine reads its response and property list by.

Specification: pipeline-specification/features/access-to-results.md
line 7 at 51Degrees/specifications 706de1f0, "Strings SHOULD only
contain alphanumeric characters, full stop or hyphen."
Checks the same elements and engines as the key format test, with every
Location provider's key, the DigitalElement one included.

Specification: pipeline-specification/features/access-to-results.md
line 10 at 51Degrees/specifications 706de1f0, "Lower-case SHOULD be
used."
…en reading configuration (CF-8)

The change that set an enumeration option by the name of its value ran
before the type's own parse method, so MissingTranslationBehavior's
spellings such as "empty_string", and its message for a value it does
not know, were refused. Name matching now applies only to an
enumeration without a parse method. TranslationEngineConfigurationTests
passes again (15 of 15).

Specification: pipeline-specification/features/pipeline-configuration.md
line 50 at 51Degrees/specifications 706de1f0, "Configuration files MUST
be human-readable and consistent between languages."
rc already carries the missing Property messages and the order the reason
is decided in, from #175, so the merge keeps rc's version of those six
files and this draft's own implementation of them goes, along with its two
test files for them, which rc covers with MissingPropertyMessageTests and
MissingPropertyReasonTests.
@jwrosewell jwrosewell changed the title Specification conformance, 51degrees-java, section 10 Give every Property its stored string, description and data tiers, accept relative paths, and check Element Data keys Nov 16, 2026
# Conflicts:
#	pipeline/pipeline.engines/src/main/java/fiftyone/pipeline/engines/flowelements/AspectEngineBuilderBase.java
rc 49bc431. One conflict, in the imports of PipelineBuilderBase. rc added
the PipelineConfigurationException import for checkRunsLast, which refuses
usage sharing anywhere but last, and this branch added the AlternateName
import for the SetAutoDisposeElements name. Both imports are kept, with
rc's checkRunsLast and closeMadeElements and this branch's AlternateName
annotation on setAutoCloseElements, which merged without conflict.
getStoredValueAsString in the Device Detection and IP Intelligence on-premise
Element Data read the value inside the getLock monitor. The thread safety
page says an Engine, its results instances and the Element Data that reads
from them MUST NOT take a lock to make those reads safe for several threads,
and that the caller coordinates a shared instance. Both methods now read the
value with no lock of their own, and their documentation says the instance
is not safe to use from several threads at once.
@ben51degrees
ben51degrees marked this pull request as ready for review November 19, 2026 05:08
@ben51degrees
ben51degrees merged commit 3bb6e71 into rc Nov 19, 2026
7 checks passed
@ben51degrees
ben51degrees deleted the remote/java-10 branch November 19, 2026 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Specification conformance work from the remote packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants