Skip to content

Specification conformance, 51degrees-java, section 13 - #171

Merged
jwrosewell merged 46 commits into
rcfrom
remote/java-13
Nov 21, 2026
Merged

jwrosewell merged 46 commits into
rcfrom
remote/java-13

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The Java packages differed from the core features of the specification in ten places, and nine more rules had no test.

  1. A result cache stored and served a cloud response that holds a 51Did, which is different for every request.
  2. The Pipeline interface did not expose whether process exceptions are suppressed, so a wrapper such as the web filter could not read it.
  3. On-premise Device Detection and IP Intelligence Element Data did not say that it is not safe to use from several threads at once, although each read writes into the native results.
  4. Latitude and Longitude converted from Azimuth and Declination were not rounded to 3 decimal places.
  5. A built Pipeline kept the builder's own list of elements, so an element added to the builder afterwards joined the Pipeline already built.
  6. An error in the web filter's own work escaped doFilter even when the Pipeline suppresses process exceptions.
  7. An unexpected error in a scheduled data update check left the background task, so nothing logged it.
  8. FlowData declared none of the exceptions its methods throw.
  9. getWhere threw when an element had not run or a Property had no value, neither of which is the caller's mistake.
  10. 83 public builder setters gave no default value beside them, and four of the defaults given were wrong.

What the specification says

Item 3 and the thread safety lines below are at 51Degrees/specifications d79bbe3b, the current specifications rc. The rest are at 706de1f0, which is older than the thread safety rules.

  1. caching.md lines 144 to 150

    A cache MUST NOT store a result that holds such a value, and MUST NOT serve one that it finds. For the Cloud Request Engine, a response whose JSON has a top-level member named fodid, in any case, is neither stored in nor served from the cache, and the request goes to the cloud service instead. Every other response is cached as before.

  2. exception-handling.md lines 42 to 44

    This property MUST be exposed as a public property as part of the Pipeline public interface as IsSuppressProcessExceptions

  3. thread-safety.md lines 205 to 211 and 219 to 221

    An Engine, its results instances and the Element Data that reads from them MUST NOT take a lock to make those reads safe for several threads.

    Code comments and documentation MUST say plainly that a results instance, and Element Data that reads from one, is not safe to use from several threads at once, and that the caller is responsible for thread-safe access.

  4. value-types.md lines 99 to 101

    A value converted from Azimuth or Declination MUST be given to the caller rounded to 3 decimal places, whether it is given as a number, as a string or as a point in a WKT shape.

  5. conceptual-overview.md lines 118 to 119

    A Pipeline instance MUST be immutable. In other words, once created, it cannot be changed by adding new elements, removing old ones, etc.

  6. exception-handling.md lines 73 to 75

    The implementor MUST ensure that exceptions/errors that happen in these areas are handled correctly in order to avoid the API taking down the customer's production website.

  7. exception-handling.md lines 83 to 84

    When errors do occur, they will be logged and MUST NOT cause the process to crash.

  8. exception-handling.md lines 17 to 18

    In all cases, standard language features SHOULD be used to let IDEs and users know what exceptions/errors a particular function can throw.

  9. exception-handling.md lines 57 to 59

    The general guidance is that these functions and accessors SHOULD only throw errors/exceptions if the failure is likely due to something that the API user has done wrong.

  10. reference-implementation-notes.md lines 79 to 80

    In all cases, there SHOULD be some mechanism to specify the default values alongside configuration property setters.

The rules that gain a test here, where the code already met them, are README.md line 95 (cloud and on-premise engines can be substituted), multi-pipeline-elements.md lines 16 to 19 (state kept for each Pipeline), thread-safety.md lines 88 to 115 at d79bbe3b (Element Data added with one compare and swap and no lock, errors from a parallel group added under a lock), conceptual-overview.md line 73 (what each element MUST give), caching.md lines 59, 71 and 82 (cache keys from the advertised Evidence, keys compared ignoring case and values with case), resource-cleanup.md lines 27 to 29 (an option not to close shared elements) and resource-cleanup.md lines 84 to 85 (Element Data that needs no cleanup is left alone).

What changed

Links are to the head, b39a1e2.

  1. A 51Did response is neither stored nor served by the cache. AspectEngineBase asks a new protected isCacheable before it stores a result and before it serves one it finds. The default answers true. CloudRequestEngineDefault answers false for a response with a top-level fodid member in any case, so that request goes to the cloud service. With the deprecated lazy loading option, the result is stored on the processing thread before it has been processed, as before this draft, and the check before serving keeps out one that may not be cached. isCacheable reads the stored response without waiting, through a new protected AspectDataBase.getWithoutWaiting, so a cached engine with lazy loading returns from process without waiting for the cloud service. Without lazy loading nothing changes.
  2. Pipeline.isSuppressProcessExceptions(). Pipeline gains it as a default method that gives the builder's default, so a class that already implements Pipeline still compiles, and PipelineDefault gives the value the builder set.
  3. Element Data that reads native results says it is not thread-safe. The class comments of DeviceDataBaseOnPremise and IPIntelligenceDataBaseOnPremise say that an instance is not safe to use from several threads at once and that the caller is responsible for thread-safe access. This draft takes no lock on the native results. The four Element Data files are otherwise as they are on rc.
  4. The native library moves forward. The cxx submodule moves from 7992af70 to 178371ac, which is 167 commits and 71 merged pull requests of 51degrees-cxx (comparison). Among them, #177 rounds a value converted from Azimuth or Declination to 3 decimal places. Between the two commits the headers the SWIG wrappers use lose only private members, and this draft does not regenerate the wrappers committed here. The native library move, below, says what else a Java user can see and why the rest changes nothing for Java.
  5. A built Pipeline keeps its own copy of the elements. PipelineDefault keeps an unmodifiable copy of the builder's list.
  6. The web filter suppresses its own errors when the Pipeline does. When PipelineFilter meets an error in its own work and isSuppressProcessExceptions() is true, it logs the error and passes the request on down the filter chain without the results. Otherwise it throws as before. The Flow Data is closed either way, including when serving the JavaScript or JSON fails with an IOException, which is then thrown as before.
  7. Background update errors are logged. The scheduled update checks and the file system watch in DataUpdateServiceDefault run through logErrors, which logs an unexpected error and keeps it inside the task. An error of the virtual machine itself is logged and thrown on.
  8. FlowData declares what it throws. FlowData gains throws clauses and @throws tags on process, get, getFromElement, getAs and the typed getters. FlowDataDefault.get(String) before processing throws IllegalStateException, which rc already does.
  9. getWhere leaves out what is not there. An element that has not run and a Property with no value are left out of the map rather than throwing. getWhere now throws IllegalStateException before processing, where it threw a plain RuntimeException only when a Property matched and gave an empty map otherwise. get by key and by type still throw for data no element added, as asking for it is the caller's mistake, and their Javadoc now says so.
  10. Defaults beside the setters. @DefaultValue is added to 83 public setters, 41 in the pipeline packages, 23 in Device Detection, 18 in IP Intelligence and 1 in the country code translation builder. Four defaults that were given wrongly are corrected in their annotations. setUpdateRandomisationMaxMillis gave 600, a number of seconds, three Cloud Request Engine endpoint defaults left out the / after the base URL, and both on-premise Pipeline builders said update on start-up defaults to enabled, where the value they fall back to, Constants.DEFAULT_UPDATE_ON_STARTUP, is false.

No public name is removed or renamed.

The native library move

Java's on-premise engines run the C and C++ code of the cxx submodule through the SWIG wrappers committed in this repository. These changes among the 71 pull requests alter what a Java user can see, and Before you upgrade lists them.

Change What a Java user sees
#136 IP Intelligence reads a client IP value only when all of it is one address. A value such as 1.2.3.4/24 or 1.2.3.4 x, or one with a part above 255, was read as far as it went and is now skipped, and a value that could not be read at all no longer stops the whole lookup.
#134 The query value of 51D_GetHighEntropyValues and 51D_StructuredUserAgent is used before the cookie value, whatever order the evidence is given in.
#184 An evidence key's prefix is found ignoring case, so evidence such as Header.User-Agent is read where it was dropped.
#146 A typed read with no value, because the matched profile holds no value for the Property or because there are too many values, gives the engine's message where Java read a null message.
#116, #115 A performance profile applied after the Concurrency keeps it, where it put back the preset's 10. No Concurrency stops an engine building, where Balanced Device Detection refused any above 22. The file pool opens at most 64 handles, and a thread that finds every handle in use waits for one rather than failing with INSUFFICIENT_HANDLES. A Device Detection value read that fails part way through now reports the failure.
#114, #135 HighPerformance with some but not all Properties named holds in memory only what they read, in both products. Balanced caches 100,000 values and 20,000 profiles, where it cached 50,000 and 10,000. The values given are the same, and the memory used changes.
#148 An engine created with no Property names keeps the first load's Property order after a data refresh, so a Property keeps its position.
#127 The Device Detection engine reads its evidence keys again when its data is refreshed.
#182 A reload holds the data set it reads, so two reloads at once can no longer free a data set still in use.
#132 Looking up a value that a Property does not have gives no value, where the C++ read on from a value it had not found.
#120 A date in a data file's header that is not a valid date gives zeros. Every data file 51Degrees publishes has valid dates.
#177 Latitude and Longitude are rounded to 3 decimal places, as item 4 of What changed says.

The rest change nothing a Java user can see, for these reasons.

  • Moved into C with the same answers, by each pull request's own tests: #121 (match metrics and matched User-Agents), #123 (the evidence key lists, in the same order), #124 (default values and value text), #128 (JavascriptHardwareProfile), #129 (type names, which that pull request says Java reads unchanged), #130 (IP Intelligence value text, which that pull request lists as unchanged for Java), #131 and #133 (property, component and profile metadata).
  • Java always applies a performance profile, Balanced by default, and a profile gives the preset's settings, so #119 (the IP Intelligence C++ configuration's starting profile) and #125 (two flags in place of the preset's shape) change nothing for it.
  • Java never turns off the processing of special evidence, which is on by default, so #126 leaves the evidence key list as it was.
  • Java gives one temporary folder, so #138 (every folder given is searched) changes nothing for it.
  • Java never calls the size functions or the memory tracker, which #160 and #180 change, nor the weighted WKT getter, which #161 changes.
  • #122 changes a SWIG interface file, and this draft does not regenerate the wrappers.
  • #113 converts each IP Intelligence value to text once, which is faster and gives the same text. #118 changes the cache's choice of shard and lowers only ASCII bytes in a substring search, with the same results for ASCII text. #139 changes a comment, and #149 changes no file.
  • The other 37 change the build, CI, examples, tests and documentation of the native library. Of these, #105, #141, #144, #145 and #179 touch files under src only so that the sources build (CMake packaging, a macro name, a file name, and a const and three includes for GCC 15), with no change to what the code does.

Before you upgrade

  1. New API. Pipeline.isSuppressProcessExceptions() is new on the public Pipeline interface, as a default method. AspectEngineBase.isCacheable(TData) is a new protected method that an engine can override to keep a result out of its cache. AspectDataBase.getWithoutWaiting(String) is a new protected method that reads a stored value without waiting for lazy loading.
  2. A cached Cloud Request Engine no longer caches a response holding a 51Did. Each such request now goes to the cloud service.
  3. getWhere throws IllegalStateException before processing. It threw a plain RuntimeException only when a Property matched, and gave an empty map otherwise. IllegalStateException is a subclass of RuntimeException.
  4. getWhere no longer throws for missing data. A caller that relied on NoSuchElementException or NullPointerException from it gets a map without those entries instead.
  5. The web filter goes on when the Pipeline suppresses process exceptions. With setSuppressProcessException(true), an error in the filter's own work is now logged and the request continues without results, where it ended the request.
  6. Latitude and Longitude are rounded to 3 decimal places when converted from Azimuth or Declination, as a number, as a string or in a WKT shape.
  7. An element added to a builder after build() no longer joins the Pipeline already built.
  8. The native library's changes in the table under The native library move. IP Intelligence skips a client IP value that is not wholly one address, the query value of the two special evidence keys wins over the cookie, evidence key prefixes match in any case, a typed read with no value gives a message, a profile keeps the Concurrency given before it, threads wait for a file handle, HighPerformance with named Properties and Balanced use different memory, an engine with no Property names keeps its Property positions across a refresh, and the Device Detection evidence keys are read again on a refresh.

Tests

Each item has a test commit followed by its fix, and each test commit was run on a runner before its fix. The two results lock tests that were here, DeviceDataHashThreadSafetyTests and IPIntelligenceDataHashThreadSafetyTests, are gone with the locks, as Changes at this head below says.

Red, run 43773302656. The run's commit is 55661b7 with only a temporary workflow added. Each of its eleven jobs checks out one test commit, builds the modules it tests with the modules they need on Java 17, runs only that commit's new tests, and fails.

Test commit Tests Result
d26dd96 CloudRequestEngineFodIdCacheTests 4 of 5 fail, "A response with a 51Did was served from the cache" for fodid, FodId and FODID, and a stale one found there
2acc1b9 PipelineSuppressProcessExceptionsTests 2 of 2 fail, NoSuchMethodException
bec88e9 PipelineImmutableTests 1 of 2 fails, expected 1 element but was 2
b4419d0 PipelineFilterErrorTests 2 of 3 fail, the IllegalStateException left doFilter
0425ec7 DataUpdateServiceBackgroundErrorTests 2 of 2 fail, "The error left the background task"
616eb15 FlowDataDeclaredExceptionsTests 5 of 5 fail, for example "process does not declare IllegalStateException, only []"
e52c0ad FlowDataAccessorErrorTests 2 of 4 fail, NoSuchElementException and NullPointerException from getWhere
2317375 BuilderDefaultValueTests in five pipeline modules each fails, listing the setters with no default
b4b0df1 BuilderDefaultValueTests in Device Detection and IP Intelligence each fails, listing the setters with no default

Red for lazy loading, run 43775828622, on 089dfbb with only a temporary workflow added. Its job checks out 8ad48e8 and runs CloudRequestEngineLazyCacheTests, and 1 of 3 fails, "process waited for the cloud service's answer".

The rules the code already met, run 43774794645 and run 43774949637. Each job checks out the commit that adds the test, breaks the rule in the code on purpose with a sed command its log shows, and runs the test, which fails.

Test commit Tests Rule broken on purpose Result
99763c0 CloudOnPremiseSubstitutionTests the cloud engine's Data Key changed to device-cloud 1 of 2 fails
f7bb902 JsonBuilderMultiPipelineTests the JSON Builder uses the first Pipeline's state for every Pipeline 1 of 1 fails, "The second Pipeline's configuration came from the first"
075ea83 FlowDataConcurrentWriteTests the Flow Data locks taken out 2 of 3 fail, "The Element Data was created twice" and 7826 of 8000 errors kept. The third, that a Pipeline without parallel elements takes no lock, still passes. Since a662e96 the same key test checks rc's add with no lock instead, as Changes at this head below says
bed28c7 FlowElementContractTests getElementDataKey given a default 1 of 3 fails
8a8f512 CacheKeyTests cache key values compared ignoring case 1 of 2 fails
f9b0f3b CloudRequestEngineCacheKeyCaseTests the cloud Evidence filter made case-sensitive 1 of 2 fails, "A key in another case was not taken as the same Evidence"
83eea04 PipelineAutoCloseTests the auto close setting ignored 1 of 2 fails, "Closing one Pipeline closed the shared element"
c334cd8 FlowDataCleanupTests closing Flow Data reads each Element Data 1 of 1 fails

Red and green with the Enterprise IP Intelligence data file, run 43777891646, on 089dfbb with only a temporary workflow added. AngleRoundingTests needs that file, the only one that holds Latitude and Longitude, so the run fetches it as the full build does and runs the test at two commits. At the test commit 7802440, still on cxx 7992af70, 2 of 3 fail, for example "Latitude 51.34891811883907 is not rounded to 3 decimal places. expected:<51.349>". At the fix 1a4236f, on cxx 178371ac, 3 of 3 pass.

Green, the builds below, in which these tests ran.

Build

Run 43775786230 built 089dfbb merged with rc (e2472ee) on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21, and every job passed. Every test class named in the tables above passed in all nine jobs with none skipped, AngleRoundingTests among them with the Enterprise IP Intelligence data file the build fetches.

Run 44419992206 built 9f27d7a, the merge of rc at ce10eaf, on the same nine configurations, and every job passed. Its workflow commit is 8c41ab4, on 9f27d7a.

Run 44451937148 built the head b39a1e2 on the same nine configurations. Its workflow commit is 7a8252d, on b39a1e2. Eight jobs passed. Windows with Java 21 passed the tests of every pipeline module, then the JVM crashed in rc's DataRefreshTests.Refresh_FromSeveralThreads_EngineStillWorks in Device Detection on-premise, with "Insufficient memory allocated for the operation" and then an access violation, so the modules after it did not run their tests in that job. They ran and passed in the other eight. The same test crashed the JVM the same way on Windows with Java 21 in two builds that keep rc's cxx 7992af70, run 43952784482 and run 43971336799, so the crash comes from rc and not from the move to 178371ac. #198 tracks it.

macOS was left out of these builds. The rc head is built with macOS before it merges into main.

Merged with rc

9f27d7a merged rc at ce10eaf, after #175 and #177 had merged, keeping the NoSuchElementException the typed key map throws in the FlowData Javadoc beside rc's cases. 8a51631 merged rc at 49bc431, which holds the parallel operation changes. Its one conflict was two imports added at the same line of PipelineBuilderBase.java, DefaultValue here and PipelineConfigurationException on rc, and both are kept.

What remains

  1. #181. The location (geo-location) builders and the developer examples' builders still have setters with no @DefaultValue.

  2. After an unexpected error in a scheduled update check, this draft logs it but does not schedule the next polling check. #170 schedules it, and a trial merge of the two heads is clean.

  3. With the deprecated lazy loading option a result is stored before it has been processed, so a response with a 51Did is kept from being served but not from being stored. The specification says the lazy loading option has no effect, and once it does nothing every result passes the check before it is stored.

  4. The locks rc already takes on the native results, in DeviceDataBaseOnPremise, IPIntelligenceDataBaseOnPremise and IPIntelligenceDataHashDefault, are not removed here. The specification's MUST NOT applies to them too, and removing them is separate work.

  5. The SWIG wrappers committed here are not regenerated from 178371ac. #122 records that the Java wrappers mark the results of ConfigIpi's getComponents, getProperties, getValues and getProfiles as owned, which its change to the SWIG interface fixes once the wrappers are regenerated. Regenerating them is separate work.

Changes at this head

Since 9f27d7a, each fix after its test, with the red of each test on a temporary branch holding one workflow commit on the test commit and building every configuration but macOS.

The green is run 44451937148, in which eight of the nine jobs passed, as Build says.

Run locally on Windows with Java 21 at b39a1e2 before pushing: FlowDataConcurrentWriteTests 3 of 3, ParallelElementDataAddTests 1 of 1, ParallelErrorsLockTests, CloudRequestEngineLazyCacheTests 4 of 4, CloudRequestEngineFodIdCacheTests 5 of 5, CloudRequestEngineCacheKeyCaseTests 2 of 2, AspectEngineCachedTests 2 of 2, AspectEngineLazyLoadTests 5 of 5, PipelineFilterErrorTests 4 of 4 and PipelineFilterTests 19 of 19 pass. The new lazy cache test and the new filter test each failed locally at their test commits.

Merging

When this squashes, the messages of the lock commits and of the same key test that later commits replace no longer describe the change, so the squash message should describe the final change only.

Head b39a1e2.

…evel fodid

caching.md line 144 at 706de1f0: "A cache MUST NOT store a result that holds such a value, and MUST NOT serve one that it finds." (CO-21)
The engine stores the fresh result it gets in its place. caching.md line 144 at 706de1f0 (CO-21).
…ache

AspectEngineBase asks isCacheable before it stores a result and before
it serves one it finds. The Cloud Request Engine answers false for a
response whose JSON has a top-level member named fodid, in any case.

caching.md line 144 at 706de1f0: "A cache MUST NOT store a result that
holds such a value, and MUST NOT serve one that it finds." (CO-21)
@jwrosewell jwrosewell added the remote-conformance Specification conformance work from the remote packs label Nov 15, 2026
exception-handling.md line 42 at 706de1f0: "This property MUST be
exposed as a public property as part of the `Pipeline` public interface
as `IsSuppressProcessExceptions` - for Pipeline wrappers such as
`WebPipeline` to be able to use it as well to make a decision on whether
the exceptions should be suppressed and logged during processing."
(CO-26)
PipelineDefault gives the setting the builder made. The interface's
default method gives the builder's default, so an implementation of
Pipeline written before this still compiles.

exception-handling.md line 42 at 706de1f0: "This property MUST be
exposed as a public property as part of the `Pipeline` public interface
as `IsSuppressProcessExceptions`" (CO-26)
…ults

Covers getValues, the property getters, the match metrics and the
device id, each started while another read is inside the native results.

thread-safety.md line 57 at 706de1f0: "Reading an Element Data instance
once the Flow Element that writes it has finished MUST therefore be safe
from several threads at once." (CO-48)

thread-safety.md line 61 at 706de1f0: "Such Element Data MUST let only
one thread at a time read from its results, for example by holding a
lock for each read" (CO-49)
…ive results

DeviceDataBaseOnPremise gives subclasses its lock as getResultsLock and
holds it while it asks whether a property is available. DeviceDataHashDefault
holds it in propertyIsAvailable, in the public getValues and while it
reads the match metrics and the device id. The lock is reentrant, so a
read that calls another read is unchanged.

thread-safety.md line 57 at 706de1f0: "Reading an Element Data instance
once the Flow Element that writes it has finished MUST therefore be safe
from several threads at once." (CO-48)

thread-safety.md line 61 at 706de1f0: "Such Element Data MUST let only
one thread at a time read from its results, for example by holding a
lock for each read" (CO-49)
Reads the weighted doubles, the weighted floats and the single number.
The free Lite file has neither property, so the test needs a paid IP
Intelligence data file and is skipped without one.

value-types.md line 99 at 706de1f0: "A value converted from Azimuth or
Declination MUST be given to the caller rounded to 3 decimal places,
whether it is given as a number, as a string or as a point in a WKT
shape." (CO-58)
From 7992af70. 51degrees-cxx #177 (a1c0504d) gives a number stored as an
angle rounded to the decimal places of its text, 3 by default, so the
weighted Latitude and Longitude lists come rounded. The headers the
wrappers use lose only private members in this range. cxx no longer has
its device-detection-data and ip-intelligence-data folders, as the
README expected.

value-types.md line 99 at 706de1f0: "A value converted from Azimuth or
Declination MUST be given to the caller rounded to 3 decimal places,
whether it is given as a number, as a string or as a point in a WKT
shape." (CO-58)
… builder

conceptual-overview.md line 118 at 706de1f0: "A Pipeline instance MUST be
immutable. In other words, once created, it cannot be changed by adding
new elements, removing old ones, etc." (CO-12)
PipelineDefault kept the builder's list, so an element added to the
builder after build joined the Pipeline already built. It now keeps an
unmodifiable copy.

conceptual-overview.md line 118 at 706de1f0: "A Pipeline instance MUST be
immutable. In other words, once created, it cannot be changed by adding
new elements, removing old ones, etc." (CO-12)
… interface

For Device Detection and IP Intelligence. Passes as the code is, so no
fix follows.

README.md line 95 at 706de1f0: "Where both Cloud and On-Premise variants
of an Engine are available, they MUST arrange that the Properties and
values produced are compatible with each other so that the Engines can
be substituted in the Pipeline without alteration to the consuming
application." (CO-4)
…iguration

Passes as the code is, so no fix follows.

multi-pipeline-elements.md line 17 at 706de1f0: "If this feature is
enabled, then all Flow Elements that maintain any such state information
MUST ensure that they have the means to keep track of isolated instances
of this state for each Pipeline to which they are added and use the
correct state information when processing." (CO-6)
…n it must

Two threads adding Element Data under one key create it once, errors
added from eight threads are all kept, and without parallel elements a
write takes no lock. Passes as the code is, so no fix follows.

parallel-processing.md line 22 at 706de1f0: "Flow Data MUST be capable
of enforcing thread-safe writes to data structures while also
maintaining performance when it is not needed." (CO-7)
…Data Key

Passes as the code is, so no fix follows.

conceptual-overview.md line 73 at 706de1f0: "Each Element MUST:" (CO-11)
…red by case

Both pass as the code is, so no fix follows.

caching.md line 59 at 706de1f0: "All Flow Elements MUST advertize the
Evidence keys that they make use of." (CO-14)

caching.md line 82 at 706de1f0: "Comparison of Evidence values MUST be
case-sensitive." (CO-17)
The Cloud Request Engine with a cache serves the second request from the
cache when only the case of the key differs, and goes to the service
when only the case of the value differs. Passes as the code is, so no
fix follows.

caching.md line 71 at 706de1f0: "Comparison of Evidence keys MUST be
case-insensitive." (CO-16)
…tions are suppressed

exception-handling.md line 73 at 706de1f0: "The implementor MUST ensure
that exceptions/errors that happen in these areas are handled correctly
in order to avoid the API taking down the customer's production
website." (CO-28)
…t to

When the Pipeline suppresses process exceptions, an error in
PipelineFilter's own work (the not modified check, processing, the
UA-CH headers or serving the JavaScript and JSON) is logged and the
request goes on down the filter chain without the results. Otherwise
it is thrown, as before. The FlowData is closed either way, and only if
processing stored one.

exception-handling.md line 73 at 706de1f0: "The implementor MUST ensure
that exceptions/errors that happen in these areas are handled correctly
in order to avoid the API taking down the customer's production
website." (CO-28)
…d kept in the task

exception-handling.md line 83 at 706de1f0: "When errors do occur, they
will be logged and MUST NOT cause the process to crash." (CO-29)
The scheduled update checks and the file system watch run through
logErrors, which logs an error the task did not expect and keeps it in
the task. Before, the executor kept it in a future nobody read, so it
was never logged. An error of the virtual machine itself is logged and
thrown on. Rescheduling after such an error is left as it is.

exception-handling.md line 83 at 706de1f0: "When errors do occur, they
will be logged and MUST NOT cause the process to crash." (CO-29)
…t open

Passes as the code is, so no fix follows.

resource-cleanup.md line 28 at 706de1f0: "However, if the multi Pipeline
Elements feature is implemented then there MUST be an option to disable
this behavior." (CO-39)
Passes as the code is, so no fix follows.

resource-cleanup.md line 84 at 706de1f0: "Element Data instances that do
not require cleanup MUST be left alone in order for result caching to
work correctly." (CO-45)
…only those

exception-handling.md line 17 at 706de1f0: "In all cases, standard
language features SHOULD be used to let IDEs and users know what
exceptions/errors a particular function can throw." (CO-24)
process, get, getFromElement, getAs and the getAs typed getters name in
their throws clauses and @throws tags the unchecked exceptions they
throw for a mistake by the caller. get(String) before processing now
throws IllegalStateException, as get(TypedKey) does, rather than a plain
RuntimeException. A caller that catches RuntimeException is unchanged.

exception-handling.md line 17 at 706de1f0: "In all cases, standard
language features SHOULD be used to let IDEs and users know what
exceptions/errors a particular function can throw." (CO-24)
An element that did not run, or a property with no value, is left out
of getWhere's map. Evidence that is missing or of another type gives no
value from tryGetEvidence.

exception-handling.md line 57 at 706de1f0: "The general guidance is that
these functions and accessors SHOULD only throw errors/exceptions if the
failure is likely due to something that the API user has done wrong."
(CO-27)
…et throws

getWhere left a NullPointerException or NoSuchElementException to the
caller when an element had not run or a property had no value, which is
not the caller's mistake. It now leaves both out, and still throws
IllegalStateException before processing. The Javadoc of get no longer
promises null for missing data: get by key throws NoSuchElementException
and get by type throws PipelineDataException, as FlowDataTests and the
TypedKeyMap tests expect, and both are declared.

exception-handling.md line 57 at 706de1f0: "The general guidance is that
these functions and accessors SHOULD only throw errors/exceptions if the
failure is likely due to something that the API user has done wrong."
(CO-27)
… value

For the Pipeline builder, the engine builders and the data file
configuration builder, the Cloud Request Engine builder, the usage
sharing and 51Degrees Pipeline builders, and the translation engine
builder. The JavaScript Builder's builder already gives every default.

reference-implementation-notes.md line 79 at 706de1f0: "In all cases,
there SHOULD be some mechanism to specify the default values alongside
configuration property setters." (CO-65)
…ne packages

@DefaultValue on the 41 setters that had none, taken from the constant
or field the builder or its configuration falls back to, or saying who
sets it where it is not a value here. The randomisation setters in
milliseconds gave the default in seconds, and three Cloud Request
Engine endpoint defaults left out the slash after the base URL, so those
are corrected.

reference-implementation-notes.md line 79 at 706de1f0: "In all cases,
there SHOULD be some mechanism to specify the default values alongside
configuration property setters." (CO-65)
The same check as for Device Detection, for getValues and the property
getters of IPIntelligenceDataHashDefault.

thread-safety.md line 61 at 706de1f0: "Such Element Data MUST let only
one thread at a time read from its results, for example by holding a
lock for each read" (CO-49, and CO-48 at line 57)
…ve results

IPIntelligenceDataBaseOnPremise gives subclasses its lock as
getResultsLock and holds it while it asks whether a property is
available. IPIntelligenceDataHashDefault holds it in propertyIsAvailable
and in the public getValues, whose body moves to readValues unchanged.

thread-safety.md line 61 at 706de1f0: "Such Element Data MUST let only
one thread at a time read from its results, for example by holding a
lock for each read" (CO-49, and CO-48 at line 57)
… its default

For the on-premise Pipeline and engine builders of both products, and
the country code translation engine builder.

reference-implementation-notes.md line 79 at 706de1f0: "In all cases,
there SHOULD be some mechanism to specify the default values alongside
configuration property setters." (CO-65)
…ce builder setter

@DefaultValue on the 42 setters that had none, from the constants the
engine builders fall back to. Both on-premise Pipeline builders said
update on startup defaults to enabled, but an unset value falls back to
Constants.DEFAULT_UPDATE_ON_STARTUP, which is false, so their Javadoc
now says disabled.

reference-implementation-notes.md line 79 at 706de1f0: "In all cases,
there SHOULD be some mechanism to specify the default values alongside
configuration property setters." (CO-65)
…d response

Lazy loading means process returns at once and the work is done on another thread. The check that keeps a 51Did out of the cache (caching.md line 144 at 706de1f0: "A cache MUST NOT store a result that holds such a value, and MUST NOT serve one that it finds.") must not make process wait for the answer. The test holds the answer back and reads the processing thread's stack, so no result depends on timing. Two more tests check that a lazy response is still cached, and that one with a fodid is not.
isCacheable read the cloud response through the getter that waits for lazy loading, so a cached lazy Cloud Request Engine waited in process for the cloud service. It now reads the stored response without waiting (AspectDataBase.getWithoutWaiting). With lazy loading the result is offered to the cache by the processing task once processing has finished, as the response is not there to check before then. Without lazy loading nothing changes. The check still answers caching.md line 144 at 706de1f0: "A cache MUST NOT store a result that holds such a value, and MUST NOT serve one that it finds."
jwrosewell and others added 11 commits November 18, 2026 23:14
Resolves three conflicts with rc.

FlowData.java: get(TypedKey) and get(String) throw NoSuchElementException
when no element added data under the key, as the typed key map does, so
this branch's javadoc stays. process() keeps rc's AggregateException, and
the getAs methods keep rc's missing property case, which
getMetaDataForProperty throws.

DeviceDataHashDefault and IPIntelligenceDataHashDefault: rc's shortcut for
a property the engine has an index for stays, and the native name lookup
after it holds the results lock, as this branch does. The IP Intelligence
reads use the base class's results lock in place of rc's second lock
object, so every read of the native results holds the same lock.
rc now holds the parallel operation changes (Element Data added with one
compare and swap and no lock, usage sharing last, the web Flow Data used by
one thread, members' errors added under a lock) and the evidence key case,
cloud response warning and example fixes.

One conflict, in PipelineBuilderBase.java, where this branch imports
DefaultValue and rc imports PipelineConfigurationException at the same line.
Both are used, so both imports are kept. Every other file merged without a
conflict.
The specification says an Engine, its results instances and the Element
Data that reads from them must not take a lock to make those reads safe for
several threads, and that the caller is responsible for thread-safe access.

So the results lock this branch added to the Device Detection and IP
Intelligence on-premise Element Data goes, with its getResultsLock()
accessor and the two test classes that required it. The four files are as
they are on rc, and the two base classes now say that an instance is not
safe to use from several threads at once and that the caller is
responsible.

https://github.com/51Degrees/specifications/blob/d79bbe3b2aef86273570742f6b660f29bfe72816/pipeline-specification/features/thread-safety.md#L196-L221
rc adds Element Data with no lock, by one compare and swap, so the test
that needed a second writer to wait for the first fails once rc is merged.
The specification says a Flow Element adds its Element Data with one
compare and swap and readers take no lock, and two Flow Elements that write
one Element Data instance must not share a parallel group.

So the same key test now checks what rc does. Both writers reach the
factory while neither has finished, and both get the one instance kept.
The class comment quotes the current thread safety page.

https://github.com/51Degrees/specifications/blob/d79bbe3b2aef86273570742f6b660f29bfe72816/pipeline-specification/features/thread-safety.md#L88-L115
The specification says a Flow Element that hands data to another thread
must copy it during processing, and must not read the Flow Data from that
thread afterwards. Making the cache key reads the Flow Data's Evidence, so
the cache write belongs on the thread that called process.

This test records the thread of each cache write. It fails until the
processing task stops writing to the cache.

https://github.com/51Degrees/specifications/blob/d79bbe3b2aef86273570742f6b660f29bfe72816/pipeline-specification/features/thread-safety.md#L32-L40
With the deprecated lazy loading option the processing task stored the
result in the cache once it was processed, and making the cache key read
the Flow Data's Evidence from that task after process had returned. The
specification says a Flow Element must not read the Flow Data from another
thread afterwards.

So the lazy path stores the result on the processing thread, as it did
before this branch, and the task only processes. The result is not yet
processed when it is stored, so the check before serving is what keeps a
response with a 51Did from being served. The lazy 51Did test now says
served, and the comment on isCacheable says which check applies when.

https://github.com/51Degrees/specifications/blob/d79bbe3b2aef86273570742f6b660f29bfe72816/pipeline-specification/features/thread-safety.md#L32-L40
https://github.com/51Degrees/specifications/blob/d79bbe3b2aef86273570742f6b660f29bfe72816/pipeline-specification/features/caching.md#L151-L156
Serving the 51Degrees JavaScript or JSON can fail with an IOException, for
example when the client has gone away. The specification says the web
integration disposes of the Flow Data when the request ends, which
releases its Element Data and native results, and that errors in the web
integration must not take down the website.

This test makes serving the JavaScript throw an IOException and checks
that the failure is thrown and the Flow Data is still closed. It fails
until the filter closes the Flow Data on that path.
The web filter caught only runtime errors from processing and serving, so
an IOException from writing the 51Degrees JavaScript or JSON left the
filter without closing the request's Flow Data. The specification says the
web integration disposes of the Flow Data when the request ends.

The filter now closes the Flow Data before throwing the IOException, and a
failure to close is added to it as a suppressed exception.

https://github.com/51Degrees/specifications/blob/d79bbe3b2aef86273570742f6b660f29bfe72816/pipeline-specification/features/web-integration.md#L27-L28
Two conflicts. In FlowDataDefault.getWhere this branch's version is kept,
which skips an element with no data and a property with no value, and the
property name is lowered in the root locale as rc now does. In
PipelineBuilderBase setAutoCloseElements keeps this branch's DefaultValue
and gains rc's AlternateName, and both imports stay.
…etters and this work's default value on setCache in the Hash engine builder
@jwrosewell
jwrosewell marked this pull request as ready for review November 21, 2026 16:19
@jwrosewell
jwrosewell merged commit 8d29d01 into rc Nov 21, 2026
9 checks passed
@jwrosewell
jwrosewell deleted the remote/java-13 branch November 21, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

remote-conformance Specification conformance work from the remote packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants