Repository navigation
Add the Derived Property Element and its example - #156
Merged
Merged
Conversation
Adds the tests for a new pipeline.derivedproperty module and a new developer example, before the code they test. The module is the Derived Property Element of the specification, which produces new properties from the properties other elements in the Pipeline produce, following rules written in a YAML or JSON script. The derived-properties repository is added as the submodule pipeline/derived-properties, at cc30638. Its scripts folder holds the built-in scripts the package ships, and its tests folder holds the case files and rejection cases every language runs. SharedCasesTests runs both, and BuiltInScriptTests fails when BuiltInScript and the scripts folder differ. The other tests cover reading YAML by the YAML 1.2 core schema, so yes stays a string, the conversion of source values, the Pipeline check, the no value message, the logging, the metadata and building from a configuration file. ExampleTests and FindOutMoreTests run the example. The two modules are added to pipeline/pom.xml, the developer examples pom and ci/packages.json, and SnakeYAML is named in the dependency management of pipeline/pom.xml. The tests do not compile until the code follows.
pipeline.derivedproperty is the Derived Property Element of the specification. It is a Flow Element rather than an Aspect Engine, holds no data file, makes no network request and accepts no evidence. Every instance writes into the Element Data under the key derived, as every Translation Engine shares translation. DerivedPropertyElementBuilder takes scripts shipped in the package (BuiltInScript), files on disk with wildcards in the file name, and text from code, and a configuration file names it as DerivedPropertyElement with the Scripts and ScriptFiles build parameters. Each script is parsed with the compose API of SnakeYAML, so every node keeps its line, and plain scalars are resolved by the YAML 1.2 core schema. The validator follows tools/validate.mjs in the derived-properties repository, with the same paths and messages, and reports every fault in every script at once. Per request the element reads each source property once, converts it to the type the script infers without coercing loosely, then runs the checks and the rules. Where a source property is not available the output has no value, using the existing AspectPropertyValue mechanism, and the message names every property that could not be read, in the wording of tools/evaluate.mjs. A request with no Element Data under a key a script reads is logged once per script. When the Pipeline adds the element it checks that some element lists each source property, without judging where that element sits, and that no two Derived Property Elements produce the same property. The developer example builds one Pipeline in code, with a script held as text and a script read from a file beside the example, and the same Pipeline from a configuration file, and shows the message a caller sees when a source property is not available. It ends with a Find out more section.
…ccepts
The tests made each pattern with Path.resolve("*.yaml"), and Windows
refuses a path that holds a wildcard with InvalidPathException, so three
DerivedPropertyElementConfigurationTests failed in every Windows job of
full build 43667464820. The element itself splits a pattern with
File.getName and File.getParentFile and was not affected.
jwrosewell
added a commit
that referenced
this pull request
Nov 15, 2026
…its verify (#161) Tests only. `DataUpdateServiceTests.DataUpdateService_Register_AutoUpdateConfiguredInterval` fails about once in 600 runs, and it failed `Ubuntu_Java_17` in full build [43667464820](https://github.com/51Degrees/51degrees-java/actions/runs/43667464820) of [#156](#156), which changes nothing in `pipeline.engines`. This fixes the cause rather than leaving it to a re-run, which is the rule for a test that fails on timing. [#122](#122), which is open and not yet merged, adds that rule to AGENTS.md.
…for a package never released
…nslation already uses
jwrosewell
added a commit
that referenced
this pull request
Nov 16, 2026
…ith no key to the drafts that carry them The Derived Property Element and its example are #156's, Comparer on the free data file is #158's, and every cloud example ending cleanly with no resource key is #163's, so each file this draft changed only for those is put back to rc's version. The cloud cases of the no-settings tests go with them, as #163's tests cover each cloud example run with no key.
The Derived Property Element page, Key Considerations 6, says each Derived Property Element writes into the same Element Data, so two of them MUST NOT be members of the same parallel group. Nothing checks this, so the documentation says it. DerivedPropertyData said the opposite, that elements in a parallel group write at the same time, and now says they must not. DerivedPropertyElement, its builder and the pipeline README say it too, and the links to the page are at the specification commit that has that sentence.
The test that shows two Derived Property Elements writing into the one Element Data put them in the same parallel group, and its comment said that is how they work. The specification says the opposite. Every Derived Property Element writes into the same Element Data, so two of them must not be members of the same parallel group. The test now adds the two elements one after the other, keeps its two assertions on the one Element Data, and processes one request, because nothing runs at once any more. It is renamed to say so, and its comment says why the two elements are never put in one parallel group. Specification, derived-property-element.md lines 242 to 245 and parallel-processing.md lines 48 to 53 at d79bbe3b.
JsText.isWhitespace held the line separator, the paragraph separator and the byte order mark as raw characters inside char literals. A reader cannot see them, and an editor or a change of encoding can drop them without a trace. They are now named constants written as hexadecimal numbers, as VERTICAL_TAB already is, so the file holds only ASCII. The values are the same, so nothing the element does changes.
DerivedPropertyElementConfigurationTests linked the specification page at 706de1f0, where every other link in the package is at 52e9b047. The page is the same at 52e9b047 and at d79bbe3b, the tip of the specification's rc, so one commit now covers every link.
Where the element data can give the string it stored for a property, through ProvidesValuesAsString as the on-premise engines can, a script that compares the property with text now reads that string. The typed accessor answers in its own type, so the Unknown that device detection stores for a property the 51Degrees JavaScript has not yet measured read as False, and HumanConfidence could never answer Unknown from the on-premise engine. A script that compares the property with a boolean or a number still reads the typed value. This is what .NET does with IProvidesValuesAsString.
They first ship in the 4.6 release, which is 4.6.2 in every repository.
Tracy51D
added a commit
that referenced
this pull request
Nov 20, 2026
…Elements, which #202's every-element check asks for
jwrosewell
marked this pull request as ready for review
November 21, 2026 16:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the Derived Property Element to Java, as .NET and Rust have it, with its example. Closes #110.
The element produces new property values from property values other Flow Elements have already produced, following rules written in a script, YAML or JSON, and writes them into the Element Data under the key
derived. It holds no data file, makes no request over the network and reads no Evidence. The scripts and their test cases are shared by every language in the derived-properties repository, which this repository now holds as the submodulepipeline/derived-propertiesat cc306386, so Java gives the same answers and the same faults as the other languages, case for case.What the specification says
The element is specified in derived-property-element.md at 706de1f0, which states its package behaviour without RFC 2119 keywords. The requirement on each language is in required-examples.md lines 30 to 51:
Since then Key Considerations 6, at 52e9b047 says
and parallel-processing.md lines 48 to 53 at d79bbe3b says
So nothing checks it, the package documents it, and its own tests never build such a Pipeline.
What changed
pipeline.derivedproperty(pipeline/pipeline.derivedproperty).DerivedPropertyElementBuildertakes the built-in scripts by name (addScript(BuiltInScript), orsetScriptswith a comma separated list, as a configuration file gives it), a script file (addScriptFile, orsetScriptFileswith a comma separated list whose file names may hold wildcards), and a script as text (addScript(String, ...)). Each script is parsed and checked when the element is built, so a script that is not valid stops the build withDerivedScriptValidationException, which names the fault. The built-in scripts,HumanConfidenceandIsGdpr, are packaged from the submodule'sscriptsfolder as resources. Per request the element reads each source property once, runs each script, and gives each derived property either its value or no value with a message that names what was missing.ProvidesValuesAsString, as the on-premise Device Data does,SourceValues.readtakes the value fromgetStoredValueAsStringfor a property the script infers asstring, and fromgetfor every other type. The typed accessor answers in its own type, and device detection's ownStoredValueAsStringTests.typedAccessor_IsUnchangedshows a property stored asUnknownreadingfalsethrough it, so without this a script could not see theUnknownthat device detection stores for a property the 51Degrees JavaScript has not yet measured.HumanConfidencecomparesdevice.IsVisibleanddevice.HasWebDriverwith"Unknown"in this way. This is what the .NET element does withIProvidesValuesAsString.pipeline.developer-examples.derivedproperty(Main.java). Its source properties come from a small element written in the example,SessionElement, so it needs no data file, no key and no network. It shows the four things the specification lists, a script held as text, a script read from ReaderEngagement.yaml beside it, the same Pipeline built from derived-property.xml, and a request without a source property, and ends with a Find out more section. It reads none of the shared scripts.DerivedPropertyElement, its builder andDerivedPropertyData, andpipeline/README.md, say so, and the links to the page are at 52e9b047 (10ee0ea8).DerivedPropertyDataused to say elements in a parallel group write at the same time. The one test with two Derived Property Elements that both write adds them one after the other (e33d0df5).pipeline/pom.xmllists the module and putsorg.snakeyaml:snakeyaml-engine2.7 in its dependency management, the YAML 1.2 readerpipeline.translationalready uses, which the package reads its scripts with (commit 6323472, in place of a second YAML library).ci/packages.jsonlists both modules, published as the other developer examples are, andpipeline/README.mdand AGENTS.md describe the package and the submodule.Tests
Tests first, in 9995692, the code in dcbab24.
SharedCasesTestsruns every case file and every rejection case in the submodule'stestsfolder, the cases every language runs.DerivedPropertyElementTests,DerivedScriptParserTests,SourceValuesTests,BuiltInScriptTestsandDerivedPropertyElementConfigurationTeststest the element, the parser, the reading of source values, the packaged scripts and the configuration file route.SourceValuesTestshas three tests for the stored string, being that a script comparing with text reads the storedUnknownwhilst one comparing with a boolean reads the typedfalse, that a stored string with no value gives its own message, and that Element Data without stored strings is still read throughget. Two of the three fail without the change inSourceValues, which was checked by running them against the previous head'sSourceValues.java.ExampleTestsruns it and checks the four parts, andFindOutMoreTestschecks its closing section and its tagged pages.Red 43620956485 ran the test commit plus a throwaway workflow on JDK 17 and JDK 8, and the tests did not compile, as the classes do not exist before the code. Green 43620965905 ran the code commit plus the same workflow, and on both JDKs the module's 213 tests passed, 167 of them the shared cases, as did the example's tests.
The first full build, 43667464820, failed in every Windows job, because
DerivedPropertyElementConfigurationTestsbuilt its file patterns withPath.resolve("*.yaml"), which Windows refuses. c9693a95 builds them as text. The element itself splits a pattern withFile.getNameandFile.getParentFile, so it was not affected. ItsUbuntu_Java_17job failed on a race in a data update test ofpipeline.engines, which this draft does not touch and #161 fixes.Full build
Run 43909692444 built a05afaa merged with
rc81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21. Configure and all nine Build and Test jobs passed, andSharedCasesTestspassed in all nine, read from each job's log. Each of its tenSetting 'PullRequestSha'lines names a05afaa, and its Configure job printed "Left out 6 macOS entries". The run completed with success.That build is of a05afaa. The four commits after it touch only
pipeline.derivedpropertyandpipeline/README.md, so the tests of what they touch are the package's and its example's. Run 44468789796 ontmp/fixconf-java-156-green2(workflow commit 898b166 on 20510b8) builds and testspipeline.derivedpropertyand its example at the head on Java 17. It passed, 213 of 213 tests inpipeline.derivedproperty, 167 of them the shared cases, and 2 of 2 in the example. Before it, run 44452193816 (workflow commit 8374af7 on 10ee0ea) did the same at 10ee0ea and passed, 213 of 213 and 2 of 2. A local run at the head, on Windows with Java 21, passed the same 213 and 2 tests.No CI has run at the head, 99cec84, or at 1e386ac before it. The Pull Requests workflow does not build a draft, because common-ci's
steps/get-pull-requests.ps1selects only pull requests that are not drafts, and no temporary workflow branch was made for either commit. The head was built and tested locally on Windows 11 with JDK 21.0.11 and the repository's Maven wrapper. Thepipelinelibraries it needs were built from the head first, with./mvnw -B -o -pl pipeline/pipeline.derivedproperty -am install -DskipTests, and thenwhich passed 216 of 216 tests in
pipeline.derivedproperty, being the 213 above and the three newSourceValuesTests, 167 of them the shared cases, and 2 of 2 in the example.Get-PackagesJsonGapsinci/packages.ps1found no gaps,ci/check-root-locale.ps1reported "Every case change in 567 main source files names its locale.", and common-ci'sscripts/utm-lint.ps1reported the repository clean.An earlier full build, run 43673959199, passed on c9693a9, before 0cca79f, 6323472 and the merge of
rc.macOS was left out of this draft's full build. The
rchead is built with macOS before it merges intomain.With the Enterprise data files
At 99cec84 the built in
HumanConfidencewas also run end to end, through a small program outside the repository, in a Pipeline of the on-premise Device Detection engine with the Enterprise data file of 19 November 2026, the on-premise IP Intelligence engine with the Enterprise IP Intelligence data file of the same date, and the Derived Property Element. Both engines were built from this branch, native code included, on Windows with JDK 21. Each request carried a User-Agent and a client IP address that the IP Intelligence data file gives aHumanProbabilityof 10 and aConnectionTypeofBroadband, except the crawler, which carried Google's. Where the 51Degrees JavaScript is taken to have run, the request also carried the cookies51D_IsVisibleand51D_HasWebDriver, which are what the JavaScript in the data file'sIsVisibleJavaScriptandHasWebDriverJavaScriptproperties writes, with the valuesTrue,FalseorUnknown.On the first request, with no cookies, the Device Data gave
IsVisibleandHasWebDriverasfalsethrough the typed accessor and asUnknownthroughgetStoredValueAsString. The same requests were then run withSourceValues.javaas it was at 20510b8, before 1e386ac.51D_IsVisible=True,51D_HasWebDriver=False51D_IsVisible=True,51D_HasWebDriver=True51D_IsVisible=Unknown,51D_HasWebDriver=FalseSo the two requests on which the window was never measured answer
Unknownwith 1e386ac, and answeredMediumbefore it. The other four answers are the same either way. The example and the package's tests need no data file, and no test in either module skips without the Enterprise data file, so nothing more was run with it.What it releases
pipeline.derivedpropertyandpipeline.developerexamples.derivedpropertyare new packages, released at their first version. The change topipeline/pom.xmlreleases thepipelineparent with the packages that depend on it. Both new entries inci/packages.jsongive aFirstVersion, which the release plan needs for a package that has never been released (commit 0cca79f). The shared release plan script reads it (steps/release-plan.ps1 lines 173 to 231).Base branch
rc81e3d4a is merged into the branch at a05afaa, with no conflict, andrc4ff58d8 is merged into the branch at the head, 99cec84, with no conflict. That merge bringsProvidesValuesAsString, which 1e386ac uses and whichpipeline.enginesgained onrcafter a05afaa, so the branch builds on its own as well as merged intorc.Changes at this head
Two commits follow the previous head, 20510b8.
SourceValuesTeststhat Tests describes. It changesSourceValues.javaandSourceValuesTests.javaonly, and its results are under Full build.rc4ff58d8, as Base branch says.Before it, three commits followed 10ee0ea, which made the documentation say that two Derived Property Elements must not share a parallel group.
DerivedPropertyElementTeststhat runs two Derived Property Elements together. It put them in the same parallel group, with a comment saying they write the one Element Data at the same time, which is the Pipeline that derived-property-element.md lines 242 to 245 and parallel-processing.md lines 48 to 53 forbid. It now adds them one after the other, keeps its two assertions on the one Element Data, and is namedtwoElementsOneAfterTheOtherShareTheElementData, with a comment saying why the two are never put in one group. It is the test that shows two Derived Property Elements in one Pipeline both giving values, which the page's lines 40 to 43 allow. The test is one this draft adds, not a released test, and it passed before the change as it does after, so there is no red to show.JsText.isWhitespaceas constants written in hexadecimal, asVERTICAL_TABis, where they were raw characters a reader cannot see and an editor can drop. The values are the same, so nothing the element does changes.DerivedPropertyElementConfigurationTestsfrom 706de1f0 to 52e9b047, where every other link in the package is. The page is the same at 52e9b047 and at d79bbe3b, the specification'src.The green of those three is run 44468789796, as Full build says, with the local run beside it.
The red and green of the tests first commit are under Tests, and the script that reads
FirstVersionis linked under What it releases.What remains
HumanConfidencenames, as With the Enterprise data files shows.