Skip to content

Add the Derived Property Element and its example - #156

Merged
jwrosewell merged 15 commits into
rcfrom
feature/derived-property-element
Nov 21, 2026
Merged

jwrosewell merged 15 commits into
rcfrom
feature/derived-property-element

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Adds the Derived Property Element to Java, as .NET and Rust have it, with its example. Closes #110.

The element produces new property values from property values other Flow Elements have already produced, following rules written in a script, YAML or JSON, and writes them into the Element Data under the key derived. It holds no data file, makes no request over the network and reads no Evidence. The scripts and their test cases are shared by every language in the derived-properties repository, which this repository now holds as the submodule pipeline/derived-properties at cc306386, so Java gives the same answers and the same faults as the other languages, case for case.

What the specification says

The element is specified in derived-property-element.md at 706de1f0, which states its package behaviour without RFC 2119 keywords. The requirement on each language is in required-examples.md lines 30 to 51:

Every reference implementation MUST provide one example for the Derived Property Element. It MUST NOT need a resource key, a data file or a network connection, so that it runs anywhere, and it MUST NOT depend on the content of the shared script repository, because a test in a language repository that read a script from that repository would break whenever a script changed.

The example MUST show all four of the following. [...]

  1. A script held in the example itself and passed to the builder as text, producing a value from source Properties an Element in the example supplies.
  2. A script read from a file beside the example.
  3. The same Pipeline built from a configuration file rather than from code, naming the builder and passing the script through build parameters.
  4. What the caller sees when a source Property named by the script is not available on the request, which is a Property with no value and a message naming what was missing. This case MUST be shown rather than only described, [...]

Since then Key Considerations 6, at 52e9b047 says

Multiple Derived Property Elements can exist in one Pipeline and each one writes into the same Element Data, so two of them MUST NOT be members of the same parallel group,

and parallel-processing.md lines 48 to 53 at d79bbe3b says

Two Flow Elements that write one Element Data instance, such as two Engines with the same key or two Derived Property Elements, MUST NOT be members of the same group, because both would write that instance at once. Whoever builds the Pipeline keeps them apart. Implementations do not check for it.

So nothing checks it, the package documents it, and its own tests never build such a Pipeline.

What changed

  • A new package, pipeline.derivedproperty (pipeline/pipeline.derivedproperty). DerivedPropertyElementBuilder takes the built-in scripts by name (addScript(BuiltInScript), or setScripts with a comma separated list, as a configuration file gives it), a script file (addScriptFile, or setScriptFiles with a comma separated list whose file names may hold wildcards), and a script as text (addScript(String, ...)). Each script is parsed and checked when the element is built, so a script that is not valid stops the build with DerivedScriptValidationException, which names the fault. The built-in scripts, HumanConfidence and IsGdpr, are packaged from the submodule's scripts folder as resources. Per request the element reads each source property once, runs each script, and gives each derived property either its value or no value with a message that names what was missing.
  • The stored string is read where a script compares a property with text (1e386ac9). Where the Element Data implements ProvidesValuesAsString, as the on-premise Device Data does, SourceValues.read takes the value from getStoredValueAsString for a property the script infers as string, and from get for every other type. The typed accessor answers in its own type, and device detection's own StoredValueAsStringTests.typedAccessor_IsUnchanged shows a property stored as Unknown reading false through it, so without this a script could not see the Unknown that device detection stores for a property the 51Degrees JavaScript has not yet measured. HumanConfidence compares device.IsVisible and device.HasWebDriver with "Unknown" in this way. This is what the .NET element does with IProvidesValuesAsString.
  • A developer example, pipeline.developer-examples.derivedproperty (Main.java). Its source properties come from a small element written in the example, SessionElement, so it needs no data file, no key and no network. It shows the four things the specification lists, a script held as text, a script read from ReaderEngagement.yaml beside it, the same Pipeline built from derived-property.xml, and a request without a source property, and ends with a Find out more section. It reads none of the shared scripts.
  • Two Derived Property Elements must not share a parallel group. The documentation of DerivedPropertyElement, its builder and DerivedPropertyData, and pipeline/README.md, say so, and the links to the page are at 52e9b047 (10ee0ea8). DerivedPropertyData used to say elements in a parallel group write at the same time. The one test with two Derived Property Elements that both write adds them one after the other (e33d0df5).
  • Wiring. pipeline/pom.xml lists the module and puts org.snakeyaml:snakeyaml-engine 2.7 in its dependency management, the YAML 1.2 reader pipeline.translation already uses, which the package reads its scripts with (commit 6323472, in place of a second YAML library). ci/packages.json lists both modules, published as the other developer examples are, and pipeline/README.md and AGENTS.md describe the package and the submodule.

Tests

Tests first, in 9995692, the code in dcbab24.

  • SharedCasesTests runs every case file and every rejection case in the submodule's tests folder, the cases every language runs.
  • DerivedPropertyElementTests, DerivedScriptParserTests, SourceValuesTests, BuiltInScriptTests and DerivedPropertyElementConfigurationTests test the element, the parser, the reading of source values, the packaged scripts and the configuration file route.
  • SourceValuesTests has three tests for the stored string, being that a script comparing with text reads the stored Unknown whilst one comparing with a boolean reads the typed false, that a stored string with no value gives its own message, and that Element Data without stored strings is still read through get. Two of the three fail without the change in SourceValues, which was checked by running them against the previous head's SourceValues.java.
  • The example's ExampleTests runs it and checks the four parts, and FindOutMoreTests checks its closing section and its tagged pages.

Red 43620956485 ran the test commit plus a throwaway workflow on JDK 17 and JDK 8, and the tests did not compile, as the classes do not exist before the code. Green 43620965905 ran the code commit plus the same workflow, and on both JDKs the module's 213 tests passed, 167 of them the shared cases, as did the example's tests.

The first full build, 43667464820, failed in every Windows job, because DerivedPropertyElementConfigurationTests built its file patterns with Path.resolve("*.yaml"), which Windows refuses. c9693a95 builds them as text. The element itself splits a pattern with File.getName and File.getParentFile, so it was not affected. Its Ubuntu_Java_17 job failed on a race in a data update test of pipeline.engines, which this draft does not touch and #161 fixes.

Full build

Run 43909692444 built a05afaa merged with rc 81e3d4a on nine configurations, Ubuntu x64 with Java 8, 11, 17 and 21, Ubuntu arm64 with Java 17, and Windows with Java 8, 11, 17 and 21. Configure and all nine Build and Test jobs passed, and SharedCasesTests passed in all nine, read from each job's log. Each of its ten Setting 'PullRequestSha' lines names a05afaa, and its Configure job printed "Left out 6 macOS entries". The run completed with success.

That build is of a05afaa. The four commits after it touch only pipeline.derivedproperty and pipeline/README.md, so the tests of what they touch are the package's and its example's. Run 44468789796 on tmp/fixconf-java-156-green2 (workflow commit 898b166 on 20510b8) builds and tests pipeline.derivedproperty and its example at the head on Java 17. It passed, 213 of 213 tests in pipeline.derivedproperty, 167 of them the shared cases, and 2 of 2 in the example. Before it, run 44452193816 (workflow commit 8374af7 on 10ee0ea) did the same at 10ee0ea and passed, 213 of 213 and 2 of 2. A local run at the head, on Windows with Java 21, passed the same 213 and 2 tests.

No CI has run at the head, 99cec84, or at 1e386ac before it. The Pull Requests workflow does not build a draft, because common-ci's steps/get-pull-requests.ps1 selects only pull requests that are not drafts, and no temporary workflow branch was made for either commit. The head was built and tested locally on Windows 11 with JDK 21.0.11 and the repository's Maven wrapper. The pipeline libraries it needs were built from the head first, with ./mvnw -B -o -pl pipeline/pipeline.derivedproperty -am install -DskipTests, and then

./mvnw -B -o -pl pipeline/pipeline.derivedproperty,pipeline/pipeline.developer-examples,pipeline/pipeline.developer-examples/pipeline.developer-examples.derivedproperty clean install

which passed 216 of 216 tests in pipeline.derivedproperty, being the 213 above and the three new SourceValuesTests, 167 of them the shared cases, and 2 of 2 in the example. Get-PackagesJsonGaps in ci/packages.ps1 found no gaps, ci/check-root-locale.ps1 reported "Every case change in 567 main source files names its locale.", and common-ci's scripts/utm-lint.ps1 reported the repository clean.

An earlier full build, run 43673959199, passed on c9693a9, before 0cca79f, 6323472 and the merge of rc.

macOS was left out of this draft's full build. The rc head is built with macOS before it merges into main.

With the Enterprise data files

At 99cec84 the built in HumanConfidence was also run end to end, through a small program outside the repository, in a Pipeline of the on-premise Device Detection engine with the Enterprise data file of 19 November 2026, the on-premise IP Intelligence engine with the Enterprise IP Intelligence data file of the same date, and the Derived Property Element. Both engines were built from this branch, native code included, on Windows with JDK 21. Each request carried a User-Agent and a client IP address that the IP Intelligence data file gives a HumanProbability of 10 and a ConnectionType of Broadband, except the crawler, which carried Google's. Where the 51Degrees JavaScript is taken to have run, the request also carried the cookies 51D_IsVisible and 51D_HasWebDriver, which are what the JavaScript in the data file's IsVisibleJavaScript and HasWebDriverJavaScript properties writes, with the values True, False or Unknown.

On the first request, with no cookies, the Device Data gave IsVisible and HasWebDriver as false through the typed accessor and as Unknown through getStoredValueAsString. The same requests were then run with SourceValues.java as it was at 20510b8, before 1e386ac.

Request At 99cec84 At 20510b8
First request, Chrome 155 (released 2 months before), no cookies Unknown Medium
Chrome 155, 51D_IsVisible=True, 51D_HasWebDriver=False High High
Chrome 142 (released 13 months before), the same cookies Medium Medium
Chrome 155, 51D_IsVisible=True, 51D_HasWebDriver=True Low Low
Googlebot User-Agent and address, no cookies Low Low
Chrome 155, 51D_IsVisible=Unknown, 51D_HasWebDriver=False Unknown Medium

So the two requests on which the window was never measured answer Unknown with 1e386ac, and answered Medium before it. The other four answers are the same either way. The example and the package's tests need no data file, and no test in either module skips without the Enterprise data file, so nothing more was run with it.

What it releases

pipeline.derivedproperty and pipeline.developerexamples.derivedproperty are new packages, released at their first version. The change to pipeline/pom.xml releases the pipeline parent with the packages that depend on it. Both new entries in ci/packages.json give a FirstVersion, which the release plan needs for a package that has never been released (commit 0cca79f). The shared release plan script reads it (steps/release-plan.ps1 lines 173 to 231).

Base branch

rc 81e3d4a is merged into the branch at a05afaa, with no conflict, and rc 4ff58d8 is merged into the branch at the head, 99cec84, with no conflict. That merge brings ProvidesValuesAsString, which 1e386ac uses and which pipeline.engines gained on rc after a05afaa, so the branch builds on its own as well as merged into rc.

Changes at this head

Two commits follow the previous head, 20510b8.

  • 1e386ac9 reads the stored string where a script compares a property with text, as What changed says, and adds the three SourceValuesTests that Tests describes. It changes SourceValues.java and SourceValuesTests.java only, and its results are under Full build.
  • 99cec84d merges rc 4ff58d8, as Base branch says.

Before it, three commits followed 10ee0ea, which made the documentation say that two Derived Property Elements must not share a parallel group.

  • e33d0df5 changes the one test in DerivedPropertyElementTests that runs two Derived Property Elements together. It put them in the same parallel group, with a comment saying they write the one Element Data at the same time, which is the Pipeline that derived-property-element.md lines 242 to 245 and parallel-processing.md lines 48 to 53 forbid. It now adds them one after the other, keeps its two assertions on the one Element Data, and is named twoElementsOneAfterTheOtherShareTheElementData, with a comment saying why the two are never put in one group. It is the test that shows two Derived Property Elements in one Pipeline both giving values, which the page's lines 40 to 43 allow. The test is one this draft adds, not a released test, and it passed before the change as it does after, so there is no red to show.
  • 6649039e names the line separator, the paragraph separator and the byte order mark in JsText.isWhitespace as constants written in hexadecimal, as VERTICAL_TAB is, where they were raw characters a reader cannot see and an editor can drop. The values are the same, so nothing the element does changes.
  • 20510b84 moves the specification link in DerivedPropertyElementConfigurationTests from 706de1f0 to 52e9b047, where every other link in the package is. The page is the same at 52e9b047 and at d79bbe3b, the specification's rc.

The green of those three is run 44468789796, as Full build says, with the local run beside it.

The red and green of the tests first commit are under Tests, and the script that reads FirstVersion is linked under What it releases.

What remains

  • The example does not use the built-in scripts, as the specification asks. The Enterprise data files give every property HumanConfidence names, as With the Enterprise data files shows.
  • The run with the Enterprise data files is not a test in the repository. The stored string read is tested in the repository with stub Element Data.
  • No CI has run at 1e386ac or 99cec84, as Full build says.

Adds the tests for a new pipeline.derivedproperty module and a new
developer example, before the code they test. The module is the Derived
Property Element of the specification, which produces new properties from
the properties other elements in the Pipeline produce, following rules
written in a YAML or JSON script.

The derived-properties repository is added as the submodule
pipeline/derived-properties, at cc30638. Its scripts folder holds the
built-in scripts the package ships, and its tests folder holds the case
files and rejection cases every language runs. SharedCasesTests runs both,
and BuiltInScriptTests fails when BuiltInScript and the scripts folder
differ.

The other tests cover reading YAML by the YAML 1.2 core schema, so yes
stays a string, the conversion of source values, the Pipeline check, the
no value message, the logging, the metadata and building from a
configuration file. ExampleTests and FindOutMoreTests run the example.

The two modules are added to pipeline/pom.xml, the developer examples pom
and ci/packages.json, and SnakeYAML is named in the dependency management
of pipeline/pom.xml. The tests do not compile until the code follows.
pipeline.derivedproperty is the Derived Property Element of the
specification. It is a Flow Element rather than an Aspect Engine, holds no
data file, makes no network request and accepts no evidence. Every
instance writes into the Element Data under the key derived, as every
Translation Engine shares translation.

DerivedPropertyElementBuilder takes scripts shipped in the package
(BuiltInScript), files on disk with wildcards in the file name, and text
from code, and a configuration file names it as DerivedPropertyElement
with the Scripts and ScriptFiles build parameters. Each script is parsed
with the compose API of SnakeYAML, so every node keeps its line, and plain
scalars are resolved by the YAML 1.2 core schema. The validator follows
tools/validate.mjs in the derived-properties repository, with the same
paths and messages, and reports every fault in every script at once.

Per request the element reads each source property once, converts it to
the type the script infers without coercing loosely, then runs the checks
and the rules. Where a source property is not available the output has no
value, using the existing AspectPropertyValue mechanism, and the message
names every property that could not be read, in the wording of
tools/evaluate.mjs. A request with no Element Data under a key a script
reads is logged once per script.

When the Pipeline adds the element it checks that some element lists each
source property, without judging where that element sits, and that no two
Derived Property Elements produce the same property.

The developer example builds one Pipeline in code, with a script held as
text and a script read from a file beside the example, and the same
Pipeline from a configuration file, and shows the message a caller sees
when a source property is not available. It ends with a Find out more
section.
…ccepts

The tests made each pattern with Path.resolve("*.yaml"), and Windows
refuses a path that holds a wildcard with InvalidPathException, so three
DerivedPropertyElementConfigurationTests failed in every Windows job of
full build 43667464820. The element itself splits a pattern with
File.getName and File.getParentFile and was not affected.
jwrosewell added a commit that referenced this pull request Nov 15, 2026
…its verify (#161)

Tests only. `DataUpdateServiceTests.DataUpdateService_Register_AutoUpdateConfiguredInterval` fails about once in 600 runs, and it failed `Ubuntu_Java_17` in full build [43667464820](https://github.com/51Degrees/51degrees-java/actions/runs/43667464820) of [#156](#156), which changes nothing in `pipeline.engines`. This fixes the cause rather than leaving it to a re-run, which is the rule for a test that fails on timing. [#122](#122), which is open and not yet merged, adds that rule to AGENTS.md.
jwrosewell added a commit that referenced this pull request Nov 16, 2026
…ith no key to the drafts that carry them

The Derived Property Element and its example are #156's, Comparer on the
free data file is #158's, and every cloud example ending cleanly with no
resource key is #163's, so each file this draft changed only for those
is put back to rc's version. The cloud cases of the no-settings tests go
with them, as #163's tests cover each cloud example run with no key.
jwrosewell and others added 8 commits November 19, 2026 03:32
The Derived Property Element page, Key Considerations 6, says each Derived
Property Element writes into the same Element Data, so two of them MUST NOT
be members of the same parallel group. Nothing checks this, so the
documentation says it. DerivedPropertyData said the opposite, that elements
in a parallel group write at the same time, and now says they must not.
DerivedPropertyElement, its builder and the pipeline README say it too, and
the links to the page are at the specification commit that has that
sentence.
The test that shows two Derived Property Elements writing into the one
Element Data put them in the same parallel group, and its comment said
that is how they work. The specification says the opposite. Every
Derived Property Element writes into the same Element Data, so two of
them must not be members of the same parallel group.

The test now adds the two elements one after the other, keeps its two
assertions on the one Element Data, and processes one request, because
nothing runs at once any more. It is renamed to say so, and its comment
says why the two elements are never put in one parallel group.

Specification, derived-property-element.md lines 242 to 245 and
parallel-processing.md lines 48 to 53 at d79bbe3b.
JsText.isWhitespace held the line separator, the paragraph separator and
the byte order mark as raw characters inside char literals. A reader
cannot see them, and an editor or a change of encoding can drop them
without a trace. They are now named constants written as hexadecimal
numbers, as VERTICAL_TAB already is, so the file holds only ASCII.

The values are the same, so nothing the element does changes.
DerivedPropertyElementConfigurationTests linked the specification page
at 706de1f0, where every other link in the package is at 52e9b047. The
page is the same at 52e9b047 and at d79bbe3b, the tip of the
specification's rc, so one commit now covers every link.
Where the element data can give the string it stored for a property,
through ProvidesValuesAsString as the on-premise engines can, a script
that compares the property with text now reads that string. The typed
accessor answers in its own type, so the Unknown that device detection
stores for a property the 51Degrees JavaScript has not yet measured read
as False, and HumanConfidence could never answer Unknown from the
on-premise engine. A script that compares the property with a boolean or
a number still reads the typed value. This is what .NET does with
IProvidesValuesAsString.
They first ship in the 4.6 release, which is 4.6.2 in every repository.
Tracy51D added a commit that referenced this pull request Nov 20, 2026
…Elements, which #202's every-element check asks for
@jwrosewell
jwrosewell marked this pull request as ready for review November 21, 2026 16:43
@jwrosewell
jwrosewell merged commit 712b03a into rc Nov 21, 2026
6 checks passed
@jwrosewell
jwrosewell deleted the feature/derived-property-element branch November 21, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants