Skip to content

Refuse writes to a finished Flow Data and to finished Element Data - #247

Merged
jwrosewell merged 12 commits into
rcfrom
remote/finished-writes-fail
Nov 21, 2026
Merged

jwrosewell merged 12 commits into
rcfrom
remote/finished-writes-fail

Conversation

@jwrosewell

Copy link
Copy Markdown
Contributor

Problem

A Flow Data could take Evidence, Element Data and errors after it had finished processing, and Element Data could be written at any time, for example by setting an attribute on it once its Flow Element had finished. The specification now makes both fail with an error, and asks for a check over every Flow Element in the repository.

Specification

specifications#190, a draft into rc, at its head a90ba516, thread safety page.

Lines 115 to 118:

Once processing has finished, a Flow Data is read only, except for lazy properties, and its implementation MUST NOT take a lock. Adding Evidence, Element Data, errors or warnings to a Flow Data that has finished processing MUST fail with an error.

Lines 143 to 149:

An Element Data instance is written only by the Flow Element that provides it, or by the Flow Elements that share it, and only while they process. Each Element Data records when a Flow Element that writes it finishes, and a Flow Element that shares the instance opens it for writing again when it starts to process. A write through its write methods while the instance is recorded as finished MUST fail with an error, except by a lazy property's task.

Lines 151 to 154 say values Element Data reads from a results instance, and values it keeps once read, are not writes. Lines 156 to 158:

Each implementation MUST include a check, run by its automated tests, that processes every Flow Element in the repository, then writes to its Element Data and expects the error.

Python has no lazy properties and no parallel groups, so neither exception applies here.

The change

Two commits, tests first.

  1. ef2ce648 adds the tests.
  2. eae137e2 makes them pass.

What changes:

  • FlowData.process() sets its processed field when processing ends, as it did. FlowData.set_element_data(), FlowData.set_error(), Evidence.add() and Evidence.add_from_dict() raise once that field is set. A second process() raises "FlowData already processed" without adding an error, whether or not process exceptions are suppressed.
  • ElementData records when the Flow Element that writes it has finished. FlowData.process() marks each Flow Element's data finished once that element's process returns, and opens data already under the element's key before it processes, so Flow Elements that share one instance, as the derived property elements do, each write it while they process.
  • Once finished, setting a public attribute on Element Data raises, and so does a write method, which in this repository is DerivedPropertyData.set_value(). Attribute names that start with an underscore hold the data's own state, such as values kept once read, and are not writes. Closing a Flow Data is not a write either.
  • Every check reads a field and takes no lock.
  • The messages are Messages.FLOW_DATA_FINISHED and Messages.ELEMENT_DATA_FINISHED in pipeline core.

The tests:

  • pipeline/fiftyone_pipeline_core/tests/test_finished_writes.py checks each Flow Data write, the second process(), Flow Elements sharing one instance, underscore names, and closing.
  • tests/test_finished_writes.py in pipeline core, engines, usage sharing, the cloud request engine, derived properties, translation, Device Detection cloud and on-premise, IP Intelligence cloud, on-premise and translation, and location processes every Flow Element that package defines, then sets an attribute on its Element Data and expects the error. The derived property test also calls set_value(). Each test also lists the Flow Element classes its package's own modules define and checks that it processed every one, apart from the bases others build on (FlowElement, Engine, and the IP Intelligence WeightedCloudEngine). Usage sharing adds no Element Data, so its test checks that a finished Flow Data refuses one.
  • The example Flow Elements of pipeline core, engines, the cloud request engine and derived properties are processed the same way.

Released tests

The last release is main at 612aa28. Every package's tests folder from it was run unaltered against rc at 2373d14 and against this draft, with no key, and the six cloud packages again with a resource key. The failing tests are the same by name in every package, so this draft changes no released result and updates no released test.

Package rc, no key Draft, no key rc, key Draft, key
Pipeline core 1 failed 1 failed
Engines 31 failed 31 failed
Usage sharing 8 failed 8 failed
Cloud request engine 7 failed 7 failed 3 failed 3 failed
Translation 0 failed 0 failed
51Did 8 failed, 5 errors 8 failed, 5 errors
Device Detection cloud 9 failed 9 failed 1 failed 1 failed
Device Detection on-premise 6 failed 6 failed
Device Detection examples 6 failed, 1 error 6 failed, 1 error 0 failed 0 failed
IP Intelligence shared and cloud 0 failed 0 failed 0 failed 0 failed
IP Intelligence on-premise 1 failed 1 failed
IP Intelligence translation 1 failed 1 failed
IP Intelligence examples 4 failed 4 failed 0 failed 0 failed
Location 1 error 1 error 0 failed 0 failed

The released tests failing on rc are dealt with in #246, or need a key.

The red and the green

Both ran through a throwaway build of each commit, without macOS, on Ubuntu and Windows with Python 3.10 to 3.14.

  • Red: run 44188488411, whose workflow commit fafc6ac sits on ef2ce64. All ten Build and Test jobs failed, and every failure was in a test_finished_writes.py.
  • Green: run 44188497169, whose workflow commit 008a431 sits on eae137e. Eight jobs passed. Windows Python 3.12 failed one browser test of a cloud web example, because Chrome did not start ("session not created from disconnected: unable to connect to renderer"). Windows Python 3.13 failed one unit test, test_allowed_window_name_from_evidence_is_used_0_self in test_javascriptbuilder_object_name.py, because its node --check call ran past its 60 second timeout on that runner. Neither test touches what this draft changes, and every test_finished_writes.py passed in all ten jobs.

Both throwaway branches are deleted.

Local counts

Each touched package's own tests on this draft, on Windows with Python 3.14.5, the free Lite data files, and a resource key for the cloud packages.

Package Result
Pipeline core 571 passed, 212 subtests
Engines 127 passed, 26 subtests
Usage sharing 122 passed, 6 subtests
Cloud request engine 151 passed, 2 skipped, 6 subtests
Derived properties 243 passed
Translation 35 passed, 49 subtests
Device Detection cloud 28 passed, 2 skipped, 3 subtests
Device Detection on-premise 104 passed, 14 subtests
IP Intelligence cloud 13 passed
IP Intelligence on-premise 79 passed, 13 subtests
IP Intelligence translation 26 passed, 1 failed
Location 19 passed, 10 subtests

The IP Intelligence translation failure is test_translations_are_packaged, which checks a built package and fails the same way on rc when run from the source tree. It passed in every job of the green run.

Against rc's source, the new tests fail in every package (pipeline core 8 of 11, engines 1 of 2, the others all), and the three that pass there check behaviour that does not change: Evidence added before processing, underscore names, and closing.

What remains

  • A Flow Element in a package added later is covered only when that package gains the same test.
  • ElementDataDictionary.contents is a plain dictionary, and changing an item in it is not a write method of the Element Data, so it is not checked.
  • The node --check timeout in test_javascriptbuilder_object_name.py needs its own fix, as a timing failure.

Before you upgrade

  1. What changed.
    • Adding Evidence, Element Data or an error to a Flow Data that has finished processing raises an exception, and a second process() raises even when process exceptions are suppressed.
    • Setting a public attribute on Element Data, or calling DerivedPropertyData.set_value(), after the Flow Element that writes it has finished processing raises an exception.
  2. Why. specifications#190 at a90ba516, lines 117 and 118 says "Adding Evidence, Element Data, errors or warnings to a Flow Data that has finished processing MUST fail with an error.", and lines 147 to 149 say "A write through its write methods while the instance is recorded as finished MUST fail with an error, except by a lazy property's task."
  3. What to change. Add all Evidence before calling process(), and create a new Flow Data for another request. Process a Flow Data once. A Flow Element writes its Element Data while it processes, and code that kept its own values on Element Data after processing keeps them in its own variables instead, or under a name that starts with an underscore in a Flow Element's own Element Data class.

jwrosewell and others added 11 commits November 17, 2026 22:16
Each package that defines a Flow Element gains a test that processes every
Flow Element the package and its examples define, then writes to its
Element Data and expects an error. It also checks that it covers every Flow
Element class the package's own modules define.

The pipeline core test also checks that a finished Flow Data refuses
Evidence, Element Data and errors, that a second process raises even when
process exceptions are suppressed, and that Flow Elements sharing one
Element Data write it while each processes.
FlowData raises once processing has finished when Evidence, Element Data or
an error is added, and a second process raises without adding an error.

ElementData records when the Flow Element that writes it has finished, and
then setting a public attribute, or a write method such as the derived
property data's set_value, raises. A Flow Element that shares the instance
opens it again while it processes. Names that start with an underscore hold
the data's own state, such as values kept once read, and closing the Flow
Data is not a write.

The checks read a field set when processing ends and take no lock.
The contents of ElementDataDictionary, AspectDataDictionary and CloudData,
and a hardware profile, which is itself a dict, can still be written once
the Flow Element that provides them has finished. Pipeline core now checks
each dict write method and that the contents stay a dict. Each package's
check over every Flow Element also writes an item, and the Device
Detection cloud check writes to a hardware profile in each way.
ElementDataDictionary and AspectDataDictionary, and so CloudData, keep
their contents in a dict that checks the data is still being written
before each write method, as setting an attribute does. Reads and the dict
type are unchanged, and a copy is a plain dict. Setting contents keeps a
checked copy.

HardwareProfileData, a dict itself, checks its own write methods the same
way. A profile is finished when the hardware data that holds it is, as the
Flow Data records that only on the data it holds.
Data added before process(), or added by a Flow Element under another
Flow Element's key, was never recorded as finished, so it could still be
written once the Flow Data had finished processing.
Once every Flow Element has run, process() records each Element Data the
Flow Data holds as finished, including data added before processing and
data a Flow Element added under another Flow Element's key.
Keeps both messages this branch adds and the usage sharing message rc adds.
Adding Element Data, errors or Evidence to a Flow Data that has finished processing must fail. These tests write through each public dictionary, being data, errors and the Evidence's values, and replace each of them, after processing, and expect the error. The two released set header tests now give their data to a Flow Data that has not been processed, because a finished one takes no more Element Data.
A Flow Data now keeps its Element Data, its errors and its Evidence values in dicts whose write methods raise once processing has finished, and data, errors and evidence cannot be replaced then. The check reads the processed field and takes no lock. add_from_dict also takes any dict, such as the Evidence of another Flow Data.
Closing Element Data releases what it holds, which is not a write. These tests close finished data directly, an on-premise engine's results in each on-premise package and a class whose close sets a public attribute in the core, and expect the close to work and the data to stay finished.
Closing Element Data releases what it holds and is not a write. Each close method an Element Data class defines or inherits now runs with the data open for writing and finished again afterwards, so calling close() on finished on-premise results releases them instead of raising. FlowData.close relies on the same and no longer opens the data itself.
…mixed into

pylint reported no-member for each write method of the mixin, because _check_writable and the dict methods come from the class that mixes it in, and the package's lint fails on any error.
@jwrosewell
jwrosewell marked this pull request as ready for review November 21, 2026 16:54
@jwrosewell
jwrosewell merged commit 1af685b into rc Nov 21, 2026
3 checks passed
@jwrosewell
jwrosewell deleted the remote/finished-writes-fail branch November 21, 2026 16:54
jwrosewell added a commit that referenced this pull request Nov 21, 2026
rc now holds the read only Flow Data and finished Element Data from
#247, whose earlier commits this branch also carried, and the linked
list of Element Data from #250. The merge result is rc with this
branch's pool commits replayed on it, and nothing else from the
branch's copies of #247.

In flowdata.py, _give_back sits after rc's _GIVEN table. close() keeps
rc's call of the close method, which opens the Element Data itself, in
place of the branch's own _set_finished calls around it, and then gives
the instance back to its pool. The second close guard and the once only
give back for data held under two keys are kept, and they read the
Element Data through rc's data view.

The core readme keeps rc's web integration section and adds the
section saying Element Data is reused.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants