Repository navigation
Refuse writes to a finished Flow Data and to finished Element Data - #247
Merged
Merged
Conversation
Each package that defines a Flow Element gains a test that processes every Flow Element the package and its examples define, then writes to its Element Data and expects an error. It also checks that it covers every Flow Element class the package's own modules define. The pipeline core test also checks that a finished Flow Data refuses Evidence, Element Data and errors, that a second process raises even when process exceptions are suppressed, and that Flow Elements sharing one Element Data write it while each processes.
FlowData raises once processing has finished when Evidence, Element Data or an error is added, and a second process raises without adding an error. ElementData records when the Flow Element that writes it has finished, and then setting a public attribute, or a write method such as the derived property data's set_value, raises. A Flow Element that shares the instance opens it again while it processes. Names that start with an underscore hold the data's own state, such as values kept once read, and closing the Flow Data is not a write. The checks read a field set when processing ends and take no lock.
The contents of ElementDataDictionary, AspectDataDictionary and CloudData, and a hardware profile, which is itself a dict, can still be written once the Flow Element that provides them has finished. Pipeline core now checks each dict write method and that the contents stay a dict. Each package's check over every Flow Element also writes an item, and the Device Detection cloud check writes to a hardware profile in each way.
ElementDataDictionary and AspectDataDictionary, and so CloudData, keep their contents in a dict that checks the data is still being written before each write method, as setting an attribute does. Reads and the dict type are unchanged, and a copy is a plain dict. Setting contents keeps a checked copy. HardwareProfileData, a dict itself, checks its own write methods the same way. A profile is finished when the hardware data that holds it is, as the Flow Data records that only on the data it holds.
Data added before process(), or added by a Flow Element under another Flow Element's key, was never recorded as finished, so it could still be written once the Flow Data had finished processing.
Once every Flow Element has run, process() records each Element Data the Flow Data holds as finished, including data added before processing and data a Flow Element added under another Flow Element's key.
Keeps both messages this branch adds and the usage sharing message rc adds.
Adding Element Data, errors or Evidence to a Flow Data that has finished processing must fail. These tests write through each public dictionary, being data, errors and the Evidence's values, and replace each of them, after processing, and expect the error. The two released set header tests now give their data to a Flow Data that has not been processed, because a finished one takes no more Element Data.
A Flow Data now keeps its Element Data, its errors and its Evidence values in dicts whose write methods raise once processing has finished, and data, errors and evidence cannot be replaced then. The check reads the processed field and takes no lock. add_from_dict also takes any dict, such as the Evidence of another Flow Data.
Closing Element Data releases what it holds, which is not a write. These tests close finished data directly, an on-premise engine's results in each on-premise package and a class whose close sets a public attribute in the core, and expect the close to work and the data to stay finished.
Closing Element Data releases what it holds and is not a write. Each close method an Element Data class defines or inherits now runs with the data open for writing and finished again afterwards, so calling close() on finished on-premise results releases them instead of raising. FlowData.close relies on the same and no longer opens the data itself.
…mixed into pylint reported no-member for each write method of the mixin, because _check_writable and the dict methods come from the class that mixes it in, and the package's lint fails on any error.
jwrosewell
marked this pull request as ready for review
November 21, 2026 16:54
jwrosewell
added a commit
that referenced
this pull request
Nov 21, 2026
rc now holds the read only Flow Data and finished Element Data from #247, whose earlier commits this branch also carried, and the linked list of Element Data from #250. The merge result is rc with this branch's pool commits replayed on it, and nothing else from the branch's copies of #247. In flowdata.py, _give_back sits after rc's _GIVEN table. close() keeps rc's call of the close method, which opens the Element Data itself, in place of the branch's own _set_finished calls around it, and then gives the instance back to its pool. The second close guard and the once only give back for data held under two keys are kept, and they read the Element Data through rc's data view. The core readme keeps rc's web integration section and adds the section saying Element Data is reused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Flow Data could take Evidence, Element Data and errors after it had finished processing, and Element Data could be written at any time, for example by setting an attribute on it once its Flow Element had finished. The specification now makes both fail with an error, and asks for a check over every Flow Element in the repository.
Specification
specifications#190, a draft into
rc, at its head a90ba516, thread safety page.Lines 115 to 118:
Lines 143 to 149:
Lines 151 to 154 say values Element Data reads from a results instance, and values it keeps once read, are not writes. Lines 156 to 158:
Python has no lazy properties and no parallel groups, so neither exception applies here.
The change
Two commits, tests first.
What changes:
FlowData.process()sets itsprocessedfield when processing ends, as it did.FlowData.set_element_data(),FlowData.set_error(),Evidence.add()andEvidence.add_from_dict()raise once that field is set. A secondprocess()raises "FlowData already processed" without adding an error, whether or not process exceptions are suppressed.ElementDatarecords when the Flow Element that writes it has finished.FlowData.process()marks each Flow Element's data finished once that element'sprocessreturns, and opens data already under the element's key before it processes, so Flow Elements that share one instance, as the derived property elements do, each write it while they process.DerivedPropertyData.set_value(). Attribute names that start with an underscore hold the data's own state, such as values kept once read, and are not writes. Closing a Flow Data is not a write either.Messages.FLOW_DATA_FINISHEDandMessages.ELEMENT_DATA_FINISHEDin pipeline core.The tests:
pipeline/fiftyone_pipeline_core/tests/test_finished_writes.pychecks each Flow Data write, the secondprocess(), Flow Elements sharing one instance, underscore names, and closing.tests/test_finished_writes.pyin pipeline core, engines, usage sharing, the cloud request engine, derived properties, translation, Device Detection cloud and on-premise, IP Intelligence cloud, on-premise and translation, and location processes every Flow Element that package defines, then sets an attribute on its Element Data and expects the error. The derived property test also callsset_value(). Each test also lists the Flow Element classes its package's own modules define and checks that it processed every one, apart from the bases others build on (FlowElement,Engine, and the IP IntelligenceWeightedCloudEngine). Usage sharing adds no Element Data, so its test checks that a finished Flow Data refuses one.Released tests
The last release is
mainat 612aa28. Every package'stestsfolder from it was run unaltered againstrcat 2373d14 and against this draft, with no key, and the six cloud packages again with a resource key. The failing tests are the same by name in every package, so this draft changes no released result and updates no released test.rc, no keyrc, keyThe released tests failing on
rcare dealt with in #246, or need a key.The red and the green
Both ran through a throwaway build of each commit, without macOS, on Ubuntu and Windows with Python 3.10 to 3.14.
test_finished_writes.py.test_allowed_window_name_from_evidence_is_used_0_selfintest_javascriptbuilder_object_name.py, because itsnode --checkcall ran past its 60 second timeout on that runner. Neither test touches what this draft changes, and everytest_finished_writes.pypassed in all ten jobs.Both throwaway branches are deleted.
Local counts
Each touched package's own tests on this draft, on Windows with Python 3.14.5, the free Lite data files, and a resource key for the cloud packages.
The IP Intelligence translation failure is
test_translations_are_packaged, which checks a built package and fails the same way onrcwhen run from the source tree. It passed in every job of the green run.Against
rc's source, the new tests fail in every package (pipeline core 8 of 11, engines 1 of 2, the others all), and the three that pass there check behaviour that does not change: Evidence added before processing, underscore names, and closing.What remains
ElementDataDictionary.contentsis a plain dictionary, and changing an item in it is not a write method of the Element Data, so it is not checked.node --checktimeout intest_javascriptbuilder_object_name.pyneeds its own fix, as a timing failure.Before you upgrade
process()raises even when process exceptions are suppressed.DerivedPropertyData.set_value(), after the Flow Element that writes it has finished processing raises an exception.process(), and create a new Flow Data for another request. Process a Flow Data once. A Flow Element writes its Element Data while it processes, and code that kept its own values on Element Data after processing keeps them in its own variables instead, or under a name that starts with an underscore in a Flow Element's own Element Data class.