Repository navigation
Keep a Flow Data's Element Data in a linked list, added with no lock - #250
Merged
Merged
Conversation
Add test_flowdata_element_data_list.py. Two of its tests fail on rc with "dictionary changed size during iteration", being a reader part way through the Element Data when an add happens, and a view taken before an add. Three guards pass on rc too, being Pipeline order, the newest add for a key read first, and an add that takes no lock.
FlowData keeps the newest node of a linked list whose nodes never change, and set_element_data puts a new node at the head with a plain write, because this package has no parallel groups. FlowData.data is now a mapping view of the list as it was when read, so get, in, [], values() and iteration work as before. The FlowData docstring says how the list is kept and that a Flow Data is not safe to use from several threads at once.
Add five tests to test_flowdata_element_data_list.py. Two fail on the current head. A key set through a view is not found in that same view, and a view has no copy method, where the dictionary it replaced had one. Three pass already and guard what the change keeps. Element Data is not removed, setting FlowData.data copies the mapping given, and a reader on another thread always sees a whole list while Element Data is added. The test that an add takes no lock now says it is a smoke test that looks only at the FlowData's own attributes.
Setting a key on the view returned by FlowData.data added to the Flow Data but left the view on its old list, so the key was not found in it and update and setdefault did not show what they added. The view now moves to the list with the add. copy gives a dict, as it did when FlowData.data was a dictionary. keys, values and items walk the list once, not once for each key, and FlowData.get searches the list from its head with no view. The view's docstring says what a write, a removal and a copy do, and that its order is the order the keys were first added, which is Pipeline order while a Pipeline processes.
Setting FlowData.data empties the list and then adds each Element Data in turn, so a reader part way through the replacement sees a part of the new list. The new test lists the Element Data while the replacement runs and expects the old list until the new one is in place. It fails on the current head.
The setter now builds the new list first and puts its head in place with one write, so a reader sees the old list or the new one, never a part of it. Every write of the head goes through _write_head, so a check that applies to every write of the list has one place to go.
Jamesr51d
marked this pull request as ready for review
November 20, 2026 08:07
jwrosewell
added a commit
that referenced
this pull request
Nov 21, 2026
rc now holds the read only Flow Data and finished Element Data from #247, whose earlier commits this branch also carried, and the linked list of Element Data from #250. The merge result is rc with this branch's pool commits replayed on it, and nothing else from the branch's copies of #247. In flowdata.py, _give_back sits after rc's _GIVEN table. close() keeps rc's call of the close method, which opens the Element Data itself, in place of the branch's own _set_finished calls around it, and then gives the instance back to its pool. The second close guard and the once only give back for data held under two keys are kept, and they read the Element Data through rc's data view. The core readme keeps rc's web integration section and adds the section saying Element Data is reused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FlowDataheld its Element Data in a plaindict. Anything that walkedflowdata.datawhile a Flow Element added to it failed withRuntimeError: dictionary changed size during iteration, and a view taken from that dict before an add went on to show the add.The specification is direct about this (thread-safety.md line 92 at d79bbe3b):
Python has no parallel groups, so the plainer route of lines 100 to 102 applies:
What changed
All of it is in
fiftyone_pipeline_core/flowdata.py.set_element_dataputs a new node at the head with one plain assignment and takes no lock.getsearches along the list and still falls back toelement_not_foundfor a key nothing has added.FlowData.datais now a property that returns anElementDataView, aMutableMappingover the list as it stood when the view was taken. It lists keys in the order they were first added, which is Pipeline order while a Pipeline processes. Setting a key through the view adds to the Flow Data and the view then shows it.copy()gives adict. Removals raiseTypeError, because Element Data is never taken out of a Flow Data.FlowData.databuilds a new list from the mapping given and swaps it in with one write, so a reader sees either the old list or the new one.Tests
tests/test_flowdata_element_data_list.pyis new and holds eleven tests. They cover a reader part way through the list when an add lands, a view taken before an add, listing order, the newest add winning for a key, writes and copies through the view, each removal, both halves of the data setter, and a reader on a second thread while 2,000 adds are made. One of them only checks the Flow Data's own attributes for a lock and says so in its docstring.No test that exists on
mainis changed. Code elsewhere that readsFlowData.datagets the view now, and every package's suite passes with it in the green run below.Build
Red: run 44624798592 built
tmp/red-250at 1e21d7a. That commit adds only a workflow file on top of 6b7b113, which is rc 823c655 with the new test file and no change to the code. All ten build-and-test jobs failed, on Ubuntu and Windows with Python 3.10 to 3.14. In each job the core package reported 5 failed and 571 passed, every other package passed, and the five failures were all in the new file:test_a_reader_part_way_through_is_not_upset_by_an_addandtest_a_view_taken_before_an_add_keeps_the_list_it_sawhitdictionary changed size during iteration.test_element_data_is_not_removedfailed because the dict let a key be removed.test_setting_data_copies_the_mapping_givenandtest_setting_data_replaces_the_whole_list_at_oncefailed because the old attribute just stored whatever object it was given.Green: run 44624783486 built a temporary branch at a489a58, which adds only a workflow file to the head e5e0628. Every job passed on Ubuntu and Windows with Python 3.10 to 3.14. The core package ran 576 tests with none failing in each of the ten jobs.
rc has moved on since that build. The only rc change under the core package since the branch's last merge of rc is to
tox.ini, and GitHub reports the branch merges cleanly.What remains
#247, which makes writes to a finished Flow Data fail, also edits
flowdata.py. Whichever of the two lands second has to merge rc first and put the finished check inFlowData._addand in thedatasetter, since every write to the list now goes through one of those two.test_a_reader_on_another_thread_always_sees_a_whole_listcannot fail by timing, but it can pass on a run where the reader never caught the list part way through. The two single-threaded reader tests carry the proof.Before you upgrade
FlowData.datais no longer adict. Line 92 of the thread safety page, quoted above, requires the change. What a caller may notice:isinstance(flowdata.data, dict)is now false. Callflowdata.data.copy()where a realdictis needed.del flowdata.data[key],pop,popitemandclearraiseTypeError. Nothing in these packages removes Element Data, so code that did so has to stop.flowdata.datakeeps the list it saw. An add made later throughset_element_data, or through a different view, does not show in it. Readflowdata.dataagain to see the latest list. A write through a view does show in that same view.flowdata.data = mappingcopies the mapping. Changingmappingafterwards no longer changes the Flow Data.Merging
Subject: Keep a Flow Data's Element Data in a linked list, added with no lock (#250)
Body: A Flow Data now keeps its Element Data in a linked list that a reader follows with no lock, so a read part way through is never upset by an add. FlowData.data returns a read-and-add view of that list rather than a dict, and Element Data can no longer be removed through it. Green run 44624783486.