Skip to content

Keep a Flow Data's Element Data in a linked list, added with no lock - #250

Merged
Jamesr51d merged 7 commits into
rcfrom
fix/parallel-element-data-list
Nov 20, 2026
Merged

Jamesr51d merged 7 commits into
rcfrom
fix/parallel-element-data-list

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

FlowData held its Element Data in a plain dict. Anything that walked flowdata.data while a Flow Element added to it failed with RuntimeError: dictionary changed size during iteration, and a view taken from that dict before an add went on to show the add.

The specification is direct about this (thread-safety.md line 92 at d79bbe3b):

A Flow Data MUST keep its Element Data in a linked list, not in a dictionary.

Python has no parallel groups, so the plainer route of lines 100 to 102 applies:

Where no parallel group can add at the same time, a plain write is enough. A node never changes once it is in the list, so a reader follows the links with no lock and always sees a whole list.

What changed

All of it is in fiftyone_pipeline_core/flowdata.py.

  • Element Data now lives in a singly linked list, newest node first. A node is never changed after it is made. set_element_data puts a new node at the head with one plain assignment and takes no lock.
  • get searches along the list and still falls back to element_not_found for a key nothing has added.
  • FlowData.data is now a property that returns an ElementDataView, a MutableMapping over the list as it stood when the view was taken. It lists keys in the order they were first added, which is Pipeline order while a Pipeline processes. Setting a key through the view adds to the Flow Data and the view then shows it. copy() gives a dict. Removals raise TypeError, because Element Data is never taken out of a Flow Data.
  • Assigning to FlowData.data builds a new list from the mapping given and swaps it in with one write, so a reader sees either the old list or the new one.
  • The class docstring now describes the list. The existing note that a Flow Data is not safe to use from several threads at once, and that the caller does any locking, is kept as it was.

Tests

tests/test_flowdata_element_data_list.py is new and holds eleven tests. They cover a reader part way through the list when an add lands, a view taken before an add, listing order, the newest add winning for a key, writes and copies through the view, each removal, both halves of the data setter, and a reader on a second thread while 2,000 adds are made. One of them only checks the Flow Data's own attributes for a lock and says so in its docstring.

No test that exists on main is changed. Code elsewhere that reads FlowData.data gets the view now, and every package's suite passes with it in the green run below.

Build

Red: run 44624798592 built tmp/red-250 at 1e21d7a. That commit adds only a workflow file on top of 6b7b113, which is rc 823c655 with the new test file and no change to the code. All ten build-and-test jobs failed, on Ubuntu and Windows with Python 3.10 to 3.14. In each job the core package reported 5 failed and 571 passed, every other package passed, and the five failures were all in the new file:

  • test_a_reader_part_way_through_is_not_upset_by_an_add and test_a_view_taken_before_an_add_keeps_the_list_it_saw hit dictionary changed size during iteration.
  • test_element_data_is_not_removed failed because the dict let a key be removed.
  • test_setting_data_copies_the_mapping_given and test_setting_data_replaces_the_whole_list_at_once failed because the old attribute just stored whatever object it was given.

Green: run 44624783486 built a temporary branch at a489a58, which adds only a workflow file to the head e5e0628. Every job passed on Ubuntu and Windows with Python 3.10 to 3.14. The core package ran 576 tests with none failing in each of the ten jobs.

rc has moved on since that build. The only rc change under the core package since the branch's last merge of rc is to tox.ini, and GitHub reports the branch merges cleanly.

What remains

#247, which makes writes to a finished Flow Data fail, also edits flowdata.py. Whichever of the two lands second has to merge rc first and put the finished check in FlowData._add and in the data setter, since every write to the list now goes through one of those two.

test_a_reader_on_another_thread_always_sees_a_whole_list cannot fail by timing, but it can pass on a run where the reader never caught the list part way through. The two single-threaded reader tests carry the proof.

Before you upgrade

FlowData.data is no longer a dict. Line 92 of the thread safety page, quoted above, requires the change. What a caller may notice:

  • isinstance(flowdata.data, dict) is now false. Call flowdata.data.copy() where a real dict is needed.
  • del flowdata.data[key], pop, popitem and clear raise TypeError. Nothing in these packages removes Element Data, so code that did so has to stop.
  • A view read from flowdata.data keeps the list it saw. An add made later through set_element_data, or through a different view, does not show in it. Read flowdata.data again to see the latest list. A write through a view does show in that same view.
  • flowdata.data = mapping copies the mapping. Changing mapping afterwards no longer changes the Flow Data.

Merging

Subject: Keep a Flow Data's Element Data in a linked list, added with no lock (#250)
Body: A Flow Data now keeps its Element Data in a linked list that a reader follows with no lock, so a read part way through is never upset by an add. FlowData.data returns a read-and-add view of that list rather than a dict, and Element Data can no longer be removed through it. Green run 44624783486.

Mark Campbell-Baker and others added 7 commits November 18, 2026 17:18
Add test_flowdata_element_data_list.py. Two of its tests fail on rc
with "dictionary changed size during iteration", being a reader part
way through the Element Data when an add happens, and a view taken
before an add. Three guards pass on rc too, being Pipeline order, the
newest add for a key read first, and an add that takes no lock.
FlowData keeps the newest node of a linked list whose nodes never
change, and set_element_data puts a new node at the head with a plain
write, because this package has no parallel groups. FlowData.data is
now a mapping view of the list as it was when read, so get, in, [],
values() and iteration work as before. The FlowData docstring says how
the list is kept and that a Flow Data is not safe to use from several
threads at once.
Add five tests to test_flowdata_element_data_list.py. Two fail on the
current head. A key set through a view is not found in that same view,
and a view has no copy method, where the dictionary it replaced had one.

Three pass already and guard what the change keeps. Element Data is not
removed, setting FlowData.data copies the mapping given, and a reader on
another thread always sees a whole list while Element Data is added.

The test that an add takes no lock now says it is a smoke test that
looks only at the FlowData's own attributes.
Setting a key on the view returned by FlowData.data added to the Flow
Data but left the view on its old list, so the key was not found in it
and update and setdefault did not show what they added. The view now
moves to the list with the add.

copy gives a dict, as it did when FlowData.data was a dictionary.
keys, values and items walk the list once, not once for each key, and
FlowData.get searches the list from its head with no view.

The view's docstring says what a write, a removal and a copy do, and
that its order is the order the keys were first added, which is
Pipeline order while a Pipeline processes.
Setting FlowData.data empties the list and then adds each Element Data
in turn, so a reader part way through the replacement sees a part of
the new list. The new test lists the Element Data while the replacement
runs and expects the old list until the new one is in place. It fails
on the current head.
The setter now builds the new list first and puts its head in place
with one write, so a reader sees the old list or the new one, never a
part of it. Every write of the head goes through _write_head, so a
check that applies to every write of the list has one place to go.
rc now has #244, whose FlowData docstring says how a FlowData is shared
across threads. The class docstring keeps both paragraphs, this draft's
linked-list paragraph first and #244's thread paragraph after it, in
place of this draft's shorter thread sentence.
@Jamesr51d
Jamesr51d marked this pull request as ready for review November 20, 2026 08:07
@Jamesr51d
Jamesr51d merged commit 70ae25f into rc Nov 20, 2026
3 checks passed
@Jamesr51d
Jamesr51d deleted the fix/parallel-element-data-list branch November 20, 2026 08:07
jwrosewell added a commit that referenced this pull request Nov 21, 2026
rc now holds the read only Flow Data and finished Element Data from
#247, whose earlier commits this branch also carried, and the linked
list of Element Data from #250. The merge result is rc with this
branch's pool commits replayed on it, and nothing else from the
branch's copies of #247.

In flowdata.py, _give_back sits after rc's _GIVEN table. close() keeps
rc's call of the close method, which opens the Element Data itself, in
place of the branch's own _set_finished calls around it, and then gives
the instance back to its pool. The second close guard and the once only
give back for data held under two keys are kept, and they read the
Element Data through rc's data view.

The core readme keeps rc's web integration section and adds the
section saying Element Data is reused.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants