Repository navigation
perf(broadcast): skip a node's own stream events before decoding them - #348
Merged
Merged
Conversation
cosmin-staicu
requested review from
alinahornet,
cosminvlad,
florin-munteanu-uipath,
litheon,
lucianaparaschivei and
razvalex
as code owners
November 2, 2026 04:00
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 2, 2026 09:39
2015211 to
50a160e
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 2, 2026 09:39
4800c73 to
a1111ad
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 2, 2026 11:17
50a160e to
a9c1f24
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 2, 2026 11:17
a1111ad to
7ad4075
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 2, 2026 11:40
a9c1f24 to
6e2ed16
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 2, 2026 11:40
7ad4075 to
bdabaa2
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 2, 2026 11:59
6e2ed16 to
1287c77
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 2, 2026 11:59
bdabaa2 to
ed4aee7
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 2, 2026 12:37
1287c77 to
f4481bd
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 2, 2026 14:58
6a137fd to
bcbc5b5
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
2 times, most recently
from
November 2, 2026 15:39
c035850 to
b68b5a8
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 2, 2026 15:46
6638c9a to
96714f8
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
2 times, most recently
from
November 2, 2026 16:02
921507f to
aafb8c5
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 2, 2026 16:29
b333496 to
3d3cf27
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
2 times, most recently
from
November 2, 2026 16:53
32df3a0 to
44380fc
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 2, 2026 17:17
5ac6cae to
84b75bb
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 2, 2026 17:17
44380fc to
615aeac
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 3, 2026 07:49
4855562 to
9c972f0
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
2 times, most recently
from
November 3, 2026 09:13
d539561 to
da6f59b
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 09:13
9c972f0 to
6b3e9d7
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 09:34
da6f59b to
1f431eb
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 09:34
6b3e9d7 to
83c0fb2
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 09:55
1f431eb to
924fc35
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 3, 2026 10:13
292dd31 to
978f2a8
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 10:13
924fc35 to
ab1e44b
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 10:56
978f2a8 to
f1a29d6
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 10:56
ab1e44b to
8b3b24e
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 11:24
f1a29d6 to
baff21c
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 11:24
8b3b24e to
30da9ae
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 12:34
baff21c to
8f9ab38
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
2 times, most recently
from
November 3, 2026 13:06
46aa3fc to
c4180bb
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 13:06
8f9ab38 to
4646ae4
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 13:33
c4180bb to
aace4a3
Compare
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
2 times, most recently
from
November 3, 2026 13:53
8f93501 to
f003b9b
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.4.29
branch
from
November 3, 2026 13:53
aace4a3 to
e84941d
Compare
Every node's consumer group reads every entry of its topic's stream, its own included. An own entry went through the full path, reply parse, System.Text.Json decode, Uri and dictionary allocations, and was then dropped by SameSource: about 2.2-2.6 KB per entry and about 4.5 MB per periodic backlog read in the load test, with the read metric's topic name built per entry on top. The topic now writes the event's source as a second field, src, after the payload. The reader compares that field with its own source, as UTF-8 bytes, before anything is decoded, and an own entry is acknowledged and moves the group on exactly as before; the gap check, loss reporting and ordering are untouched. The same shortcut applies to the entries a group made at the head is checked against. The topic name for the read metric is built once. Mixed versions share a stream during a rolling deploy: - A 3.0 reader takes the payload by its field name (entry[FieldName], at v2.1.4, v3.0.0 and main) and ignores the extra field; an integration test reads a published entry back with the same client call and decodes it. - A 3.1 reader decodes an entry without the field, a 3.0 writer's, and compares the source as before; an entry whose source matches only case-insensitively is also decoded, so the shortcut never drops more than SameSource did. - With EmitStreamReceivedEvent on, own entries are still decoded so the receipt carries the event id. Both the Redis 7 script read and the script-free read below 7 pass the entries through unchanged, so both skip the own entry; tests cover both on the in-memory stream and on Redis. An allocation probe over a 1000-entry own backlog on the in-memory stream goes from 2835 bytes per entry (decoded) to 1072 (parse and acknowledge only); an old-format own entry still costs 2654. Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
cosmin-staicu
force-pushed
the
feat/skip-own-events
branch
from
November 3, 2026 15:44
f003b9b to
3aa016e
Compare
|
alinahornet
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Every node reads every Redis Streams broadcast event, its own included, and decoded each one in full before dropping its own. A writer now adds the event's source as a separate
srcfield; readers compare it first and skip their own entries without decoding.@event[FieldName], a lookup by name that ignores the extra field; a test pins that expression over the new field layout, and a Redis integration test reads a published entry back with the client. Entries from old writers have nosrcfield and are decoded and compared as before.EmitStreamReceivedEventis on (default off), so that receipt keeps the event id. A topic whoseFieldNameissrcwrites no source field and keeps the decode path.Evidence
Own-event backlog of 1000 entries, in-memory stream, script path: 2835 -> 1072 bytes per own entry, 0 decodes. An old-format own entry still costs 2654 bytes (decoded, as compatibility requires). Each reader now parses two more values per entry, about 100-130 bytes.
New tests: own-entry skip on both read paths (in-memory and real Redis, including a forced below-7 connector), ACK and position advance, old-format own and foreign entries, a case-only source difference, diagnostic-on decode, the field layout, and the allocation probe. net10.0 4327 passed, net8.0 4285 passed.
Merge Danger
Door: two-way for the code; the extra
srcfield is written to shared streams, which older readers ignore.Blast Radius: broadcast readers and writers on the Redis Streams transport. Own entries no longer log the Debug "event received" line on the fast path.