Skip to content

perf(broadcast): skip a node's own Pub/Sub messages before decoding them - #349

Merged
cosmin-staicu merged 1 commit into
mainfrom
feat/pubsub-skip-own
Nov 3, 2026
Merged

cosmin-staicu merged 1 commit into
mainfrom
feat/pubsub-skip-own

Conversation

@cosmin-staicu

Copy link
Copy Markdown
Member

Summary

On the Redis Pub/Sub transport a node also receives the messages it publishes, and decoded each one in full (System.Text.Json) before dropping its own. Formatters can now read only the event source from the payload, so a node's own message is dropped without a decode. The message format does not change.

 OnMessage(payload)
+  formatter has a source matcher and it says "from this node" -> drop, no decode
   decode -> SameSource? drop : apply                            // unchanged otherwise
  • New public interfaces in Abstractions: IEventSourceMatcherProvider.CreateSourceMatcher(Uri) builds a matcher once per subscriber; IEventSourceMatcher.IsFromSource(ReadOnlyMemory<byte>) checks one payload.
  • CacheEventFormatter implements it (Source); the CloudEvents formatter implements it when it wraps a JSON formatter (source). Both use one Utf8JsonReader-based matcher that compares with ValueTextEquals (JSON escapes handled) and allocates nothing.
  • A message counts as own only when the decode would drop it: a case-only difference, a missing or repeated source, or malformed JSON still goes through decode and SameSource. Custom formatters without the capability keep decode-then-compare.
  • Messages are still decoded while Debug or Trace logging is on, so the own-message logs keep the event id.
  • GetOrAddResultRedisIntegrationTests.A_redis_hit_allocates_about_what_get_or_add_does moves to a new non-parallel RedisAllocation collection with the same Redis fixture, so its allocation count no longer includes other tests running in parallel.

Evidence

1000 own messages: 1751 -> 0 bytes per own message.

37 new cases: own skipped with 0 decodes, foreign applied, escaped, non-ASCII and case-only sources, Source after Data or missing, custom formatter, CloudEvents, allocation probe. net10.0 4364 passed, net8.0 4322 passed; the moved test passed 6/6 full runs.

Merge Danger

Door: two-way; the wire format is unchanged.

Blast Radius: Pub/Sub readers. An own message that is itself malformed no longer logs the "Event invalid" warning when Debug and Trace are off. The CloudEvents formatter test-decodes a sample payload once per subscriber when the topic is created.

@github-actions github-actions Bot added the needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md) label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

🔎 Maintainer heads-up: automated triage flagged this PR as potentially material, so it may need a signed CLA in addition to the DCO sign-off.

Strong signals

  • adds public API surface (PublicAPI.Unshipped.txt in src/UiPath.Caching.Abstractions, src/UiPath.Caching.CloudEvents, src/UiPath.Caching)

This is advisory only — the bot does not decide. Please judge against the CLA criteria (material, product-critical, patent-sensitive, corporate contributor, broad commercial use). Note that thresholds can be gamed by splitting PRs, so use your judgement.

  • If a CLA is needed → add the cla-required label (a contributor comment with signing steps is posted automatically).
  • If it is not needed → replace needs-cla-review with cla-not-required so later pushes don't re-flag it.

@cosmin-staicu
cosmin-staicu force-pushed the feat/pubsub-skip-own branch 2 times, most recently from 9bb74cd to 8566125 Compare November 2, 2026 16:29
@cosmin-staicu
cosmin-staicu force-pushed the feat/pubsub-skip-own branch 2 times, most recently from 09ead8f to fe736a6 Compare November 3, 2026 09:55
@cosmin-staicu
cosmin-staicu force-pushed the feat/skip-own-events branch 2 times, most recently from 292dd31 to 978f2a8 Compare November 3, 2026 10:13
@cosmin-staicu
cosmin-staicu force-pushed the feat/pubsub-skip-own branch 2 times, most recently from 57fe475 to b2330cf Compare November 3, 2026 13:33
@cosmin-staicu
cosmin-staicu force-pushed the feat/skip-own-events branch 2 times, most recently from 8f93501 to f003b9b Compare November 3, 2026 13:53
@cosmin-staicu cosmin-staicu self-assigned this Nov 3, 2026
@cosmin-staicu cosmin-staicu added cla-not-required Maintainer reviewed: no CLA required for this contribution and removed needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md) labels Nov 3, 2026
Base automatically changed from feat/skip-own-events to main November 3, 2026 15:57
A Pub/Sub subscriber also receives what its own node publishes. Each such
message went through a full System.Text.Json decode, about 1.75 KB, and was
then dropped by SameSource. A Pub/Sub message is a single value, so unlike
the stream entries it cannot carry the source beside the payload.

Formatters get an optional public capability, IEventSourceMatcherProvider,
which builds an IEventSourceMatcher for one source. The matcher reads the
payload with Utf8JsonReader, finds the top-level source property wherever it
is, and compares its unescaped text with the node's source, allocating
nothing. The subscriber builds the matcher once and drops a matching message
without decoding it. The wire format is unchanged.

The matcher only reports a message as own when the decode would drop it:
- It compares against the source as this node writes it (OriginalString)
  and its normalized form. Each text is kept only when the formatter's own
  decode of a probe payload that carries it gives an event SameSource
  counts as this node's.
- A case-only difference, a missing, null or nested source, a second
  source property (the last one wins the decode), trailing content and
  malformed JSON are not matched. Those messages are decoded and compared
  as before.
- While the subscriber's debug or trace log is on, every message is
  decoded, so the own-message logs keep the event id.

CacheEventFormatter matches "Source". The CloudEvents formatter matches the
structured-mode "source" attribute when it wraps a JsonEventFormatter; it
writes the same structured JSON, and its fast decode still allocates the id,
the data and the wrapper per own message. The matcher source is linked into
UiPath.Caching.CloudEvents. A custom formatter without the capability keeps
decode-then-compare.

The Streams reader is unchanged. A stream entry without the src field comes
from a 3.0 writer, so it is another node's except for a node's own
pre-upgrade backlog.

An allocation probe over 1000 own messages goes from 1751 bytes per message
(decoded) to 0.

Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

@cosmin-staicu
cosmin-staicu merged commit 2c859a9 into main Nov 3, 2026
11 checks passed
@cosmin-staicu
cosmin-staicu deleted the feat/pubsub-skip-own branch November 3, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-not-required Maintainer reviewed: no CLA required for this contribution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants