Skip to content

Follow-ups to #14208: doc fixes, OTLP AI evaluation tests and e2e download hardening - #14209

Merged
wu-sheng merged 1 commit into
masterfrom
fix/tal-followups
Nov 10, 2026
Merged

wu-sheng merged 1 commit into
masterfrom
fix/tal-followups

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

Fix the doc inaccuracies and test gaps left after #14208

These are the follow-ups #14208 left out of scope, plus two gaps in its own docs.

Docs

  • tal.md: blocks or rules that share a metric must also agree on its labelLimit (TALMetricDeclaration.equals compares it); an omitted one counts as 50.
  • tal-otlp.md: the meta table now states what the receiver does: serviceInstance falls back to service.instance, a start or end time left at 0 is replaced by the other one (or by the arrival time when both are 0), and latency is clamped to 0..Integer.MAX_VALUE.
  • trace-sampling.md: the setting is named default.rate throughout, the duration.rate and default.rare typos are fixed, and the example rates are 3500 / 5500 (35% / 55% at the 1/10000 precision) instead of 35 / 55.
  • virtual-genai.md, trace-analyzer.md: AI evaluation of natively stored OTLP spans needs service.layer=OTLP_GENERAL, the layer otlp-ai-evaluation is declared for. The Virtual GenAI metrics and the Zipkin-path evaluation don't need it. trace-analyzer.md's instance row names the service.instance fallback. The changelog's otlp-ai-evaluation line no longer says every OpenTelemetry rule needs OTLP_GENERAL.
  • new-monitoring-feature skill: iOS does get service_cpm and the other core OAL metrics, from IOSHTTPSpanListener. Browser has its own OAL metrics (browser_app_*), and the anti-pattern note now points at the mini-program pattern.

Code

  • AliasRegistrar.hostOf, used by the gateway check, checks the port in place instead of through a substring and a character stream. The result is the same for every input: an old-vs-new run over 2,327,680 addresses (every UTF-16 character as the port, plus 2 million random strings) found no difference. A new AliasRegistrarTest case pins the forms, including the empty port and non-ASCII digits.

Tests for documented behaviour of the OTLP AI evaluation that had no dedicated test:

  • OtlpAIEvaluationAnalysisTest runs the shipped otlp-ai-evaluation rule through TraceAnalyzerServiceImpl:
    • another rule's drop vote does not stop the sample;
    • a span sent again is sampled once while the dedup window remembers it, and twice with the window off;
    • a failed dispatch fails only that rule, and a rule whose source dispatch fails has its persist {} vote ignored, with a control showing the vote counts when the dispatch succeeds.
  • OTLPAnalysisBeforeStorageHandlerTest: a span that a span listener vetoes still reaches the TAL analysis, and only the other span is stored.

Four mutations were checked, each making its matching test fail: stopping at the first drop vote, counting a failed rule's vote, turning dedup off, and skipping the analysis of a vetoed span.

E2E

  • download-mysql.sh used curl -Lo, which exits 0 on an HTTP 429 and saves the error body as the driver jar, and had no timeout. It now makes up to five attempts, 5 seconds apart, each with --fail and --max-time 60, and exits non-zero when all fail. After curl is installed, the download takes at most about 320 seconds, and if every attempt fails the OAP does not start, since the entrypoint runs it with &&. The attempts are retried by the script, not by curl's --retry: that sleeps for as long as a 429's Retry-After asks, and with --retry-max-time added, curl 8.19 exited 0 without writing the file when Retry-After exceeded it.
  • Tested against a local server answering 429 with Retry-After: 1000 (5 attempts, exit 1 after 20 seconds, no file) and one that never answers (exit 1 after 5 timed-out attempts), and inside the OAP base image eclipse-temurin:25-jre, where it installed curl 8.18 and downloaded the jar.
  • Why: in Move the native trace analysis from Java listeners to TAL rules (SWIP-17) #14208's run 42756183940, the Zipkin BanyanDB job's OAP never became healthy within about 11.5 minutes, while another job got HTTP 429 from Maven Central. A download stalled at startup (curl had no timeout) is a possible explanation, though a 429 alone would not have stalled the old curl -Lo, and no OAP log was collected to confirm it.

Verification

  • Full unit-test suite (4,163 tests, 0 failures, 0 errors, 5 skipped), checkstyle, the -Pall build with javadoc, and the license-header check of every tracked file pass.

  • Add a unit test to verify that the fix works.

  • Explain briefly why the bug exists and how to fix it.

  • If this pull request closes/resolves/fixes an existing issue, replace the issue number. Not applicable; there is no issue.

  • Update the CHANGES log.

@wu-sheng
wu-sheng requested a balanced review from Copilot November 10, 2026 08:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues
What changed in this PR

Follow-up changes after #14208 to correct documentation, close test gaps around OTLP AI evaluation / trace analysis behavior, and harden an e2e dependency download step.

Changes:

  • Hardened MySQL driver download in e2e setup with bounded retries and curl failure handling.
  • Added unit tests for OTLP AI evaluation behavior and for analysis running even when a span is vetoed from storage.
  • Optimized and expanded coverage for gateway host parsing, plus multiple documentation corrections.
File Description
test/​e2e-v2/​script/​prepare/​setup-oap/​download-mysql.sh Adds retry + timeout + --fail to prevent saving HTTP error bodies as a jar.
oap-server/​server-receiver-plugin/​otel-receiver-plugin/​src/​test/​java/​org/​apache/​skywalking/​oap/​server/​receiver/​otel/​otlp/​OTLPAnalysisBeforeStorageHandlerTest.java New test asserting TAL analysis runs even if a span listener vetoes persistence.
oap-server/​analyzer/​trace-analyzer/​src/​test/​java/​org/​apache/​skywalking/​oap/​trace/​analyzer/​v2/​provider/​span/​OtlpAIEvaluationAnalysisTest.java New tests for shipped otlp-ai-evaluation behavior (dedup, failures, vote handling).
oap-server/​analyzer/​trace-analyzer/​src/​test/​java/​org/​apache/​skywalking/​oap/​trace/​analyzer/​v2/​provider/​span/​AliasRegistrarTest.java Adds cases covering host parsing edge cases, including empty/non-ASCII ports.
oap-server/​analyzer/​trace-analyzer/​src/​main/​java/​org/​apache/​skywalking/​oap/​trace/​analyzer/​v2/​provider/​span/​AliasRegistrar.java Replaces substring+stream digit check with in-place loop.
docs/​en/​setup/​service-agent/​virtual-genai.md Documents service.layer=OTLP_GENERAL requirement for native OTLP AI evaluation only.
docs/​en/​setup/​backend/​trace-sampling.md Fixes setting name to default.rate and corrects examples/typos.
docs/​en/​setup/​backend/​trace-analyzer.md Clarifies OTLP AI evaluation layer scoping and service.instance fallback.
docs/​en/​concepts-and-designs/​tal.md Documents labelLimit compatibility requirement when sharing metrics.
docs/​en/​concepts-and-designs/​tal-otlp.md Clarifies OTLP meta fallbacks and timestamp/latency normalization behavior.
docs/​en/​changes/​changes.md Updates changelog to capture doc fixes and metadata behavior.
.claude/​skills/​new-monitoring-feature/​SKILL.md Corrects guidance about browser/iOS metrics sources and anti-pattern note.

Review effort: Lite

Comment thread docs/en/setup/backend/trace-sampling.md Resolved
@wu-sheng
wu-sheng requested a review from wankai123 November 10, 2026 09:43
@wu-sheng wu-sheng added this to the 11.2.0 milestone Nov 10, 2026
@wu-sheng wu-sheng added backend OAP backend related. enhancement Enhancement on performance or codes labels Nov 10, 2026
wankai123
wankai123 previously approved these changes Nov 10, 2026
…nload hardening

Docs:
- tal.md: blocks or rules that share a metric must also agree on its labelLimit; an omitted one counts as 50.
- tal-otlp.md: the meta table states the service.instance fallback of serviceInstance, the start and end time
  fallbacks, and the latency clamp.
- trace-sampling.md: the setting is named default.rate throughout; the duration.rate and default.rare typos are
  fixed, and the example rates are 3500 and 5500 at the 1/10000 precision.
- virtual-genai.md, trace-analyzer.md: AI evaluation of natively stored OTLP spans needs service.layer=OTLP_GENERAL,
  the layer otlp-ai-evaluation is declared for; the Virtual GenAI metrics and the Zipkin-path evaluation do not.
  trace-analyzer.md's instance row names the service.instance fallback. The changelog's otlp-ai-evaluation line no
  longer says every OpenTelemetry rule needs OTLP_GENERAL.
- new-monitoring-feature skill: iOS does get service_cpm and the other core OAL metrics, from IOSHTTPSpanListener;
  browser has its own OAL metrics (browser_app_*); the anti-pattern note points at the mini-program pattern.

Code:
- AliasRegistrar.hostOf checks the port in place instead of through a substring and a character stream, with the
  same result for every input; a new AliasRegistrarTest case pins it.

Tests:
- OtlpAIEvaluationAnalysisTest runs the shipped otlp-ai-evaluation rule through TraceAnalyzerServiceImpl: another
  rule's drop vote does not stop the sample, a span sent again is sampled once while the dedup window remembers it,
  and a failed dispatch fails only its rule, whose persist vote is then not counted.
- OTLPAnalysisBeforeStorageHandlerTest: a span that a span listener vetoes still reaches the TAL analysis, and only
  the other span is stored.

E2E:
- download-mysql.sh makes up to five download attempts, 5 seconds apart, each with curl --fail and --max-time 60,
  and exits non-zero when all fail, so an HTTP error such as a 429 is not saved as the driver jar. After curl is
  installed, the download takes at most about 320 seconds, and if every attempt fails the OAP does not start. curl's
  --retry is not used: it sleeps for as long as a 429's Retry-After asks.
@wu-sheng
wu-sheng merged commit b68696f into master Nov 10, 2026
483 of 487 checks passed
@wu-sheng
wu-sheng deleted the fix/tal-followups branch November 10, 2026 12:30
Moutia-Ben-Yahia pushed a commit to HotOpenSourcing/skywalking that referenced this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend OAP backend related. enhancement Enhancement on performance or codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants