Repository navigation
Conversation
On AWS m6i.16xlarge, 'metrics' takes the whole instance down during metadata
collection. The cause is not a slow perf and not a missing event: opening an
event that lands on fixed counter 3 faults the guest kernel.
Oops: general protection fault, maybe for address 0x1
x86_perf_event_update+0x48 <- intel_pmu_set_period <- x86_pmu_start
<- x86_pmu_enable <- __perf_event_enable <- _perf_ioctl
The guest advertises 4 fixed-purpose counters, which claims counter 3,
TOPDOWN.SLOTS, while lacking GLOBAL_CTRL_EN_PERF_METRICS -- so the kernel
programs a counter whose MSR the hypervisor never implemented. The faulting task
then dies inside x86_pmu_enable still holding perf's context lock and parks in D
state in perf_event_release_kernel, after which every perf_event_open on the
machine blocks in account_event/perf_event_alloc, in D state, ignoring SIGKILL.
RCU stalls and soft lockups follow and the instance leaves the network.
This is also why wrapping the probes in a shell-level timeout does not fix it: a
D-state task takes no signals, so the probes outlive their own timeout and the
machine is gone either way.
What reaches the fault is 'perf list', which calls perf_event_open per candidate
event to test support. Measured on the target, one probe at a time: instructions,
ref-cycles, OCR.READS_TO_CORE.LOCAL_DRAM and an over-subscribed fixed-counter
group all complete in about a second, the PEBS and TMA probes fail to parse in
10ms because those events are not exposed, and 'perf list --json' kills the box.
Both scriptPerfSupportedEvents and scriptPerfAllSupportedEvents run it, inside
the concurrent metadata batch, which is why one faulting script strands the other
39 and leaves only 'get architecture' -- the one script that never touches perf.
So on an affected target, enumerate events from sysfs instead. Reading sysfs
opens no events and cannot fault, but the reason to prefer it is stronger than
that: sysfs is the kernel's own list of events it will accept, so every event
absent from it -- including the topdown events at the root of the fault -- is
dropped from the metric definitions by the existing IsCollectable checks. Metric
collection therefore cannot walk into the same fault later either. The event set
is narrower than 'perf list' reports, which is the price of collecting anything
at all here; OCR.READS_TO_CORE.LOCAL_DRAM does work on this guest and is now
skipped along with the rest.
Affected targets are identified by the incoherence itself rather than by a model
or instance-type list: 4 or more fixed-purpose counters advertised while the
kernel exposes no slots or topdown event. Every comparable cell is coherent one
way or the other and keeps today's behavior -- bare-metal Ice Lake exposes slots,
m7i.24xlarge advertises 2 fixed counters, m5.24xlarge is Skylake. All three
collect metrics normally today, and the unit test pins all four cases.
The check reads only dmesg and sysfs, so it is safe on the very targets it is
meant to protect, and it runs before the metadata batch rather than as part of it
because the command it guards against is in that batch. It fails open: if the
counter count cannot be read, perf enumeration is kept, because wrongly narrowing
a healthy target's event set is the worse error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The syscalls telemetry script enumerated syscalls:sys_enter_* with a bare 'perf list'. That form also lists hardware events, and it decides what to list by calling perf_event_open on each candidate -- which on a guest whose PMU advertises a fixed counter the kernel cannot use faults the kernel, strands every later perf_event_open in D state, and takes the machine off the network. The sibling commit on this branch stops metrics metadata from doing the same thing; this is the same fault reached through a different door, in a script that guard cannot see. Observed on AWS m6i.16xlarge with the metadata fix in place: all 32 flame, lock, metrics, report and benchmark tests passed, then 'telemetry duration' -- which collects the kernel category by default, and so runs this script -- lost the instance mid-collection. The controller returned ssh's exit code 255 and every test after it failed with "Network unreachable". 'perf list tracepoint' reads tracefs and opens nothing, so it cannot trip the fault. On that same guest it returned in 0.01s even after the kernel had faulted and every perf_event_open was already blocking, which is as direct a demonstration as there is that it opens no events. It is also strictly less work for the same answer: verified both forms enumerate an identical set of 356 syscalls:sys_enter_* tracepoints. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Intel PMU driver renamed its boot output in 6.15. Through 6.14 it prints ... fixed-purpose events: 4 and from 6.15 on ... fixed-purpose counters: 4 The safety probe matched only the older wording, so on a current kernel it found nothing, reported the count as unknown, and failed open -- keeping perf enumeration on exactly the guests the gate exists to protect. The gate would have stopped working the moment the test images moved past 6.14, and silently: failing open looks identical to a healthy PMU. Found on a dedicated AWS m6i.16xlarge running 7.0.0-1012-aws, which carries the same broken vPMU as the CI cell -- version 2, 8 generic counters, 4 fixed-purpose counters, and no slots or topdown event exposed -- and where the probe as written reported "unknown". With both spellings matched it reports fixed_purpose_counters=4, slots_event_exposed=no, verified by running the extracted script under sudo on that host. That kernel also renames "event mask" to "global_ctrl mask" for the same value, confirming what the field is: the fixed-counter bitmap in the high nibble, the general-purpose bitmap in the low byte, PERF_METRICS at bit 48. Note the counter count is only available from dmesg -- cpu/caps/ exposes just max_precise and pmu_name -- so a --noroot run cannot read it and still fails open by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same as #720 but includes a patched version of excelize which had vulnerabilities in v2.11.0