[kezhenxu94]

Fix OAP running out of memory when receiving Envoy metrics from a large mesh

• Add a unit test to verify that the fix works.
• Explain briefly why the bug exists and how to fix it.

PrometheusMetricConverter#escapedName replaced . and / with _ through a Guava LoadingCache capped at 1,000 entries. Envoy metric names embed the upstream cluster name (e.g. cluster.outbound|9080||reviews.default.svc.cluster.local.upstream_rq_total), so a large mesh has far more distinct names than the cache can hold. The cache kept missing, and every miss ran the regex and evicted an entry while holding a cache segment lock shared by all Envoy metrics streams.

In a thread dump from an affected OAP, about 1,850 of 2,000 grpc-default-executor threads were parked on that lock in escapedName within about 2 minutes of startup, and 16.6% of all allocations came from the cache's loader. The threads and their pending messages piled up until the container was OOMKilled.

The fix drops the cache and the regex and uses String.replace(char, char), which needs no lock and returns the same string without allocating when there's nothing to replace.

JMH comparison of the old cache and the new code (JMH 1.37, JDK 21), with Envoy-style names cluster.outbound|9080||svc-N.ns-M.svc.cluster.local.upstream_rq_total. 200 distinct names fit in the old cache; 20,000 don't:

Threads  Names   Old cache                 New replace
1        200      40 ns/op,  24 B/op        52 ns/op, 112 B/op
1        20000   364 ns/op, 816 B/op        70 ns/op, 119 B/op
8        200     452 ns/op                  82 ns/op
8        20000  3356 ns/op                  90 ns/op

The old cache contends even when every lookup hits (452 ns with 8 threads and 200 names): a size-capped Guava cache records each read and drains those records under the segment lock. The only case where the new code is slower is a single uncontended thread with names that fit in the cache (+12 ns, about +90 B per call), which is small next to the rest of the per-metric conversion. The benchmark reuses the same name strings, so their hash codes are cached; in OAP each name is a fresh string from protobuf, which the old cache had to hash on every lookup.

Benchmark source

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 3, time = 2)
@Measurement(iterations = 5, time = 2)
@Fork(1)
@State(Scope.Benchmark)
public class EscapedNameBenchmark {
    private static final Pattern P = Pattern.compile("[/.]");
    private static final LoadingCache<String, String> CACHE = CacheBuilder.newBuilder().maximumSize(1000)
        .build(new CacheLoader<String, String>() {
            @Override
            public String load(String name) {
                return P.matcher(name).replaceAll("_");
            }
        });

    @Param({"200", "20000"})
    int distinctNames;

    String[] names;

    @Setup
    public void setup() {
        names = new String[distinctNames];
        for (int i = 0; i < distinctNames; i++) {
            names[i] = "cluster.outbound|9080||svc-" + i + ".ns-" + (i % 50) + ".svc.cluster.local.upstream_rq_total";
        }
    }

    private String next() {
        return names[ThreadLocalRandom.current().nextInt(names.length)];
    }

    @Benchmark
    public String oldCache() throws Exception {
        return CACHE.get(next());
    }

    @Benchmark
    public String newReplace() {
        String n = next();
        return n.replace('.', '_').replace('/', '_');
    }
}

• Update the CHANGES log.