[sonic16x]

{
    exitOnError: {
        enabled: true,
        filters: [
            { name: "quota", filter: /Too Many Requests/i, limit: 5 },
        ],
    }
}

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/gemini-testing/testplane@1343

commit: 8ef1395

[github-actions]

✅ Testplane browser-env run succeed

Report

[github-actions]

✅ Testplane E2E run succeed

Report

[shadowusr]

This looks hard to read & maintain. If we want some sort of integration test here, why not create the needed files in a separate fixture folder and copy them as needed. I think it would be simpler & cleaner.

[shadowusr]

I am a bit skeptical of this shape.

First, the object is 100% reduntant here and the wrong shape to use. If we choose to preserve the overall idea of this PR, I'd use something like this:

stopOnErrors: {
    rules: [
        { filter: /Service unavailable/, limit: 2 },
        { filter: text => text.includes('Something'), limit: 20 },
    ],
}

We could support passing exitOnErrors: false to make it easy to disable via env vars for example. Much simpler & cleaner IMO.

---

However, I would think more about the API shape... We already have patterns on reject. With a very simple shape: regexp array. And I think they strongly correlate. So what about this:

system: {
    patternsOnReject: myPatternsTemplate,
    maxFailures: 20, // Stops after any 20 errors
    maxFailures: { // OR, in object shape like this
        limit: 20,
        patterns: myPatternsTemplate, // <-- Easy reuse of the existing patterns
    }
}

This shape is even easier to work with & lets reuse existing patterns on reject collection that users already have! We can even consider defaulting to those patterns, but this might be unexpected for some users.

Let's discuss.