Skip to content

ci: alert on daily signature update failures, fix Zscaler parsing - #317

Merged
liquidsec merged 3 commits into
devfrom
feat/ci-failure-alerts
Oct 19, 2026
Merged

liquidsec merged 3 commits into
devfrom
feat/ci-failure-alerts

Conversation

@liquidsec

@liquidsec liquidsec commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #303.

Alert on daily signature update failures

New workflow signature-update-alerts.yml triggers on workflow_run when Daily Signature Update completes:

  • Failure, timeout or startup failure: opens an issue titled "Daily signature update failing" (label signature-update-failure) with the run link, trigger, commit, and failed jobs and steps. If that issue is already open, it adds a comment instead, so a failure that keeps happening stays in one issue.
  • Success: if the issue is open, comments "Recovered in " and closes it.

It only uses the gh and jq that come with the runner, and its permissions are actions: read and issues: write. workflow_run only fires from the default branch, so this goes live once it reaches stable.

Count provider crashes as errors

_update_provider caught any exception from a provider and returned no errors, so a crashing provider never counted toward ERROR_THRESHOLD and kept its stale data while the run passed. A crash now counts as one error, and the provider's existing data is still kept. The crash log line also printed <property object> instead of the provider name, which is fixed too.

The error list is now written to the job summary, so the run link in an alert opens on a readable list.

Fix Zscaler

Zscaler's CENR feed gained a top-level svpnIPs list next to the per-cloud sections. The parser called .items() on it and crashed on every run, so Zscaler currently has 0 CIDRs stored. The parser now skips any value that isn't a per-cloud section. All 176 of those IPs are already inside the published ranges. Against the live feed it now returns 682 CIDRs.

Context: recent failures

  • Daily update, 2026-10-19: asndb returned HTTP 500 on every lookup for about two minutes, giving 270 errors against a threshold of 10. The threshold blocked the commit as designed, and asndb has since recovered. This is the kind of failure the new alert would have reported.
  • Windows x86 wheel on stable, 2026-10-18: pyo3 0.27 can't build against Python 3.15. This is already fixed on dev by build(deps): bump pyo3 to 0.29 for Python 3.15 #310.

Successful runs currently have 6–9 errors against the threshold of 10. Besides Zscaler, most of them repeat every day: asndb lookups for UKMoD, Baidu, Nhncloud and IBM (probably outdated org IDs) and Salesforce's missing v2fly domains. Those are left for a follow-up.

Pin Python 3.14 where the dev group is installed

setup-python's 3.x now resolves to 3.15 on some runners. blasthttp 0.10.0 has no cp315 wheels, so uv builds it from source, and that build fails in openssl-sys. This is what made Pipeline Tests fail intermittently on this PR and #316. Pipeline Tests and the PyPI publish job are pinned to 3.14 until blasthttp ships 3.15 wheels. The publish pin matters for #305: without it, the release could randomly fail before reaching PyPI.

Adds a workflow_run watcher on Daily Signature Update that opens a
labelled issue on failure, comments on it while failures continue, and
closes it on the next success.

Also count a provider that crashes during update as an error. It was
previously dropped with zero errors, so it never counted toward
ERROR_THRESHOLD and its data went stale silently. The error list is now
written to the job summary for the alert's run link.

Closes #303
The feed gained a top-level "svpnIPs" list next to the per-cloud dicts.
Calling .items() on it raised every day, so Zscaler has had no CIDRs
(0 stored). The listed IPs are all inside the published ranges, so they
are skipped rather than added.
@liquidsec liquidsec self-assigned this Oct 19, 2026
setup-python's 3.x now resolves to 3.15 on some runners, and blasthttp
0.10.0 has no cp315 wheels, so uv falls back to a source build that
fails in openssl-sys. Pin Pipeline Tests and the PyPI publish job, the
two places that install the dev group, until blasthttp ships 3.15 wheels.
@liquidsec
liquidsec merged commit 2c0c323 into dev Oct 19, 2026
20 checks passed
@liquidsec
liquidsec deleted the feat/ci-failure-alerts branch October 19, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants