Skip to content

fix(kraken): count a recovered standing-charge 404 once and catch undecodable REST bodies (#5584) - #5641

Merged
springfall2008 merged 4 commits into
mainfrom
fix/kraken-rest-failure-accounting-5584
Oct 24, 2026
Merged

springfall2008 merged 4 commits into
mainfrom
fix/kraken-rest-failure-accounting-5584

Conversation

@springfall2008

Copy link
Copy Markdown
Owner

This is an automated draft PR generated from issue #5584 — a maintainer should review it before merging.

Fixes #5584 (points 1 and 2; point 3 is the duplicate tracked in #5573 / draft PR #5636 and is deliberately not touched here)

Summary

  • Point 1 — expected 404 no longer counted as a failure. async_fetch_standing_charges() used to bump failures_total on every non-200, before checking whether the status was the expected 404/410. For E.ON Next TOU tariffs, where the applicableStandingCharges GraphQL fallback works, that meant one extra failure every cycle with nothing wrong. The fix uses the guard the rates path already has (async_fetch_rates, from ed8db21a): take a snapshot of failures_total before the fallback, and add one only if the fallback returns None without having counted a failure itself. The Warn: ... Standing charges HTTP log now fires only on the paths that really fail. The expected 404 logs only the existing info line about falling back.
  • Point 2 — HTTP 200 with a body that doesn't parse. _fetch_rates_rest() and async_fetch_standing_charges() now catch ValueError as well as (aiohttp.ClientError, asyncio.TimeoutError). json.JSONDecodeError is a subclass of ValueError. The GraphQL path in the same file (async_graphql_query) already catches it this way. Before this change, a 200 whose body claimed to be JSON but didn't parse escaped run(). ComponentBase.start() caught it, but the rest of that cycle was skipped (dispatches, sensor publishing, import/export wiring), and startup backoff kicked in on the first path. Now the bad body is logged and counted like any other failed fetch. In _fetch_rates_rest it returns the existing (None, None) network-error result, so async_fetch_rates handles it the way it already handles network errors.

Testing

  • coverage/run_pre_commit: all hooks passed (ruff, black, cspell, markdownlint, …), and the quick suite it runs ended All tests passed (4 slow tests skipped).
  • tools/triage_test.sh kraken: passes with the fix.
  • Fails without the fix: I stashed only kraken.py (git stash push -u -- apps/predbat/kraken.py). The kraken runner stops at its first failure, so I re-ran it once per failure, temporarily disabling the test that had just failed (renames reverted before committing). Without the fix, each of the four new behaviour tests failed:
    • test_fetch_rates_undecodable_200_is_counted_not_raised: JSONDecodeError: Expecting value
    • test_fetch_standing_charges_404_graphql_failure_counts_once_not_twice: got 2
    • test_fetch_standing_charges_404_recovered_by_graphql_is_not_a_failure: got 1
    • test_fetch_standing_charges_undecodable_200_is_counted_not_raised: JSONDecodeError: Expecting value
  • Two guard tests pass both with and without the fix, as intended. They pin behaviour that must not change: test_fetch_standing_charges_404_graphql_empty_counts_one_failure, and a new failures_total == 1 assertion added to the existing test_fetch_standing_charges_transient_error_does_not_fall_back_to_graphql (429/500/503).

Notes

  • The "snapshot before the fallback, count only if it didn't" guard now exists twice: once in async_fetch_rates and once in async_fetch_standing_charges. I kept the copies separate instead of adding a helper, because each is three lines and a shared helper would mean changing the rates path too, which this issue doesn't need.
  • Possible overlap with fix(kraken): standing charge must cover now before it is used #5636: that PR edits the results[0] pick just below the hunk changed here, so there could be a small textual merge conflict depending on which merges first. The two changes don't depend on each other's logic.
  • Out of scope: a 200 whose body parses as JSON but isn't an object (for example a bare list) would still raise AttributeError at data.get(...). The issue didn't report that case, so I didn't widen the change for it.
  • Impact check (GitNexus): async_fetch_standing_charges and _fetch_rates_rest each have one direct caller, both inside the Kraken run flow, so risk is LOW. detect_changes also reports low risk.

🤖 Generated with Claude Code

…ecodable REST bodies (#5584)

Mirror the rates path's guard in async_fetch_standing_charges: a 404/410 whose
GraphQL applicableStandingCharges fallback succeeds no longer bumps
failures_total, and a fallback that already counted its own failure is not
counted twice. Add ValueError (JSONDecodeError) to the REST except clauses in
_fetch_rates_rest and async_fetch_standing_charges so a 200 with a malformed
body is logged and counted instead of escaping run().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@springfall2008 springfall2008 self-assigned this Oct 19, 2026
@springfall2008 springfall2008 added BOT_REVIEW Trigger an autotriage BOT_CLEANUP Trigger: bot should address PR review feedback and CI failures, then commit and push labels Oct 19, 2026

@springfall2008 springfall2008 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated comment from the triage bot.

Reviewed 82e05be (PR head, based on main 697c44c): 1 finding posted inline, nothing else in the body. Otherwise sound.

What was verified by reading the PR's tree (nothing executed, no tests run):

  • The count-once guard added around the standing-charges GraphQL fallback matches the existing rates-path guard (apps/predbat/kraken.py:970-977) and is correct for every async_graphql_query() outcome: a fallback that counts its own failure is not counted twice, an empty-but-uncounted fallback counts once, and a recovered 404/410 counts zero. It also compensates the one async_graphql_query() None return that does not increment (apps/predbat/kraken.py:413, a 200 whose body has no data).
  • The ValueError addition catches json.JSONDecodeError in both REST paths. In _fetch_rates_rest() the (None, None) return is the existing network-error sentinel and is counted once in async_fetch_rates() (apps/predbat/kraken.py:982-984); on the authenticated-retry path the existing handling (apps/predbat/kraken.py:948-954) keeps the earlier 404/410 status so the GraphQL fallback still runs.
  • Moving the Warn log and failure count out of the inner response block keeps every non-recovered non-200 counted exactly once (429/500/503 land on apps/predbat/kraken.py:1086-1088), preserving the behaviour the transient-error test pins.
  • The new tests are collected automatically: run_kraken_tests() (apps/predbat/tests/test_kraken.py:2426) discovers test_* functions via dir(), so no TEST_REGISTRY change is needed. They mutate no shared fixture state. The red/green claims in the PR body are consistent with the code as written.
  • failures_total is diagnostic-only (an account_status attribute, not a liveness input), so no docs change is warranted for the counter's corrected semantics; the docs/components.md wording stays true.
  • #5636 is still open, so the PR body's merge-conflict note about results[0] stands.

The one remaining exposure in the same class as this PR's fix is posted inline.

Comment thread apps/predbat/kraken.py Resolved
@springfall2008 springfall2008 removed the BOT_REVIEW Trigger an autotriage label Oct 19, 2026
@springfall2008

Copy link
Copy Markdown
Owner Author

Automated reply from the triage bot (PR cleanup run). Nothing has been pushed.

Branch sync: already up to date with main (697c44c). CI on 82e05be is green.

Inline finding (r4319741971, a 200 whose JSON is not an object): checked against the code and it holds. await response.json() succeeds for [], null or a bare string, and then data.get(...) raises AttributeError. Neither REST path catches that. I fixed it locally, but the fix is not pushed (reason below). The local fix:

  • apps/predbat/kraken.py _fetch_rates_rest(): after each .json(), if not isinstance(data, dict) logs a Warn: and returns the existing (None, None) network-error sentinel. async_fetch_rates() already counts that once. On the authenticated-retry path it still keeps the earlier 404/410, so the GraphQL fallback runs.
  • apps/predbat/kraken.py async_fetch_standing_charges(): the same guard on the 200 path, after the non-200 block. It logs, counts one failure and returns None.
  • I used an explicit isinstance guard rather than adding AttributeError to the except tuple. That matches fix(givtcp): treat a 200 REST response that is not a JSON object as a failed read (#2828) #5619's GivTCP treatment, and it avoids swallowing an unrelated AttributeError from a real bug.
  • Two new tests in apps/predbat/tests/test_kraken.py cover [], None, "Bad gateway" and 42 on both paths: test_fetch_rates_non_object_200_is_counted_not_raised and test_fetch_standing_charges_non_object_200_is_counted_not_raised.
  • Red/green: without the kraken.py change, the rates test fails with 'list' object has no attribute 'get'. The runner stops at the first failure, so the standing-charges test wasn't reached on the red run. It shares the same unguarded data.get(), but its red result is not confirmed. With the fix, tools/triage_test.sh kraken passes all 116 tests.
  • GitNexus impact: _fetch_rates_rest and async_fetch_standing_charges are both LOW risk, each with one direct caller (Kraken run).

Why it wasn't pushed: coverage/run_pre_commit ran every hook clean (ruff, black, cspell, markdownlint, …). Its quick suite then failed in octopus_url / day_night_rates Tests 1 and 7 with Expected 16 rate entries (8 days), got 15. This branch changes only kraken.py and test_kraken.py. Tonight's cleanups on #5643, #5644, #5645 and #5646 hit the same failure, and #5645's run reproduced it on main. It looks like a wall-clock/DST dependency in that test around the 25 Oct BST→GMT change, but that is not confirmed. The cleanup rules don't allow pushing past a failed local gate, so the fix stays local. A later run can push it once day_night_rates passes again, or once an octopus-side fix merges.

The review thread above has no reply yet. It will get one when the change is pushed.

@springfall2008

Copy link
Copy Markdown
Owner Author

Automated cleanup failed to complete for this PR. Reason: The run exited cleanly but left 2 file(s) with uncommitted changes, so its fixes were never pushed. They are stashed in the bot's clone before the next flow runs. Failed at: triage_daemon.py:2161 in process_bot_cleanup_pr() (daemon 063549db). Not retrying automatically; remove BOT_FAILED and re-add BOT_CLEANUP to try again.

@springfall2008 springfall2008 added BOT_FAILED An automated triage/PR attempt failed; see the comment for details and removed BOT_CLEANUP Trigger: bot should address PR review feedback and CI failures, then commit and push labels Oct 19, 2026
@chalfontchubby chalfontchubby added BOT_CLEANUP Trigger: bot should address PR review feedback and CI failures, then commit and push and removed BOT_FAILED An automated triage/PR attempt failed; see the comment for details labels Oct 20, 2026
CI and others added 2 commits October 20, 2026 10:34
@springfall2008 springfall2008 removed the BOT_CLEANUP Trigger: bot should address PR review feedback and CI failures, then commit and push label Oct 20, 2026
@springfall2008
springfall2008 marked this pull request as ready for review October 20, 2026 19:03
@springfall2008
springfall2008 merged commit ccac645 into main Oct 24, 2026
2 checks passed
@springfall2008
springfall2008 deleted the fix/kraken-rest-failure-accounting-5584 branch October 24, 2026 10:48
@chalfontchubby chalfontchubby added the BOT_PR_OPENED Bot PR flow opened a draft PR for this ticket label Oct 25, 2026
@chalfontchubby chalfontchubby self-assigned this Oct 25, 2026
thewhale21 pushed a commit to thewhale21/batpred that referenced this pull request Oct 26, 2026
…es) and refresh entries fixed on main

Folded: GH#5686 (retention negative-delta), GH#5690 (dp2 dull-day calibration), GH#5691 (sigenergy onboard codes), GH#5693 (db row + now_utc_exact), GH#5694/springfall2008#5697 (.seconds wrap), GH#5695 (dead-tz checklist, post-springfall2008#5696), 5670/5699 (gateway EV plan/status wiring), GH#5702 (poll-trim), GH#5704 (FoxESS Back-up hold trap), GH#5705 (predheat test harness), GH#5706 (slow-charge cancels + ? marker), GH#5683 (template-sensor name/unique_id), GH#5688 (sharpen only).

Merge-refresh on main: springfall2008#5617, springfall2008#5622, springfall2008#5623, springfall2008#5639, springfall2008#5641, springfall2008#5646, springfall2008#5696, springfall2008#5697, springfall2008#5705, springfall2008#5725, springfall2008#5727, springfall2008#5735, springfall2008#5737, springfall2008#5750, springfall2008#5754, springfall2008#4745; fixes a PR-number typo in the HA-write row.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BOT_PR_OPENED Bot PR flow opened a draft PR for this ticket

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kraken REST fetches: expected 404 double-counted as failure, malformed-200 JSONDecodeError uncaught, standing charge picked with no validity check

2 participants