Skip to content

fix(solar): Open-Meteo radiation is a preceding-hour mean, so stop placing the forecast 30 minutes late - #6237

Open
chalfontchubby wants to merge 4 commits into
mainfrom
fix/open-meteo-preceding-hour-mean
Open

chalfontchubby wants to merge 4 commits into
mainfrom
fix/open-meteo-preceding-hour-mean

Conversation

@chalfontchubby

@chalfontchubby chalfontchubby commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Claude, posting on behalf of chalfontchubby.

Summary

Predbat's Open-Meteo solar forecast sits 30 minutes late: too low in the morning and too high in the afternoon. The cause is that Predbat reads each hourly radiation value as a reading at one instant, but Open-Meteo documents it as the average over the hour before its timestamp. This PR reads the values the way Open-Meteo documents them.

This affects everyone who uses open_meteo_forecast, or the Open-Meteo backup for Forecast.Solar, and also the annual prediction tool, which shares the same conversion. The daily totals barely change; what moves is when in the day the energy is expected. That matters most in winter, when the day is short and a 30-minute shift is a larger part of it.

What Open-Meteo says

From the Open-Meteo forecast API docs:

  • "Solar radiation is averaged over the past hour." (Solar Radiation Variables)
  • global_tilted_irradiance: "Total radiation received on a tilted pane as average of the preceding hour." (Hourly Parameter Definition)
  • "By default, the Open-Meteo API provides backward-averaged solar radiation to simplify energy calculations." An _instant variant is available for anyone who wants instants. Predbat requests the default variable, not the _instant one.
  • "Most weather variables are given as an instantaneous value for the indicated hour", and the hourly table lists temperature_2m as "Instant".

So a global_tilted_irradiance value stamped 12:00 is the mean power from 11:00 to 12:00. A mean power over one hour, in kW, is that hour's energy in kWh.

What the code did

gti_hourly_to_period_kwh() in solar_model.py (on main) said:

Open-Meteo returns point-in-time irradiance (W/m2) at the start of each hour, so the samples are integrated trapezoidally across each adjacent pair rather than treated as period energy.

It built the period starting at t from the average of the samples stamped t and t+1h. Take the sample stamped 12:00, which Open-Meteo says is the energy of 11:00–12:00. The old code put half of it into the period starting 11:00 and half into the period starting 12:00. Every hour's energy was spread over itself and the hour after, so its centre moved from the middle of the hour to the end: 30 minutes late.

The new test in test_solar_model.py ("a sample is the energy of the hour ENDING at its stamp") shows this most simply. It gives the function one sunny sample stamped 01:00, with dark samples either side. On main, half the energy lands in 00:00–01:00 and half in 01:00–02:00. With this PR, all of it lands in 00:00–01:00, where Open-Meteo says it belongs. This test fails on main.

The fix

Each sample is now the energy of the hour that ends at its timestamp, returned under that hour's start time. In addition:

  • Temperature and wind are instants, so the temperature derate for an hour uses the average of the readings at its two ends (or the sample's own reading when the previous hour is missing).
  • P10/P90 ratios (from the ensemble) and monthly shading factors follow the hour they belong to. Shading uses the month the hour starts in, which matters only for the hour that crosses midnight at the end of a month.
  • Annual prediction: the window's first sample (00:00 on the first day) covers 23:00–00:00 on the previous day. That one-hour stub is now dropped, so it is not counted as a short day in the monthly P10 ratios. The effect is small, but it could skew P10 for sites east of UTC.

The ensemble API (used for the P10/P90 band) uses the same preceding-hour convention for radiation, so the band and the central forecast stay aligned.

Other sources checked, no change needed

  • Forecast.Solar: the watts keys mark the end of each period, and the value is the average over the period. The existing handling already matches this.
  • Solcast: period_end is already handled correctly.

Side effect: PV calibration (#6231)

PV calibration learns a correction from forecast compared with actual generation, over recent days. Evidence it recorded while the forecast was 30 minutes late will over-correct the fixed forecast until it ages out (weights halve every 3.5 days, so about two weeks). That makes the morning a little high and the afternoon a little low for a while after upgrading. Only the per-slot shape is affected, not the daily total, and the shape correction is already shrunk towards zero.

Clearing the stored evidence on upgrade would not avoid this. pv_calibration_scan() rebuilds the evidence from the recorder's history over pv_calibration_history_days (28 by default), and that history still holds the late forecasts.

Decision: accept the transient. It fades over about two weeks, or longer where pv_calibration_history_days has been raised. While it lasts the calibrated forecast is no more wrong than it is today, when the forecast itself is 30 minutes late. The alternative was a minimum evidence date, so the scan ignores days recorded before the upgrade. That adds code for a problem that clears itself, so it is not done here.

Tests

  • test_solar_model.py: the single-sample test above; one period per sample, including across a gap; a None sample; a 23:00 period from a midnight sample; shading at a month boundary; per-hour band ratio alignment; temperature averaging.
  • test_open_meteo.py, test_annual_weather.py (also a check that the annual estimate has no data for the day before its window; fails without the fix), test_solcast.py: expectations updated for the corrected alignment.

coverage/run_pre_commit passes, including the quick test suite.

🤖 Generated with Claude Code

chalfontchubby and others added 3 commits November 16, 2026 18:45
…acing the forecast half an hour late

Open-Meteo documents its hourly radiation as "averaged over the past hour": the
value stamped t is the mean power from t-1h to t. gti_hourly_to_period_kwh()
read each value as an instant at t and built the hour t..t+1 from the average of
the samples at t and t+1, so every hour mixed in the previous one and the whole
forecast sat half an hour late (too high in the afternoon, too low in the morning,
worst on short winter days).

Each sample now becomes the energy of the hour ending at its stamp, derated by
its own temperature and wind. P10/P90 ratios and monthly shading follow the hour
they belong to. The live Open-Meteo forecast and the annual estimate share this
function, so both change; the annual estimate's totals barely move.

Tests: a single sunny sample must land wholly in the hour ending at its stamp
(main splits it 50/50 across two hours), one period per sample including across
a gap, the shading month at a month boundary, and updated expectations in the
Open-Meteo, annual weather and Solcast suites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rom a midnight sample

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, and drop the annual stub hour

Review follow-ups to the preceding-hour fix:
- Temperature and wind are instants, so an hour is now derated with the average of
  the values at its two ends (its own sample's value when the previous hour is missing).
- The annual estimate's first sample (00:00 on the window's first day) covers the hour
  before the window; that stub hour is now dropped so the previous day is not counted
  as a one-hour day in the monthly P10 ratios (it mattered for sites east of UTC).
- Open-Meteo test comments no longer describe the removed trapezoid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chalfontchubby chalfontchubby added the bug Something isn't working label Nov 16, 2026
@github-actions github-actions Bot added the BOT_REVIEW Trigger an autotriage label Nov 16, 2026

@springfall2008 springfall2008 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated comment from the triage bot.

Reviewed head 771b9fc5 (3 commits, dd847a7d..771b9fc5 on main da2b0a5b). No correctness defect found in the conversion: 0 findings inline, 1 note below on the PV-calibration side effect the PR defers to the maintainer.

What I checked

  • The premise, against the Open-Meteo docs: global_tilted_irradiance is "Total radiation received on a tilted pane as average of the preceding hour", solar radiation is backward-averaged by default, _instant variants exist and are not requested, and temperature_2m is "Instant". So reading each sample as the energy of the hour ending at its stamp is the documented convention.
  • The "other sources already correct" claim: Forecast.Solar's watts keys are period ends and its existing minute-by-minute fill already places the value on the period before the key; the Solcast period_end path is untouched. No inconsistency is introduced.
  • Downstream of the now hour-keyed series: minute_data(..., spreading=60) for the live path, publish_pv_stats' this_point >= midnight_today guard, pack_and_store_forecast and pv_calibration iterating from minute 0, execute reading .get(pv_minute), and calc_dawn resetting its per-day light latch. The new pre-midnight stub hour (minute −60..−1) therefore reaches no planner series and cannot change today's dawn or the pv_today total; it only costs 60 dead entries in pv_forecast_minute and in the debug dump.
  • Annual: _window() returns date objects, so stamp.date() < start_date is a valid comparison; WeatherYear.pv_minutes already discarded pre-midnight stamps, and select_samples builds candidate days from the calendar, so the filter loses no day the tool needs.
  • Tests are discriminating, not restatements: the impulse test, the month-edge shading test, the 11:00 period_start assertion and the annual has_actual(2024-12-31) assertion each fail on main (the first three by raising on a missing key or returning the misaligned hour). Every changed expectation in test_open_meteo.py / test_solcast.py follows from the shift; those two files otherwise only updated comments.
  • Maintainer rules: no new setting or entity, so no CONFIG_ITEMS/APPS_SCHEMA or docs entry is owed; no getattr, no direct file access, no ComponentBase boundary, no kernel-parity-relevant simulation change.

Note (not a defect): apps/predbat/solcast.py:1255 — clearing the stored evidence would not remove the lagged evidence

This is about the merge decision the PR defers, not the diff. The PR says clearing the stored calibration evidence on upgrade would avoid the ~2 weeks of over-correction. Verified in the checkout that it would not: pv_calibration_scan() derives the evidence from the recorder's own history (pv_forecast_history() for the uncalibrated forecast, plus the pv_today history) over pv_calibration_history_days (28 by default), not only from what the store accumulated. pv_calibration_update_evidence() deliberately drops last_success for a discarded store, so a cleared store re-scans the full 28 days from history — which holds the late forecasts — and rebuilds the same lagged evidence.

So the choice the maintainer actually faces is: accept the transient, or add a minimum evidence date so days recorded before the upgrade are ignored (the scan already keeps a day only when the history spans it, so a cutoff drops in cleanly). Two facts that bound it: only the per-slot shape is affected (the day factor is a scalar and unaffected), the shape is shrunk toward zero by shrink_slot 0.20, and weights halve every half_life 3.5 days, so evidence older than ~2 weeks carries under 6% weight — this is a partial reintroduction of the 30-minute tilt in the shoulder slots, decaying over about two weeks, not a step change in daily totals. pv_calibration_history_days is user-settable, so a site with a longer window carries it for longer.

@springfall2008 springfall2008 removed the BOT_REVIEW Trigger an autotriage label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants