Repository navigation
fix(solar): Open-Meteo radiation is a preceding-hour mean, so stop placing the forecast 30 minutes late - #6237
chalfontchubby wants to merge 4 commits into
Conversation
…acing the forecast half an hour late Open-Meteo documents its hourly radiation as "averaged over the past hour": the value stamped t is the mean power from t-1h to t. gti_hourly_to_period_kwh() read each value as an instant at t and built the hour t..t+1 from the average of the samples at t and t+1, so every hour mixed in the previous one and the whole forecast sat half an hour late (too high in the afternoon, too low in the morning, worst on short winter days). Each sample now becomes the energy of the hour ending at its stamp, derated by its own temperature and wind. P10/P90 ratios and monthly shading follow the hour they belong to. The live Open-Meteo forecast and the annual estimate share this function, so both change; the annual estimate's totals barely move. Tests: a single sunny sample must land wholly in the hour ending at its stamp (main splits it 50/50 across two hours), one period per sample including across a gap, the shading month at a month boundary, and updated expectations in the Open-Meteo, annual weather and Solcast suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rom a midnight sample Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, and drop the annual stub hour Review follow-ups to the preceding-hour fix: - Temperature and wind are instants, so an hour is now derated with the average of the values at its two ends (its own sample's value when the previous hour is missing). - The annual estimate's first sample (00:00 on the window's first day) covers the hour before the window; that stub hour is now dropped so the previous day is not counted as a one-hour day in the monthly P10 ratios (it mattered for sites east of UTC). - Open-Meteo test comments no longer describe the removed trapezoid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
springfall2008
left a comment
There was a problem hiding this comment.
Automated comment from the triage bot.
Reviewed head 771b9fc5 (3 commits, dd847a7d..771b9fc5 on main da2b0a5b). No correctness defect found in the conversion: 0 findings inline, 1 note below on the PV-calibration side effect the PR defers to the maintainer.
What I checked
- The premise, against the Open-Meteo docs:
global_tilted_irradianceis "Total radiation received on a tilted pane as average of the preceding hour", solar radiation is backward-averaged by default,_instantvariants exist and are not requested, andtemperature_2mis "Instant". So reading each sample as the energy of the hour ending at its stamp is the documented convention. - The "other sources already correct" claim: Forecast.Solar's
wattskeys are period ends and its existing minute-by-minute fill already places the value on the period before the key; the Solcastperiod_endpath is untouched. No inconsistency is introduced. - Downstream of the now hour-keyed series:
minute_data(..., spreading=60)for the live path,publish_pv_stats'this_point >= midnight_todayguard,pack_and_store_forecastandpv_calibrationiterating from minute 0,executereading.get(pv_minute), andcalc_dawnresetting its per-day light latch. The new pre-midnight stub hour (minute −60..−1) therefore reaches no planner series and cannot change today's dawn or thepv_todaytotal; it only costs 60 dead entries inpv_forecast_minuteand in the debug dump. - Annual:
_window()returnsdateobjects, sostamp.date() < start_dateis a valid comparison;WeatherYear.pv_minutesalready discarded pre-midnight stamps, andselect_samplesbuilds candidate days from the calendar, so the filter loses no day the tool needs. - Tests are discriminating, not restatements: the impulse test, the month-edge shading test, the 11:00
period_startassertion and the annualhas_actual(2024-12-31)assertion each fail onmain(the first three by raising on a missing key or returning the misaligned hour). Every changed expectation intest_open_meteo.py/test_solcast.pyfollows from the shift; those two files otherwise only updated comments. - Maintainer rules: no new setting or entity, so no
CONFIG_ITEMS/APPS_SCHEMAor docs entry is owed; nogetattr, no direct file access, noComponentBaseboundary, no kernel-parity-relevant simulation change.
Note (not a defect): apps/predbat/solcast.py:1255 — clearing the stored evidence would not remove the lagged evidence
This is about the merge decision the PR defers, not the diff. The PR says clearing the stored calibration evidence on upgrade would avoid the ~2 weeks of over-correction. Verified in the checkout that it would not: pv_calibration_scan() derives the evidence from the recorder's own history (pv_forecast_history() for the uncalibrated forecast, plus the pv_today history) over pv_calibration_history_days (28 by default), not only from what the store accumulated. pv_calibration_update_evidence() deliberately drops last_success for a discarded store, so a cleared store re-scans the full 28 days from history — which holds the late forecasts — and rebuilds the same lagged evidence.
So the choice the maintainer actually faces is: accept the transient, or add a minimum evidence date so days recorded before the upgrade are ignored (the scan already keeps a day only when the history spans it, so a cutoff drops in cleanly). Two facts that bound it: only the per-slot shape is affected (the day factor is a scalar and unaffected), the shape is shrunk toward zero by shrink_slot 0.20, and weights halve every half_life 3.5 days, so evidence older than ~2 weeks carries under 6% weight — this is a partial reintroduction of the 30-minute tilt in the shoulder slots, decaying over about two weeks, not a step change in daily totals. pv_calibration_history_days is user-settable, so a site with a longer window carries it for longer.
Claude, posting on behalf of chalfontchubby.
Summary
Predbat's Open-Meteo solar forecast sits 30 minutes late: too low in the morning and too high in the afternoon. The cause is that Predbat reads each hourly radiation value as a reading at one instant, but Open-Meteo documents it as the average over the hour before its timestamp. This PR reads the values the way Open-Meteo documents them.
This affects everyone who uses
open_meteo_forecast, or the Open-Meteo backup for Forecast.Solar, and also the annual prediction tool, which shares the same conversion. The daily totals barely change; what moves is when in the day the energy is expected. That matters most in winter, when the day is short and a 30-minute shift is a larger part of it.What Open-Meteo says
From the Open-Meteo forecast API docs:
global_tilted_irradiance: "Total radiation received on a tilted pane as average of the preceding hour." (Hourly Parameter Definition)_instantvariant is available for anyone who wants instants. Predbat requests the default variable, not the_instantone.temperature_2mas "Instant".So a
global_tilted_irradiancevalue stamped 12:00 is the mean power from 11:00 to 12:00. A mean power over one hour, in kW, is that hour's energy in kWh.What the code did
gti_hourly_to_period_kwh()insolar_model.py(on main) said:It built the period starting at
tfrom the average of the samples stampedtandt+1h. Take the sample stamped 12:00, which Open-Meteo says is the energy of 11:00–12:00. The old code put half of it into the period starting 11:00 and half into the period starting 12:00. Every hour's energy was spread over itself and the hour after, so its centre moved from the middle of the hour to the end: 30 minutes late.The new test in
test_solar_model.py("a sample is the energy of the hour ENDING at its stamp") shows this most simply. It gives the function one sunny sample stamped 01:00, with dark samples either side. On main, half the energy lands in 00:00–01:00 and half in 01:00–02:00. With this PR, all of it lands in 00:00–01:00, where Open-Meteo says it belongs. This test fails on main.The fix
Each sample is now the energy of the hour that ends at its timestamp, returned under that hour's start time. In addition:
The ensemble API (used for the P10/P90 band) uses the same preceding-hour convention for radiation, so the band and the central forecast stay aligned.
Other sources checked, no change needed
wattskeys mark the end of each period, and the value is the average over the period. The existing handling already matches this.period_endis already handled correctly.Side effect: PV calibration (#6231)
PV calibration learns a correction from forecast compared with actual generation, over recent days. Evidence it recorded while the forecast was 30 minutes late will over-correct the fixed forecast until it ages out (weights halve every 3.5 days, so about two weeks). That makes the morning a little high and the afternoon a little low for a while after upgrading. Only the per-slot shape is affected, not the daily total, and the shape correction is already shrunk towards zero.
Clearing the stored evidence on upgrade would not avoid this.
pv_calibration_scan()rebuilds the evidence from the recorder's history overpv_calibration_history_days(28 by default), and that history still holds the late forecasts.Decision: accept the transient. It fades over about two weeks, or longer where
pv_calibration_history_dayshas been raised. While it lasts the calibrated forecast is no more wrong than it is today, when the forecast itself is 30 minutes late. The alternative was a minimum evidence date, so the scan ignores days recorded before the upgrade. That adds code for a problem that clears itself, so it is not done here.Tests
test_solar_model.py: the single-sample test above; one period per sample, including across a gap; aNonesample; a 23:00 period from a midnight sample; shading at a month boundary; per-hour band ratio alignment; temperature averaging.test_open_meteo.py,test_annual_weather.py(also a check that the annual estimate has no data for the day before its window; fails without the fix),test_solcast.py: expectations updated for the corrected alignment.coverage/run_pre_commitpasses, including the quick test suite.🤖 Generated with Claude Code