Skip to content

Add open-data example notebooks for interp2poi and interp2grid - #4

Open
matschaer wants to merge 11 commits into
mainfrom
feature/example-notebooks
Open

matschaer wants to merge 11 commits into
mainfrom
feature/example-notebooks

Conversation

@matschaer

@matschaer matschaer commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Adds four example notebooks to the documentation (new "Examples" page) illustrating grid_process.grid_interp.interp2poi and interp2grid with open data only. They replace the draft examples of the internal gridefix-process branch feature/sphinx-nb-display. Each notebook compares the interpolation methods visually and with a runtime benchmark table (first call vs. subsequent cached calls). On regular grids, nearest and linear are also run with try_reg=False (generic path) and compared with the default xarray path.

Notebook Data → target Methods
01 interp2poi wind gusts ICON-CH2 control VMAX_10M (MeteoSwiss OGD, native unstructured grid) → SwissMetNet stations measuring gusts, vs. observed hourly max gust linear, idw (k=4, weight_power=2), nearest_alt (r_search=1.7)
02 interp2poi temperature IFS-HRES 0.1° 2t (ECMWF open data, oldest run, 0–144 h) → all SwissMetNet stations; 14 stations from the Plateau to the peaks plotted vs. 10-min observations nearest, temp (z_fact=15, k=50, weight_power=0.75), temp_fixgrad (r_search=2)
03 interp2grid precipitation ERA5-Land 2024 yearly precipitation (Copernicus CDS) → RhiresY 1 km LV95 grid, vs. RhiresY nearest, idw (k=4, weight_power=2), linear
04 interp2grid temperature IFS-HRES 0.1° and ICON-CH2 2t, same valid time → swisstopo DHM25/200 grid over Valais (496×833) nearest, linear, idw (k=4, weight_power=2), temp_fixgrad

Details:

  • Notebooks are rendered with myst-nb and nb_execution_mode = "off": they are committed with their outputs, since open forecast data expire within days and notebook 03 needs a CDS account.
  • Their dependencies (earthkit-data with the CDS, ECMWF open data and MeteoSwiss OGD sources, earthkit-plots, eccodes-cosmo-resources-python, netcdf4, …) are in a new optional example dependency group (poetry install --with example), so CI does not install them. myst-nb is added to the doc group.
  • The POI notebooks adapt the grid_process data model through environment variables (GRID_PROCESS__data_model__poi__coord_from, ..._alt_from) instead of renaming the SwissMetNet metadata columns.
  • Model orography for the altitude-aware methods: HRES surface z at step 0 (IFS), HSURF from the horizontal constants (ICON-CH2); destination altitude from DHM25/200.
  • Data are downloaded at runtime; no raw data is committed. Requests to the ECMWF open data portal are retried with backoff. Attribution is given in each notebook (MeteoSwiss, ECMWF and Copernicus under CC BY 4.0, swisstopo, Natural Earth).
  • The notebooks do not modify the user's earthkit configuration; the Examples page explains how to enable the earthkit cache.
  • The notebook code was checked once with ruff, mypy and pylint (converted with nbconvert --to script); no tooling was added for this.

No breaking changes.


Related issue

None.


Checklist

  • Code is well documented and follows project conventions (linted, formatted)
  • CHANGELOG updated (if applicable)
  • License header is added to all new files — n/a: documentation files only (notebooks, rst)
  • For a new contributors, their name has been added to the AUTHORS file — n/a
  • No sensitive data or secrets are exposed
  • Unit tests added covering the new code — n/a: no library code changed; docs build checked locally with sphinx-build -W, ruff format --check and pytest pass

🤖 Generated with Claude Code

Mathieu Schaer and others added 10 commits October 28, 2026 15:56
Four notebooks in doc/examples illustrate the spatial interpolation functions
with open data (MeteoSwiss OGD, ECMWF open data, ERA5-Land, swisstopo DHM25),
comparing methods visually and with a runtime benchmark. They are rendered in
the documentation with myst-nb without execution. Their dependencies are in
the optional "example" dependency group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ekd.config.set() is saved to the global earthkit-data config file, so the
notebooks no longer set the cache policy. The examples page explains how to
enable the earthkit cache instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit conversions use grid_process.units_conv.ds_to instead of hard-coded
formulas. On regular source grids (IFS-HRES, ERA5-Land), the nearest and
linear methods are run and benchmarked with try_reg=True and try_reg=False,
and the differences between both paths are reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run lookup of the IFS notebooks sends many small requests to
data.ecmwf.int, which sometimes times out. Send them through a requests
session retrying with backoff on connection errors and transient HTTP
errors. The field downloads already retry within earthkit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the notebook with the 2024 ERA5-Land data from the Climate Data
Store and write its takeaways.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With k=4 on a regular grid, weight_power=2 gives the result closest to
bilinear interpolation, as recommended in the interp2poi docstring.
NB4 already used it; NB1 and NB3 are re-executed with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- NB1: map the difference to the observations for each method, at the
  latest lead time with observations
- NB2: six stations (Plateau, valleys, mountains) and temp_fixgrad
- NB4: larger DHM25 domain, read_dhm25 with its fixed LV03 CRS, both
  models side by side in the benchmark table
- all: data model set through GRID_PROCESS__ environment variables
  instead of renaming the station metadata, benchmark tables without
  the kwargs column, typed helper functions, clean mypy and pylint
- outputs: download RhiresY and DHM25 with earthkit instead of pooch
  (its tqdm.auto import warned without ipywidgets), hide the ECMWF
  connection limit notice, and disable progress bars in the ICON
  notebooks, which showed signed download URLs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stations now cover two valleys, two peaks, a mountain slope and the
Plateau. Re-executed with complete observations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cover the plain, valleys, slopes and peaks with cases where the altitude
correction works well and cases where it does not, and give the
benchmark more points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Interpolate to all SwissMetNet stations and plot the 14 selected ones, so
that the benchmark covers about 160 stations instead of 14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matschaer
matschaer marked this pull request as ready for review October 29, 2026 16:18
@matschaer matschaer self-assigned this Oct 29, 2026
earthkit always shows its download progress bars (disable=False is passed
to tqdm, so TQDM_DISABLE has no effect), and for ICON they include the
signed download URLs. Earlier outputs were clean only because the files came
from the earthkit cache. Redirect stderr during the downloads in notebooks
01, 02 and 04 instead.

Notebook 04: name IFS-HRES and ICON-CH2 in the title and print the source
grid shape of each model above the benchmark table. Re-executed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant