Cross-version drift

Tip

  1. Need help? Please let us know in the SUEWS Community.

  2. Please report issues with the manual on GitHub Issues (or use Report Issue for This Page for page-specific feedback).

  3. Please cite SUEWS with proper information from our Zenodo page.

10. Cross-version drift#

Complementing the per-site benchmark report, this page describes how model results move across releases. SUEWS tracks three loosely coupled notions of “version”: the package version (the CalVer release that compiled the physics), the schema version (the YAML input surface), and the documented version history. The model-version registry binds these together and records, for every release, how reproducible its results are.

10.1. Reproducibility taxonomy#

Each release carries exactly one status:

benchmarked

Pip-installable and present in the benchmark results index with a recorded deterministic fingerprint. Drift is measured.

pip-installable

pip install supy==X works, but no benchmark entry exists yet. Reachable; drift not yet measured.

legacy-external-ref

Not pip-installable; reference outputs exist in the external SUEWS-Benchmark repository. Reproduction is deferred; the registry records where the reference lives.

documented-only

Appears in the version history but has no reference outputs and no install path. Catalogued for completeness; not reproducible.

10.2. What “drift” means#

Drift is the change in energy-balance error metrics (MAE, MBE) between two releases run on the same site, forcing, and observations. It is descriptive – it measures how far the model has moved – and is not an assertion that an older release is reproduced bit-for-bit.

Two framings are reported:

  • Consecutive – each release against the one before it.

  • Baseline – each release against a single pinned baseline (the earliest benchmarked release, recorded as drift_baseline_tag in the registry).

10.3. Querying the lineage#

The registry ships with supy and is queryable directly:

import supy as sp

for v in sp.list_model_versions():
    print(v.tag, v.reproducibility)

sp.schema_for("2026.4.3")        # -> "2026.4"
sp.model_version_info("2026.6.5")

The drift tables are produced by benchmark/assemble_drift.py and rendered on the benchmark site.