Per-lead scores (0 to 100) use the same formulas as the VeriSky app, on hourly
forecast/observation pairs over the last 30 days. Temperature and wind are each a
blend of two components,
0.7 · Accuracy + 0.3 · Extremes; open any
row to see the split.
- Accuracy is the share of hours the forecast landed inside a meaningful
band: within 2 °C for temperature, within
max(5 km/h, 20%) for wind. Beyond the band a 3 °C miss and a
9 °C miss cost the same.
- Extremes is SEDI, an
event score computed over the hours that were unusual for this location
(a temperature anomaly or a wind speed beyond roughly the local 90th percentile,
with floors of 3 °C and 20 km/h). It counts false alarms as well
as catches, so a model cannot win it by crying wolf. A calm window has no extremes
to judge: the component needs at least 8 event and 8 quiet hours, and where it has
fewer, its weight folds back into Accuracy.
- Rain: F1 of rain/no-rain calls at a 0.1 mm/h threshold with a
0.1 mm deadband (drizzle right at the threshold is a coin flip, not skill).
Rain is only scored when the window had a genuine rain signal
(≥ 3 rain-event hours totalling ≥ 1 mm), since otherwise every model
would ace a dry month, and only for models whose own sample holds at least 8
observed wet hours.
- Sharpness (in the expanded row, deliberately not in the score) is
how much the forecast series moves compared with the observed one. 1.0× moves like
the weather; a much lower number is a forecast hedging toward an average. It stays
out of the headline because it is a property, not an accuracy: scoring it would
just invite the opposite game.
Why not RMSE? Root-mean-square error, which this board used until
August 2026, is minimized by the conditional mean — so a model that hedges toward a
smooth average is playing the metric optimally, and a sharp forecast that is slightly
off in time or place is punished twice (a miss and a false alarm) where a
blurred one is punished once. That is the documented reason MSE-trained AI models can
top an error leaderboard while looking like a blurry mess on a map
(Ben Bouallègue et al. 2024,
Bonavita 2024). Scoring only the
hours where something happened is not the fix either — selecting hours on the
observations rewards whoever over-forecasts drama
(Lerch et al. 2017) — which is why
Extremes is a contingency table with false alarms in it. The old error score is still
computed: each expanded row says where the model would have ranked under it, and the
long-term trends page still charts it, because a
five-year comparison needs a yardstick that cannot move with the season.
The headline Skill number is a soft horizon-adjusted score. First,
each model gets its 1/d-weighted score over the lead days it actually serves.
Then that score is multiplied by 0.75 + 0.25·C, where
C is the share of the full 1/d-weighted 7-day horizon it covers.
Tomorrow counts 7× as much as next week, which is roughly how much people lean
on each, while accurate longer-range forecasts still add points. A perfect
D-1-only model can score about 85 instead of being treated as a failed 7-day
model; a perfect D-1…D-7 model can reach 100. Per-metric columns use the same
weighting.
Skill by lead time shows the numbers behind that weighting, in the app's two
views. The grid is a models × lead-days matrix: brightness is the score, so
reading down a column answers "who do I trust three days out" and reading across a row
shows the decay. Its bands are fixed (temperature and wind step at 50/60/70/80, rain at
30/40/50/60, since an F1 rain score genuinely lives lower), so a cell means the same
thing on every board. A blank cell is a lead day the model doesn't serve, and a model
that stops early gets a muted average — it is only being scored on the easy near days,
which is exactly why the column headers sort the grid too. The curves view draws
every model but colours at most four: a preset board carries thirteen, and thirteen
hues cannot be told apart, so the rest stay one context gray instead of pretending to
be readable.
Rain W–L and form dots are next-day (D-1) only, since "did it rain the next
day" is the question most people care about. A win is a correct daily rain/no-rain
call at 0.1 mm; a form dot fills when that day's next-day skill ≥ 70.
The ▲▼ movement column compares today's ranking with the same table computed
a week ago (the window minus its last 7 days).
Airport boards (the ✈ toggle) re-score every model at the city's main
airport, whose METAR weather station reports real instrument readings around the
clock at most airports. The server verifies temperature and wind against available
METAR observations and rain against archived radar where covered. Missing
observations fall back to model analysis; older scores can also retain analysis
from before a station or radar source was connected. METAR precipitation groups
are too sparse for the server's rain scoring. The in-browser charts
(Compare, labs) still compare public models to the analysis at the airport's
coordinates, so their curves can differ slightly from the standings. Chicago,
Mexico City and Sydney's city boards already sit on their airport's grid cell,
so for them the two views read the same measurement point. Airport standings
started accruing later than the city boards at some airports, so a shorter
"last N days" window there is expected until the archive fills. Unavailable city
or airport views are greyed out in the toggle.
Public-model forecasts come from Open-Meteo's
previous runs archive
(what each model actually said 1 to 7 days earlier). The reference series is
best_match with
past_days=30: Open-Meteo picks the single highest-resolution
model for the location (see "Data Sources" in their docs), and for past
hours serves that model's most recent, observation-fed runs. It is one
consistent series per location, not an agglomeration of forecasts, but it
is an analysis rather than a station record, so at short leads it can
slightly flatter models related to the one that provides it. The most
recent (incomplete) day is
always excluded. Preset cities add stored commercial scores and daily Compare
summaries: temperature and wind maxima, rain totals and wet-hour ranges.
Full commercial forecasts stay on the server. Public data is cached locally
for ~6 hours; commercial summaries are held in memory for 15 minutes.