W2 backtest: history layer, leave-one-winter-out

Rerun 2026-09-29 · scripts/backtest_history.py (tests: tests/test_backtest_history.py) · outputs in out/backtest/ (a_curve.csv, scores_by_family.csv, reliability.csv, summary.json) · data: the ACIS rebuild with the W1 decisions

Decided 2026-09-29 (Ryan): the Season Book's fair value is TREND-ADJUSTED, a = 10 kept. The strong El Niño winters and the 1991–2020 winters are both moved to the 2026–27 climate by each station's own trend before the fair value (k + a·p₀)/(n + a) is taken. Evidence: over every strong El Niño winter 1950–2024 the (additive) trend-adjusted price scored +6.2% against its base rate where the raw one scored +3.6%; out of sample, winters 2017–18 to 2025–26, the trend-adjusted history price scored +0.05 on the book's lines where the raw one scored −0.06 (docs/MODEL_LAYER_V2.md §6).

Refined the same day (Integrator, pre-set rule; "Trend form for snow and precipitation" below): the temperature families (Dec–Feb temperature, heating degree days, coldest night, ice days) keep the additive straight line; snow uses a multiplicative trend (each winter scaled by the log-link trend of the station's mean snowfall, so a snowless winter stays snowless); Nov–Mar precipitation is untrended. With these forms, against the same raw base rate: +10.2% vs +3.5% for strong El Niño winters (gain +6.7 pt [+3.3, +10.2]) and +7.0% vs +4.3% for all classes (+2.6 pt [+0.5, +4.6]). a = 10 stays inside the flat optimum (a = 15 is now the pooled argmin, 0.02% lower Brier; the paired difference includes 0).

Decided 2026-09-27: Ryan confirmed a = 10 (raw construction), and the W1 data decisions were adopted the same day (docs/W1_ACIS_REVIEW.md).

Method (every choice)

• Lines. All 1,381 lines in out/acis/book_v1_*_history.csv (the strike ladders run_vps.sh builds from the W1 data): six families, 53 station series (52 cities; New York is two threads).
• Targets. Every winter from 1950–2024 with a known ONI class: 74 winters and 101,673 forecasts. 1950 ("unclassified") and 2025 ("pending") are never targets.
• LOWO forecast for target w. Everything is computed without w:
  • the station's trend, fitted over its usable seasons 1950–2025 other than w (0 with 10 or fewer seasons, as the calculator);
  • every history season moved to w's climate in the family's trend form (winter_odds_history.TREND_FORM): additive, value + slope·(w − year) (OLS slope), for the temperature families; multiplicative for snow, value · exp(b·(w − year)) with b the quasi-Poisson (log-link) trend of the mean; none for precipitation; always within the family's physical bounds (BOUNDS: snow, precipitation, heating degree days and ice days never below 0; ice days never above the Nov–Mar window of w);
  • p₀ = the share of the shifted 1991–2020 winters (season starts 1990–2019) that clear the line;
  • k/n = shifted winters in w's class group within the 1950–2025 pool that clear it. The groups are strong El Niño (SE+VSE), other El Niño (WE+ME), neutral (N) and La Niña (WL+ML+SL);
  • fair = (k + a·p₀)/(n + a), with p₀ used when n = 0.
• Outcome. The held-out winter's raw value against the strike.
• Mirrored from the calculator (winter_odds_history.price_station, which prices 2026–27 the same way with the trend fitted on every season 1950–2025 and the target 2026):
  • Rows with incomplete=1 are dropped. No suspect list or zero mask is applied, as in run_vps.sh.
  • Self-check (books rebuilt by the new calculator on 2026-09-29): recomputing every line without holdout at a = 10 reproduces the printed base %, k/n, trend-adjusted fair %, trend-adjusted base % and raw fair % for every line of five families; 12 of 420 snow lines differ (Seattle), because run_vps.sh builds the books with the adopted W1 drop list and the backtest, as before W1, does not.
• Scores. Brier score, reported as BSS vs the LOWO base rate p₀ (here: in w's climate) and vs the LOWO full-record (1950–2025) climatology. A comparison of the two constructions on one reference (the raw base rate) is below.
• Uncertainty. 95% intervals come from 2,000 bootstrap resamples of winters, never of lines.
• Choosing a. a is chosen by pooled Brier over all classes. Nested LOWO, which re-chooses a without the held-out winter, gives the honest skill.

The a-curve, trend-adjusted (pooled over all families; a_curve.csv has each family and class)

a	BSS vs base, all classes	Brier above min	BSS vs base, strong El Niño (9 winters)	argmin share (bootstrap)
0	+0.6%	1.74%	+1.5%	0%
2	+1.5%	0.89%	+4.6%	0%
4	+1.9%	0.45%	+5.8%	0%
6	+2.1%	0.21%	+6.1%	4%
8	+2.3%	0.08%	+6.1%	10%
10 (the book)	+2.3%	0.02%	+6.0%	30%
15	+2.3%	0.00%	+5.4%	38%
20	+2.3%	0.08%	+4.8%	16%
30	+2.0%	0.30%	+3.9%	2%
50	+1.7%	0.70%	+2.7%	0%
∞ (base rate)	0	2.39%	0	0%

a = 10 is still inside the flat optimum. The curve is flat from 4 to 30 (every a within 0.5% of the minimum Brier). With the refined forms a = 15 is the pooled argmin by 0.02% of the Brier score; a = 10 minus a = 15 is −0.0 pt, 95% interval [−0.2, +0.2]. 10 or 15 wins 68% of the bootstrap resamples, nested LOWO gives +2.2% (a = 15 in 65 folds, 10 in 9), and the strong-El-Niño curve alone peaks at 6–8 (+6.1%, 0.1 point above a = 10). Nothing here argues for moving a; a = 10 is kept, as decided.

At a = 10: BSS vs base is +2.3% [+1.0, +3.7] for all classes and +6.0% [+2.1, +9.4] for strong El Niño winters. At a = 6: +2.1% [+0.5, +3.7] and +6.1% [+1.0, +10.7].

In-sample optimism is large. Scored without holding out the target, the same book shows +7.3% at a = 10 (+14.8% for strong El Niño) and would pick a = 0.

Trend-adjusted against raw, on one reference

Each construction's own BSS is measured against its own base rate (the trend-adjusted p₀ for the new price, the raw p₀ for the old one), so the two headline numbers are not directly comparable. On the same reference, the raw LOWO 1991–2020 base rate (paired over the same resampled winters, a = 10):

	Trend-adjusted (refined forms)	Raw (before 2026-09-29)	Difference
All classes, 74 winters	+7.0% [+4.1, +9.7]	+4.3% [+2.3, +6.3]	+2.6 pt [+0.5, +4.6]
Strong El Niño, 9 winters	+10.2% [+7.0, +13.0]	+3.5% [+1.1, +6.4]	+6.7 pt [+3.3, +10.2]
vs the full-record climatology, all / strong	+2.3% / +11.5%	−0.5% / +4.9%

The trend alone (the adjusted base rate, with no ENSO information) scores +4.8% against the raw base rate; the ENSO composite adds the rest. (With the additive form everywhere, the first draft of this decision, the gains were +2.1 and +5.7 points.)

Trend form for snow and precipitation (2026-09-29; scripts/trend_form_test.py, out/model_tests/trend_form_test.json)

An additive straight line with a zero floor is the wrong shape for bounded, skewed amounts: at Nashville it moved most recent winters to zero and the ≥ 1" line from 96% to 38%. Four forms were tested for the snow and precipitation families only, each built exactly as the price (the form applied to both k and p₀, everything refitted without the scored winter), scored against the raw LOWO base rate:

• a additive: value + slope·(target − year), OLS, floored at 0;
• b multiplicative: value · exp(b·(target − year)), where b is the trend of the log of the mean, fitted by quasi-Poisson regression (log link). Chosen over a log(1 + x) fit because it models the mean directly, allows zero winters without an offset whose size depends on the unit (inches), and scales every winter by the same factor, so zeros stay zero and the distribution keeps its shape;
• c significant: the additive shift only where the station's OLS trend is significant (Newey-West HAC standard error, Bartlett kernel, lag ⌊4(n/100)^(2/9)⌋ = 3; p < 0.05), else unshifted;
• d none: unshifted.

Pre-set rule: per family, the best strong-El-Niño BSS in the long backtest; within 0.5 point, the simpler, in the order d, b, c, a.

Family	Form	Backtest, all classes [95%]	Backtest, strong El Niño [95%]	Out of sample 2017–2025 [90%]
Snow (420 lines)	a additive	+1.9% [−0.9, +4.6]	+5.1% [+0.9, +8.7]	−1.5% [−5.5, +2.2]
	b multiplicative	+3.4% [+1.3, +5.7]	+7.2% [+4.4, +9.7]	+0.4% [−2.7, +3.3]
	c significant	+2.5% [+0.3, +4.7]	+5.1% [+1.4, +8.9]	−1.4% [−4.2, +1.3]
	d none	+2.7% [+1.1, +4.2]	+5.1% [+2.9, +7.4]	−1.0% [−4.6, +2.4]
Precipitation (298 lines)	a additive	+3.4% [+1.3, +5.3]	+7.5% [+0.7, +13.1]	+1.8% [−2.3, +5.4]
	b multiplicative	+3.4% [+1.3, +5.3]	+7.5% [+0.9, +13.0]	+1.9% [−2.1, +5.6]
	c significant	+4.0% [+2.0, +6.0]	+8.8% [+1.7, +14.6]	+3.0% [−1.5, +7.2]
	d none	+3.8% [+1.8, +5.8]	+8.7% [+1.7, +14.6]	+4.1% [−0.9, +8.9]

Chosen: snow multiplicative (best by 2.1 points; the only form ahead of the base rate out of sample), precipitation none (c is best by 0.04 point, within 0.5, and d is simpler; d is also the best out of sample). The multiplicative form is also the better snow form on every reference; the out-of-sample check is nine winters and cannot separate the forms.

Sanity check, the chosen snow form against the low-snow stations' last 10 winters (2016–17 … 2025–26; Atlanta 9):

Line	Last 10 winters	1991–2020	Multiplicative (chosen)	Additive	Untrended
Nashville ≥ 1"	90%	93%	82%	38%	96%
Nashville ≥ 2" / ≥ 3" / ≥ 6"	60 / 50 / 30%	68 / 50 / 25%	63 / 41 / 18%	36 / 34 / 18%	73 / 63 / 39%
Atlanta ≥ 0.1" / ≥ 1" / ≥ 2"	44 / 22 / 22%	65 / 50 / 46%	73 / 50 / 48%	73 / 50 / 48%	75 / 56 / 48%
Raleigh ≥ 0.1" / ≥ 1" / ≥ 3" / ≥ 8"	80 / 70 / 40 / 20%	87 / 70 / 43 / 27%	82 / 74 / 49 / 11%	75 / 51 / 47 / 16%	82 / 74 / 49 / 30%
Memphis ≥ 0.1" / ≥ 2" / ≥ 3" / ≥ 6"	100 / 70 / 50 / 20%	97 / 59 / 41 / 10%	98 / 55 / 34 / 12%	72 / 44 / 34 / 14%	98 / 62 / 53 / 21%
Dallas ≥ 0.1" / ≥ 1" / ≥ 3"	80 / 60 / 10%	67 / 33 / 17%	65 / 39 / 7%	40 / 37 / 7%	67 / 39 / 25%

The multiplicative prices sit with the recent record where the additive ones collapsed (Nashville, Memphis and Dallas trace lines); Atlanta's recent winters were drier than any form's price (its trend is near zero, so every form gives the same price), and every form prices it on 9 strong winters and the 1991–2020 base, not on the last decade.

By family, trend-adjusted in each family's form, a = 10 (BSS in % vs that construction's own base; scores_by_family.csv also has each class group and a = 6)

Family	Lines / stations	All classes vs base [95% CI]	vs full record	Strong El Niño vs base [95% CI]	vs full record	Best a (all / strong)
All (pooled)	1381 / 53	+2.3 [+1.0, +3.7]	−0.1	+6.0 [+2.1, +9.4]	+5.6	15 / 6
Snow total (multiplicative)	420 / 45	+1.5 [+0.2, +2.8]	−0.9	+5.2 [+3.3, +7.4]	+3.5	20 / 8
HDD Nov–Mar	134 / 50	+1.9 [−0.5, +4.3]	+0.7	+14.1 [+9.0, +18.4]	+12.5	10 / 3
DJF mean temp	131 / 52	+2.8 [−0.4, +5.8]	+1.5	+8.2 [−2.3, +15.5]	+11.8	10 / 6
Min temp Nov–Mar (≤)	230 / 53	+2.5 [−0.1, +5.0]	−1.9	−3.0 [−6.0, −0.3]	−1.5	15 / ∞
Ice days	168 / 41	+1.6 [−0.3, +3.5]	−1.0	+6.8 [+1.7, +10.7]	+6.0	15 / 6
Precip Nov–Mar (untrended)	298 / 53	+3.8 [+1.8, +5.8]	+1.6	+8.7 [+1.7, +14.6]	+8.0	10 / 6

Strong El Niño only (9 winters per station; tiny, so read the intervals). - With the trend taken out, the temperature-driven families gain the most (degree days +14.1%, winter temperature +8.2%, ice days +6.8%; in the raw construction they were +3.6, +0.9 and −0.3). Precipitation and snow keep their signal. - The coldest night is still no better than the base rate, and significantly worse; its own curve prefers a = ∞. Do not headline ENSO shifts on cold extremes. - Reliability (reliability.csv): all classes, forecasts near 0.05 verify at 0.07 and forecasts near 0.96 at 0.94 (both ends a little over-confident, as before); strong El Niño, 0.96 → 0.93.

The base rate is still a weak reference over 1950–2024

Against the LOWO full-record climatology the pooled trend-adjusted price adds nothing for all classes (−0.1%); only strong El Niño is clearly ahead (+5.6%). On the 1990–2024 targets, which resemble the live forecast, all-class skill is +1.8% (best a = 15) and strong El Niño +9.0% at a = 10 (best a = 3, on only 4 winters). So a = 10 is right for the strong-El-Niño lines sold this winter; any future La Niña or neutral book should shrink harder, or quote the base rate.

Sensitivities (not in the numbers above)

Variant	Best a: all / strong	BSS at a = 10, all / strong
Raw construction (the calculator before 2026-09-29; vs the raw base)	10 / 10	+4.3% / +3.5%
Targets 1990–2024 only	15 / 3	+1.8% / +9.0%
+ 1940–49 classed by MEI-ERA5 DJF ≥ 1.5σ (no 1940s winter qualifies as strong; 1940–41 is 1.43σ)	15 / 8	+2.5% / +4.7%
+ 1940–49, quantile-matched 1.25σ (1940–41 becomes strong)	15 / 8	+2.6% / +4.6%

At 1.5σ, MEI-ERA5 flags only 6 of the 9 ONI-strong winters. It misses 1957, 1965 and 1987.

What the change does to the book (draft catalogue, rebuilt 2026-09-29 with the refined forms)

The 1,433 draft contracts keep their ids and strikes. Against the raw price: median change 0 (369 unchanged, 867 move 5 points or more, 624 move 10 or more). Winter temperature up (median +14); degree days, coldest night and ice days down (median −16, −11, −12); snow down a little (median −3; largest Detroit ≥ 50" 15% → 29%, Albuquerque ≥ 12" 43% → 16%); precipitation unchanged. Against the first (all-additive) draft, only snow and precipitation moved: 60 snow lines by 5 points or more (Nashville ≥ 1" 38% → 82%, Denver ≥ 70" 32% → 9%) and 81 precipitation lines (back to the raw price: Dallas ≥ 12" 88% → 63%, Billings ≥ 2" 75% → 93%). The raw price stays in the catalogue as fair_value_raw_pct.

Caveats

• n is small: 9 strong winters, and 4 after 1990.
• Lines are highly correlated, both within a ladder and across stations in the same winter. That is why every CI resamples winters. The effective sample is about 74 winters, not 101,673 forecasts.
• The trends are fitted over 1950–2025 in one form per family. The forms for snow and precipitation were chosen on these same 74 winters (the rule was set before the test), and the out-of-sample check has nine winters.
• The strike catalog was filtered at 3–97% base rate on the raw 1991–2020 winters, including the target, which is a mild selection effect.