Probabilistic forecasting

DeepAR was the most accurate model I tested — and the most overconfident.

Role: Solo
Timeline: May 2026

Python 3.12
GluonTS (PyTorch)
statsmodels
pmdarima
properscoring
scikit-learn
MLflow
DVC
pytest

Source Report

TL;DR

Problem
A forecast's intervals matter as much as its point estimate. If "90%" intervals catch the truth only 74% of the time, every decision built on them is miscalibrated.
What I built
A reproduction of DeepAR on four years of hourly Beijing PM2.5 data, benchmarked against seasonal-naive, ARIMA and ETS with proper scoring rules, plus a calibration diagnosis and an attempted fix.
Result
DeepAR led on accuracy (CRPS 43.6 vs ARIMA's 47.4). But its 90% intervals covered only 74% of outcomes, and post-hoc recalibration didn't fix that under the test period's shift.

DeepAR seed

Seasonal naive
ARIMA
ETS
DeepAR, mean of 5 (computed)

DeepAR · mean ± sd over 5 seeds

CRPS: 43.62 ± 1.75; ARIMA 47.418
RMSE: 99.52 ± 2.26; ARIMA 103.454
90% coverage: 74% ± 4%; ARIMA 80.4%
ECE: 0.13 ± 0.03; ARIMA 0.072

ECE by forecast horizon · best seed (1337)

1 h ahead: 0.032
6 h ahead: 0.060
12 h ahead: 0.116
24 h ahead: 0.167

Isotonic recalibration, fitted on the validation period

Before and after, on the same run

Metric	Before	After
Test ECE	0.090	0.098
Validation ECE	0.014	0.016
90% coverage	80.4%	79.7%
CRPS	42.048	42.871
90% interval width	156.2	164.3

Test ECE

Before: 0.090
After: 0.098
Test-period fit (diagnostic): 0.020

A fit on the test period itself reaches 0.020. The method works; the shift is the problem.

Source

ECE here is the average gap between nominal and observed interval coverage across nine levels.

Observed against nominal coverage of the forecast intervals on the test set, at nine levels. Lines below the diagonal are overconfident: their intervals are too narrow.Source

View data

Observed interval coverage at each nominal level
Nominal	Seasonal naive	ARIMA	ETS	DeepAR, seed 42	DeepAR, seed 123	DeepAR, seed 456	DeepAR, seed 789	DeepAR, seed 1337	DeepAR, mean of 5 (computed)
0.1	0.097	0.187	0.173	0.065	0.069	0.066	0.075	0.088	0.073
0.2	0.169	0.32	0.34	0.118	0.131	0.141	0.136	0.167	0.139
0.3	0.244	0.405	0.472	0.177	0.201	0.21	0.207	0.238	0.207
0.4	0.326	0.48	0.58	0.25	0.261	0.291	0.279	0.311	0.278
0.5	0.4	0.554	0.656	0.313	0.34	0.376	0.352	0.392	0.355
0.6	0.472	0.624	0.72	0.383	0.409	0.47	0.429	0.478	0.434
0.7	0.571	0.677	0.781	0.474	0.504	0.558	0.512	0.571	0.524
0.8	0.679	0.741	0.839	0.571	0.592	0.663	0.615	0.677	0.624
0.9	0.786	0.804	0.879	0.679	0.722	0.768	0.739	0.797	0.741

Setup

Data: UCI Beijing air-quality data from the Aotizhongxin station — hourly PM2.5 from March 2013 to February 2017, 35,064 rows.
Splits:
- train to June 2016;
- validate July–December 2016;
- test January–February 2017, on 59 non-overlapping 24-hour windows. Each test window is forecast from the training data and the earlier test hours; the validation months aren't in its history (the same for every model).
DeepAR: a one-week context, a 24-hour horizon and Student-t outputs, trained over 5 seeds.
Scoring: every model produces 100 sample paths, and the same code scores them all.

Results

Test set; DeepAR is mean ± sd over 5 seeds.

Model	CRPS	RMSE	90% coverage	ECE
Seasonal naive	68.4	137.5	78.6%	0.084
ARIMA	47.4	103.5	80.4%	0.072
ETS	47.5	104.9	87.9%	0.109
DeepAR	43.6 ± 1.7	99.5 ± 2.3	74.1% ± 4.0	0.125 ± 0.027

ECE here is the average gap between nominal and observed interval coverage across nine levels.

Where it breaks

Miscalibration grows with the horizon. For the best seed, the coverage gap is 0.03 one hour ahead and 0.17 at 24 hours.

The fix that didn't work

The method: isotonic recalibration (Kuleshov et al.), fitted on the validation period.
Validation: DeepAR was already well calibrated there (gap 0.014), so there was nothing to learn.
Test: on the shifted winter test period, the gap went from 0.090 to 0.098.
The diagnostic: a fit on the test period itself reaches 0.020. The method works; the shift is the problem.

Lessons

Measure calibration explicitly.
Report results across seeds; CRPS ranged from 41.4 to 46.2 across them.
A validation set has to look like the future.

Limitations and what's next

Scope: a single station, PM2.5 history only (no weather covariates), and untuned hyperparameters.
Next: shift-aware and conditional recalibration, more stations, and newer models.

How this was built

I designed the experiments and the evaluation, ran them, and wrote up the findings; Claude Code helped write the code.