Reservoir RL
Teaching PPO agents to run a reservoir for a year — trading flood risk against water supply and river health.
- Role
- Solo research project
- Timeline
- Jun 2026
- Python 3.12
- Gymnasium
- Stable-Baselines3 (PPO)
- NumPy
- Matplotlib
- pytest
TL;DR
Problem
A reservoir operator can't maximise everything. Holding water protects supply but risks floods; releasing it prevents floods but can starve the river and the people downstream.
What I built
A custom Gymnasium environment for a 365-day year, eight PPO agents trained under different objective weightings, and an honest comparison with a conventional rule-curve policy.
Result
The balanced agent cut the flood penalty from 82.4 to 1.7 and the supply deficit from 194.7 to 183.4. The rule curve still wins on ecology.
- Flood
- 0.34
- Supply
- 0.33
- Ecology
- 0.33
Drag the cursor or tap the triangle. Arrow keys move it; Shift moves it further.
balanced, seed 45. Weights: Flood 0.34, Supply 0.33, Ecology 0.33.
Mean ± sd · lower is better
balanced vs Rule curve
Flood penalty
balanced0.0 ± 0.0
Rule curve82.4 ± 29.1
Supply deficit
balanced177.5 ± 15.1
Rule curve194.7 ± 10.0
Ecological deficit
balanced16.4 ± 1.9
Rule curve0.0 ± 0.0
Each point is one training run (one seed). Two runs with the same balanced weights landed at different points — see Limitations.
View dataHide data
| Policy | Weights (flood, supply, eco) | Flood | Supply | Ecology |
|---|---|---|---|---|
| flood_focused (seed 42) | 0.8, 0.1, 0.1 | 0 ± 0 | 269.2 ± 25.1 | 22.9 ± 2.1 |
| supply_focused (seed 43) | 0.1, 0.8, 0.1 | 28.6 ± 23.8 | 137.9 ± 11.8 | 17.9 ± 2 |
| ecology_focused (seed 44) | 0.1, 0.1, 0.8 | 0.1 ± 0.7 | 157.9 ± 15.8 | 9.2 ± 1.5 |
| balanced (seed 45) | 0.34, 0.33, 0.33 | 0 ± 0 | 177.5 ± 15.1 | 16.4 ± 1.9 |
| flood_supply (seed 46) | 0.45, 0.45, 0.1 | 0 ± 0 | 135.7 ± 12.4 | 11.7 ± 1.6 |
| supply_ecology (seed 47) | 0.1, 0.45, 0.45 | 4.2 ± 10.9 | 141.2 ± 18.9 | 15.4 ± 1.9 |
| flood_ecology (seed 48) | 0.45, 0.1, 0.45 | 0 ± 0 | 167 ± 20.7 | 13 ± 1.7 |
| mild_balanced (seed 49) | 0.5, 0.3, 0.2 | 0.5 ± 3 | 142.2 ± 15.1 | 10.3 ± 1.6 |
| Rule curve | — | 82.4 ± 29.1 | 194.7 ± 10 | 0 ± 0 |
| Balanced PPO, seed 42 (nominal) | 0.34, 0.33, 0.33 | 1.7 ± 7 | 183.4 ± 16.2 | 22.2 ± 2.2 |
| Balanced PPO, seed 42 (drier climate) | 0.34, 0.33, 0.33 | 0 ± 0 | 349.4 ± 18.9 | 40.5 ± 2.9 |
The environment
- One reservoir: capacity 300 units, a maximum release of 8 per day, and demand peaking at 7 per day.
- Inflows are synthetic, and the agent sees a noisy forecast of tomorrow's inflow.
- Each year is scored on three penalties:
- flooding (overflow);
- supply deficit;
- ecological deficit, when the river drops below a minimum flow of 1.5.
What went wrong first
- The first sweep was degenerate: every weighting learned to drain the reservoir.
- It took four changes to the environment before any real trade-off appeared:
- capacity;
- the release limit;
- a discount factor of 0.999;
- random starting storage.
- The modelling work lives in the environment, not in the algorithm.
The frontier
- The sweep: eight weightings, each trained for 150k steps and evaluated on 100 shared episodes.
- Against the rule curve:
- every agent beats it on flooding;
- seven of eight beat it on supply;
- none reaches zero ecological deficit, because the rule enforces minimum flow as a hard floor. That keeps the rule curve on the Pareto frontier for ecology.
Results
100 episodes, lower is better.
| Objective | Balanced PPO | Rule curve | Random |
|---|---|---|---|
| Flood penalty | 1.7 ± 7.0 | 82.4 ± 29.1 | 582.9 |
| Supply deficit | 183.4 ± 16.2 | 194.7 ± 10.0 | 467.5 |
| Ecological deficit | 22.2 ± 2.2 | 0.0 | 52.0 |
Stress test: a drier climate
- Setup: inflow 20% lower and 30% noisier, with no retraining.
- What happened: the balanced agent's supply deficit rose from 183.4 to 349.4, and its ecological deficit from 22.2 to 40.5.
- The lesson: an agent doesn't adapt to a world it wasn't trained on — the same lesson as my drift work.
Limitations and what's next
- Scope:
- a single reservoir;
- weighted-sum training rather than true multi-objective RL;
- ecology as a soft penalty;
- synthetic inflows.
- One seed per weighting. Two runs with identical balanced weights landed at different points (supply deficit 183.4 vs 177.5), so more seeds are the first fix.
- The flood term. Because the release limit equals the flood threshold, the flood term reduces to overflow.
- Why it matters to me. This is the kind of problem I want to study next: decisions under competing objectives and uncertainty.
How this was built
I designed the environment, the experiments and the evaluation, ran them, and wrote up the findings; Claude Code helped write the code.