Upper Bound Evaluation
Este conteúdo não está disponível em sua língua ainda.
Purpose
Section titled “Purpose”This spec defines Cobre’s upper-bound mechanisms: the statistical Monte-Carlo and exact deterministic forward-pass estimators that supply the per-iteration training bound (sections 1–2), the gap computation that compares that bound against the lower bound from cuts (section 9), the post-training sampled and census simulation estimator (section 11), and the reserved vertex-based inner approximation (SIDP) target design (sections 3–8, 10). It complements the outer approximation (cuts) described in SDDP Algorithm by providing the convergence-certificate half of the bound pair.
For notation conventions (index sets, parameters, decision variables, dual variables), see Notation Conventions.
1 Overview of Upper-Bound Mechanisms
Section titled “1 Overview of Upper-Bound Mechanisms”Standard SDDP produces only a lower bound on the optimal cost, through the outer (cut) approximation. A convergence certificate additionally requires an upper bound , closing the gap
(section 9 gives the full per-iteration gap computation; Stopping Rules, section 5, describes how the gap-based stopping rule consumes it).
Cobre computes this per-iteration upper bound via one of two forward-pass mechanisms, selected by the forward pass’s sampling mode:
- Statistical Monte-Carlo upper bound — implemented. Under a sampled forward pass, the sample mean of the scenario costs gathered that iteration, together with a 95% confidence-interval half-width, estimates the expected cost under the policy. This is a statistical estimate: it carries genuine sampling error that narrows only as more scenarios are drawn.
- Exact deterministic upper bound — implemented. Under an enumerated forward pass, the probability-weighted expectation over every enumerated leaf path is the exact expected cost under the policy, with no sampling error at all. See section 2.
A third mechanism — a vertex-based concave inner approximation of the cost-to-go function (SIDP) — is a reserved, not-yet-implemented target design, described in sections 3–8 (with its computational overhead in section 10). Once built, it would evaluate the upper bound independently of the forward pass’s sampling mode.
This chapter also describes a distinct, later-phase estimator: the post-training out-of-sample simulation procedure (section 11), which reruns the trained policy on scenarios independent of the training tree, in a sampled and a census variant. That estimator is a diagnostic on the finished policy — it is not part of the per-iteration training loop the mechanisms above feed, and it is not consumed by any stopping rule.
2 Exact Deterministic Upper Bound
Section titled “2 Exact Deterministic Upper Bound”Under an enumerated forward pass — one that visits every node of the policy graph deterministically rather than sampling it — the forward pass evaluates every leaf path of the scenario tree exactly once. Let be that leaf path’s probability and
its realized total discounted cost (see Discount Rate). The exact upper bound is the probability-weighted expectation over the full enumeration:
Because the enumeration is exhaustive rather than sampled, is the exact expectation of total cost under the current policy — not an estimate of it. Its standard deviation and 95% confidence-interval half-width are consequently identically zero: a deduplicated enumeration carries no sampling distribution to estimate a spread over.
The bound above is the exact upper bound under an expectation objective. Under a CVaR measure held uniform across every stage, the same exhaustive enumeration yields an exact bound of a different shape — a nested risk recursion — developed in section 2.1; see Risk Measures for the risk-measure background.
Contrast with the statistical mechanism (section 1): a sampled forward pass computes the same weighted-sum form — sample weight per scenario — but that sum is a Monte Carlo estimator of the expectation, carrying genuine sampling error. Only the enumerated forward pass’s weights (the true leaf-path probabilities) make the sum exact rather than an estimate.
This is the training-phase, per-iteration mechanism: it is evaluated once per training iteration and feeds the gap computation (section 9). It is distinct from the post-training out-of-sample simulation’s census variant (section 11.4), which computes the same probability-weighted-sum form over an independently drawn simulation population, evaluated once after training completes.
2.1 Nested Risk-Adjusted Exact Bound (uniform CVaR)
Section titled “2.1 Nested Risk-Adjusted Exact Bound (uniform CVaR)”When every stage applies the same CVaR measure , the probability-weighted path sum of section 2 is no longer the quantity the policy optimizes. SDDP minimizes a nested, time-consistent risk functional,
in which the measure is applied stage by stage, not once to whole-path totals. The exact upper bound must be evaluated in that same nested form. Over the enumerated scenario tree, let node carry immediate cost at stage , with children reached under conditional probabilities . Define, from the leaves up,
where applies the stage measure to the children’s values weighted by — the same risk aggregation the cut construction applies in the backward pass. The exact upper bound is the value at the root:
It is exact for the same reason the expectation form is — the recursion visits every node of the deduplicated enumeration exactly once, so it carries no sampling distribution, and its standard deviation and confidence-interval half-width are identically zero.
Why the nested form, not the path-total form. Applying once to the distribution of whole-path totals — an end-of-horizon form — is not a valid upper bound on the nested objective. The nested functional dominates the end-of-horizon one, , so the path-total quantity can fall below the risk-averse lower bound. Fed to the gap rule, it would then produce a persistent negative gap that halts training at a spurious lower/upper crossover before the policy has converged. The nested instead satisfies at every iteration, so the gap stays non-negative and closes only at true convergence.
Under expectation — or a CVaR with , which is expectation-equivalent — the measure is linear, the nesting telescopes, and collapses exactly to the probability-weighted path sum of section 2; the two forms coincide.
3 Vertex-Based Inner Approximation
Section titled “3 Vertex-Based Inner Approximation”The inner approximation would be constructed from vertices (visited state-value pairs):
where each vertex would store:
- : State vector visited during forward passes
- : Upper bound on expected cost-to-go from that state (computed recursively)
4 Lipschitz Interpolation
Section titled “4 Lipschitz Interpolation”For a new state not in , the upper bound would be computed via Lipschitz interpolation:
where is the Lipschitz constant for stage .
Interpretation: The upper bound at would be the minimum over all vertices of “vertex value plus distance penalty.” This forms a concave piecewise-linear function — the inner (concave) counterpart to the outer (convex) cut approximation.
5 Lipschitz Constant Computation
Section titled “5 Lipschitz Constant Computation”The Lipschitz constant would bound the maximum rate of change of the value function with respect to the state. For SDDP with penalty-based feasibility (relatively complete recourse):
Backward accumulation:
where:
- is the maximum penalty coefficient at stage (e.g., deficit penalty in $/MWh)
- is the discount factor for transition (see Discount Rate)
Example: With deficit penalty $/MWh over 5 stages, no discounting:
| Stage | Lipschitz |
|---|---|
| 5 | 1,000 |
| 4 | 2,000 |
| 3 | 3,000 |
| 2 | 4,000 |
| 1 | 5,000 |
6 Vertex Value Computation
Section titled “6 Vertex Value Computation”During the upper bound evaluation pass (a backward pass variant), vertex values would be computed as follows.
At terminal stage :
At stage :
For each vertex :
- For each scenario , the stage subproblem would be solved with incoming state and realization
- The optimal next-stage state would be obtained
- The inner approximation at the next stage would be evaluated:
- The vertex value would be set as the expected discounted cost-to-go:
7 Upper Bound Evaluation LP
Section titled “7 Upper Bound Evaluation LP”For policy evaluation with the inner approximation, the stage LP would replace the outer approximation (cut constraints on ) with the inner approximation (vertex constraints on ).
Standard LP (outer approximation — lower bound; this part is implemented today):
Inner approximation LP (upper bound; reserved):
8 Linearized Upper Bound LP
Section titled “8 Linearized Upper Bound LP”The absolute value in the vertex constraints would be linearized using standard splitting:
Additional variables (per vertex , per state component ):
| Variable | Domain | Description |
|---|---|---|
| Positive deviation from vertex in dimension | ||
| Negative deviation from vertex in dimension | ||
| free | Upper bound on future cost |
Constraints (for each vertex ):
9 Gap Computation
Section titled “9 Gap Computation”At each training iteration , the current lower and upper bounds are evaluated at stage 1’s fixed initial state .
Lower bound (from cuts, the outer approximation):
Upper bound (from whichever forward-pass mechanism is active, section 1): under a sampled forward pass, is the statistical estimator’s sample mean; under an enumerated forward pass, (section 2).
Gap:
Under the exact mechanism specifically, this is . The clamp absorbs floating-point noise once the gap has closed to (numerically) zero; in exact arithmetic always holds.
Relative gap:
normalized by the lower bound, floored at so the ratio stays bounded as — never by the upper bound.
Convergence: As , for convex problems with finitely many scenarios, provided the upper bound is exact. Under a sampled forward pass, carries sampling error, so a small reported gap reflects that noise as well as genuine convergence.
For stopping rules that use the gap, see Stopping Rules, section 5.
10 Computational Considerations
Section titled “10 Computational Considerations”| Aspect | Impact |
|---|---|
| Vertices per stage | Typically |
| LP size increase | additional variables |
| Evaluation frequency | Trade-off between gap accuracy and runtime |
| Memory | Vertices stored separately from cuts |
Recommendation: A reasonable operating point would enable upper bound evaluation every 5-10 iterations after an initial burn-in period (10+ iterations), trading convergence-monitoring frequency against overhead.
11 Simulation-Phase Upper Bound Estimator
Section titled “11 Simulation-Phase Upper Bound Estimator”Cobre can also estimate an upper bound on expected total cost by running the trained policy on scenarios drawn independently of the training tree — a separate, post-training procedure distinct from the per-iteration training-phase mechanisms in sections 1–2. It supports two variants: a sampled estimator (Monte Carlo; sections 11.2–11.3) over an independently drawn scenario sample, and a census estimator (section 11.4) over an exhaustively enumerated population of scenarios.
11.1 Independence from Training
Section titled “11.1 Independence from Training”The core methodological guarantee is that the noise used for the simulation forward pass is drawn independently of the noise used during training. Training forward passes sample from the opening tree (see Scenario Generation) to build the scenario tree that drives cut generation; any cost computed by re-running the policy on those same training scenarios would produce a biased estimator — the cuts were shaped to be tight at those states. The out-of-sample simulation avoids this by drawing a fresh set of scenarios from the same opening tree via a separate seed. Because cuts have no dependence on these independent draws, the resulting cost sample is an unbiased estimator of the true expected cost under the current policy.
11.2 Sampled Estimator (Monte Carlo)
Section titled “11.2 Sampled Estimator (Monte Carlo)”The sampled variant executes a complete forward pass for each of the independently drawn scenarios, recording the total discounted cost for scenario :
where is the immediate cost at stage of scenario , and is the cumulative discount factor from stage 1 to stage (see Discount Rate).
The sample mean is the Monte Carlo estimator of expected total cost:
This estimator is unbiased under independent draws: . The sample standard deviation is:
the Bessel-corrected estimator appropriate to a drawn sample. For the census variant’s population-level counterpart, see section 11.4.
11.3 Confidence Interval
Section titled “11.3 Confidence Interval”Under the normal approximation, the 95% confidence interval for has half-width:
The approximation is reliable once is large enough for the central-limit-theorem regime to apply. The reported interval is .
Trade-off: every doubling of narrows the confidence interval by a factor of , but costs proportionally more LP solves — the per-check cost scales with times the horizon length. Because the half-width shrinks as , a sufficiently large scenario count resolves the interval finely enough to distinguish a converged policy from one still improving.
This confidence interval applies to the sampled variant only. The census variant (section 11.4) reports the exact mean and population variance of an exhaustively enumerated population — there is no sampling error left to bound, so it carries no confidence interval.
11.4 Census Estimator
Section titled “11.4 Census Estimator”When the out-of-sample scenarios come from an exhaustive enumeration rather than a sample — a declared census — the per-scenario weight is that scenario’s leaf-path probability rather than a uniform sample weight, and the weights sum to one: .
The census weighted mean replaces the sample mean:
and the census weighted standard deviation is the true weighted population variance — no Bessel correction:
The population form omits the sampled variant’s correction because a census is exhaustive, not sampled: ranges over the entire population of scenarios rather than a draw from it, so there is no downward bias in the naive variance to correct for.
Because the population is fully enumerated rather than estimated from a draw, the census estimator carries no confidence interval — and are exact statistics of the enumerated population, not estimates of an unknown expectation.
11.5 Number of Simulation Scenarios
Section titled “11.5 Number of Simulation Scenarios”The sole knob governing the out-of-sample procedure — sampled or census — is the number of simulation scenarios . It controls the statistical resolution of the sampled estimator and the compute cost of the procedure simultaneously.
Raising narrows the sampled confidence interval as , while the compute cost — proportional to times the horizon length in LP solves — grows linearly in , so the scenario count trades statistical resolution directly against compute. Under the census variant, is instead fixed by the size of the declared enumeration rather than chosen for statistical resolution, since there is no confidence interval to narrow.
11.6 Relation to Training-Phase Bounds
Section titled “11.6 Relation to Training-Phase Bounds”This out-of-sample simulation estimator is independent of the training loop: it is not consumed by any stopping rule and does not gate training termination. Training termination is governed by the gap-based stopping rule (see Stopping Rules, section 5), which compares the training-phase upper bound — the statistical or exact forward-pass estimator, sections 1–2 — against the lower bound at every iteration, using the training tree rather than an independent out-of-sample population.
This section’s estimator instead reports the trained policy’s out-of-sample cost distribution once training has finished: the sampled mean/standard-deviation/confidence-interval (sections 11.2–11.3), or the census weighted mean/population variance (section 11.4). See Running Cobre → Convergence & Diagnostics for how this output is consumed operationally.
12 Cyclic Mode
Section titled “12 Cyclic Mode”For the reserved cyclic policy graphs design (see Horizon Modes), the inner approximation would operate on the same seasonal cut-pool structure: vertices organized by season , not by absolute stage ID. The Lipschitz constant would need to account for the cumulative discount around the cycle, which bounds the geometric series of future contributions.
The convergence guarantee would still hold: with , both the outer (cut) and inner (vertex) approximations would converge to the true value function at the fixed point.
13 References
Section titled “13 References”Costa, B.F.P., & Leclère, V. (2023). “Duality of upper bounds in stochastic dynamic programming.” Optimization Online. https://optimization-online.org/?p=23738
Philpott, A.B., de Matos, V.L., & Finardi, E.C. (2013). “On solving multistage stochastic programs with coherent risk measures.” Operations Research, 61(4), 957-970. https://doi.org/10.1287/opre.2013.1175
Cross-References
Section titled “Cross-References”- SDDP Algorithm — Core algorithm providing the outer approximation (lower bound) that this spec complements
- Notation Conventions — Standard symbols for state variables, value functions, and cost-to-go
- Discount Rate — Discount factor used in the exact bound’s discounted cost (section 2), the reserved vertex value computation (section 6), and the reserved Lipschitz accumulation (section 5)
- Policy Graphs — The enumerated-versus-sampled forward-pass distinction that selects between the statistical and exact upper-bound mechanisms (sections 1–2)
- Horizon Modes — The reserved cyclic policy-graph target design and the season-indexed pool structure section 12’s reserved inner approximation would mirror
- Cut Management — Outer approximation cuts that provide the lower bound counterpart
- Stopping Rules — The gap-based stopping rule, which compares this chapter’s training-phase upper bound (sections 1–2) against the lower bound (section 9)
- Risk Measures — Risk-averse formulations for which neither implemented upper-bound mechanism is valid; the reserved inner approximation (sections 3–8) is designed to remain valid under any risk measure
- Scenario Generation — Opening-tree definition from which the out-of-sample simulation (section 11) draws independent scenarios
- Running Cobre: Convergence & Diagnostics — the software guide for reading and assessing this estimator’s output.