Glossary/Recommendation confidence
What is recommendation confidence?
Recommendation confidence is the probability that taking a recommended action leaves the business better off than doing nothing. It is a relative measure, anchored to the current setting rather than to zero, and it is a probability rather than an amount: it answers how often the action wins, not how much it wins by. A confidence of 0.8 on a budget increase means that across the plausible futures the model considers, the higher budget produces more profit than the current one in four cases out of five. A confidence near 0.5 means the change is a coin flip and the honest recommendation is to leave it alone.
01/Formula
Formula
confidence = P( profit(recommended) ≥ profit(current) ) Evaluated on paired scenarios — the same simulated demand applied to both settings — so the comparison isolates the change: Δ_i = profit_i(recommended) − profit_i(current) confidence = share of scenarios where Δ_i ≥ 0
Example
A campaign flagged for a spend cut. probability of loss at current spend 0.65 confidence that cutting spend helps 0.017 The campaign loses money on most days, and cutting it beats keeping it in under 2% of scenarios. Both numbers are true. Only the second one is about the decision being made.
02/In detail
Why must the comparison be paired?
Because otherwise the noise swamps the signal. If the current setting and the proposed setting are each simulated with their own independent random demand, the difference between them carries the randomness of both, and a small genuine improvement disappears into simulation jitter. Pairing means applying the identical simulated demand to both settings — the same quiet Tuesday, the same campaign-day surge — so that whatever is common to both cancels and what remains is the effect of the change itself. This is the common random numbers technique, and without it a confidence figure is mostly measuring the simulator.
How is it different from probability of loss?
Different reference point, different question, and they routinely disagree. Probability of loss asks whether profit at the current setting is below zero. Confidence asks whether the proposed setting beats the current one. A lumpy but profitable campaign scores badly on the first and, correctly, protects itself on the second. Because they are anchored differently, one minus the probability of loss is not the confidence, and treating them as complements is the single most common error made with these two metrics.
What does confidence deliberately not tell you?
How much is at stake. Being a probability, it is blind to magnitude in the same way probability of loss is: an action that wins 55% of the time by a tiny amount and loses 45% of the time catastrophically scores as a modest positive. That is the acknowledged blind spot, and it is covered by pairing confidence with a magnitude-aware tail measure such as conditional value at risk. The reason confidence is still the right primary gate for most advertising decisions is that it is robust — a probability does not get dragged around by a single enormous simulated outcome, whereas an average does.
03/Why it matters
The trap, in one paragraph.
Most automated recommendations are generated by comparing a metric against a threshold, which asks whether a campaign looks bad rather than whether the proposed fix is an improvement. Those are different questions and they disagree most sharply on exactly the campaigns worth getting right: the profitable-but-lumpy ones, which look bad on any absolute metric and should not be touched. Confidence puts the comparison where the decision is.
Common mistake
Treating confidence as a quality score for the campaign. It scores the action, not the campaign. A very low confidence on a proposed pause is a strong statement that the campaign should keep running — it is not a statement that the campaign is performing badly.
04/In DataGlass
How Recommendation confidence is used in DataGlass.
DataGlass evaluates every ads recommendation as a paired comparison against the current setting on the same simulated demand, and surfaces the resulting confidence on the recommendation itself. Actions whose confidence does not clear the bar are not shown as recommendations, which is why a campaign that looks alarming on absolute metrics may correctly produce no suggested change.
05/Sources
- [1] Russo, Van Roy, Kazerouni, Osband & Wen — A Tutorial on Thompson Sampling (arXiv:1707.02038)
Background on posterior-based sequential decision making and on comparing candidate actions across sampled scenarios rather than against point estimates.
- [2] Expected shortfall
The magnitude-aware complement that covers the acknowledged blind spot of a probability-based decision gate.