When an AI model tells you a setup has a "31% chance of working," that number is only useful if it's true — if, across all the times the model says 31%, roughly 31% actually work. A model can rank trades well and still be badly wrong about its own confidence. The fix is calibration.
Ranking vs. calibration
Two different things matter in a predictive model. Discrimination (often measured by AUC) is whether the model puts better setups above worse ones. Calibration is whether its stated probabilities match reality. A model can have strong AUC (~0.84 in TradeScaner's case) yet output probabilities that are systematically too high or too low.
Why calibration matters for trading
Position sizing and selection both depend on honest probabilities. If the model says 40% but the real rate is 20%, you'll over-size losers and over-trust weak setups. Calibrated probabilities let you set a sane edge floor ("only take setups above X%") and size positions sensibly.
How we calibrate
TradeScaner runs the gradient-boosted (XGBoost) classifier through isotonic regression — a monotonic, non-parametric calibration step that maps raw model scores onto probabilities that match observed outcomes, without assuming a fixed curve shape. The result: when the engine shows a win-probability, it reflects the historical base rate for setups like it.
What it does not do
Calibration makes probabilities honest; it does not make them certainties. A calibrated 30% setup still loses 70% of the time. The value is that you know that going in — and can build risk rules around it instead of around hope.