Model selection, done honestly: corrected intervals, a pre-registered rule and a model card with a fairness audit

From a baseline to a champion

A champion is not the model with the best number in a table, but the one that beats the baseline by a margin you can prove.

The comparison puts four models on the same footing: the L1-regularised logistic regression from the baseline milestone as the reference, a random forest, gradient boosting, and a kernel model (either an SVC with an RBF kernel or a logistic regression on random Fourier features, with the choice justified in the decision log). The methods behind them are covered in kernels, SVM and decision trees and random forests and gradient boosting.

The question is asked honestly: does any challenger beat the baseline once uncertainty is counted? If none does, the simplest model stays champion. That is not a failure of the project; it is a result, and often the most useful one for a business that has to explain its decisions.

Back to contents.

One command, fifteen measurements, a corrected variance

The whole comparison must rerun from one command, and its uncertainty must account for the fact that cross-validation folds share data.

A single target, make compare, reads the frozen split and, for each of three seeds, cuts five folds in the training and validation part: fifteen AUC and Brier measurements per model. Hyperparameters are tuned inside each fold by nested cross-validation, never on the fold being scored. The test set stays outside the comparison entirely and is used once, at the end, for the champion only.

The fifteen measurements are not independent, because the folds overlap in their training rows. The naive variance of their mean is therefore too small. Nadeau and Bengio (2003) proposed a corrected estimate that inflates it according to the ratio of test to training sizes.

Worked example: how much wider the interval gets

1. Count. Three seeds times five folds give $J = 15$ measurements.
2. Ratio. Each validation fold is a quarter the size of the training part, so $n_{\text{test}}/n_{\text{train}} = 1/4$.
3. Naive factor. Without correction, the variance of the mean uses $1/15 \approx 0.067$.
4. Corrected factor. With the correction, $1/15 + 1/4 \approx 0.317$, that is 4.75 times larger, so the interval is about $\sqrt{4.75} \approx 2.2$ times wider.

Back to contents.

Reading a confidence interval honestly

Two models whose scores fall inside each other's uncertainty have not been shown to differ, however tempting the third decimal looks.

Every score is written with its 95 % interval, for instance an AUC of 0.78 [0.76, 0.80], and never as 0.7834 alone. The comparison table has one row per model, the mean AUC and Brier with their corrected intervals, and a corrected p-value for the paired test of each model against the logistic regression.

An illustrative case from the brief

1. The figures. Gradient boosting shows an AUC of 0.78 with interval [0.76, 0.80]; the logistic baseline shows 0.77.
2. The naive reading. "Boosting gains 0.01" is not a statement the data support, since 0.77 lies inside the boosting interval.
3. The right statistic. Because both models are scored on the same folds, compute the fold-by-fold difference and its corrected interval.
4. The verdict. If that interval contains zero, the gain is not established and the simpler model stays champion.

Back to contents.

A champion rule written before the results

If you choose the selection rule after seeing the table, you will always find a rule under which your favourite model wins.

The team files the rule as a GitHub issue before running the comparison, so the server timestamp proves its priority. A typical rule reads: the champion is the model with the best mean AUC, provided its gain over the logistic regression is established by the corrected test (p < 0.05) and its Brier score does not worsen by more than a small tolerance; otherwise the logistic regression remains champion. The seeds and the number of folds go into the same issue and into the configuration file.

Rules are sometimes badly written, for example with no provision for a tie. Changing one is acceptable if it is done in the open: a log entry with the old rule, the new rule, the reason and the date. What destroys trust is a change made in silence.

Back to contents.

The model card and a fairness audit that looks for proxies

A model card tells a reader who will use the model, how uncertain it is and where it must not be used, and the fairness audit checks it does not discriminate through the back door.

The model card follows Mitchell et al. (2019) in seven sections: intended and excluded uses, data, metrics with their intervals, threshold and costs, calibration with a reliability curve, the fairness audit, and limits. Limits are concrete: Taiwanese clients in 2005, a target that measures a one-month payment default rather than a "bad payer", training data from a period without a crisis. Explainability with SHAP, as in SHAP, fairness, PCA and clustering, completes it.

The audit in three moves

1. Remove the protected attribute. The sex column leaves the champion, since EU Directive 2004/113/EC prohibits sex discrimination in access to services, financial services included.
2. Test for proxies. Train a model that predicts sex from the remaining columns; an AUC clearly above 0.5 means the information is still there.
3. Measure by group. For each sex and age band, report acceptance rate, true and false positive rates and calibration, each with an interval.

The verdict has three values: fair, established gap, or inconclusive. A subgroup of a few hundred people is rarely "fair"; it is usually inconclusive, and the card should say so. A simple test of the whole document: hand the card alone to someone outside the team and ask them for the AUC interval, the threshold, its cost and the uses that are ruled out.

Back to contents.

Three algorithms by hand, each with an agreement test

Reimplementing boosting, kernel ridge and EM, then matching the library within a stated tolerance, shows you know what the library does.

Each hand-written algorithm comes with an agreement test that runs in continuous integration. Logistic gradient boosting fits, at each step, a regression tree to the pseudo-residuals and moves by a learning rate $\nu$. Kernel ridge has a closed form. EM for a one-dimensional Gaussian mixture alternates responsibilities and weighted updates.

The tolerances differ for a reason. Kernel ridge is exact linear algebra. Boosting differs slightly because scikit-learn takes a Newton step in each leaf, which the notebook should mention. EM only reaches a local optimum, so the comparison needs the same starting point.

Back to contents.

A tested surprise, the worst errors, and four pitfalls

Naming what you did not expect, and reading the cases where the champion fails worst, are the parts of the work an assistant cannot do for you.

Somewhere in the comparison, the fairness audit or the errors, a result will be unexpected: the forest beating boosting, a Brier score that worsens while AUC improves, a band where calibration drifts. The team names it in an issue and tests it on another seed or split, or with an explanation that makes a checkable prediction. The twenty worst test-set errors are then read case by case, one per member, as a diagnosis only: nothing learned there may go back into the model, or the test set stops being a test.

• Comparing on the test set. The comparison module never imports the test split; a search of the code checks it.
• Seeds that move afterwards. Seeds live in the pre-registration issue and the configuration file; the file's history shows any change.
• Dropping a protected column but keeping its proxy. The proxy test and the group audit are run regardless.
• A model card that repeats the README. The README says how to run; the card says for whom, for what, with what uncertainty, and for what not.

In a bank, the people who build a model do not validate it. Independent model validation reruns the code, often builds its own challenger, rereads the intervals, checks that the rule predates the results and hunts for proxies; a committee then approves with conditions, and internal audit checks the process. Supervisory guidance on model risk, such as the Federal Reserve's, calls this effective challenge. A champion package written this way is what that team expects to receive.

Back to contents.

Key takeaways

1. A champion has to beat the baseline by a margin established with intervals, otherwise the simpler model stays.
2. Tune inside nested cross-validation, keep the test set out of the comparison, and use it once for the champion.
3. Overlapping folds make the naive variance too small; the Nadeau and Bengio correction can widen intervals by a factor of two or more.
4. Compare models by the paired, fold-by-fold difference, not by two separate intervals.
5. Write the champion rule, seeds and folds in a timestamped issue before computing, and log any later change openly.
6. A fairness audit removes the protected attribute, tests for proxies and measures by group, with an honest inconclusive verdict when data are thin.

Glossary

Champion model

The model chosen for use after a documented comparison against the baseline.

Challenger

A candidate model put against the current champion or baseline.

Nested cross-validation

Tuning in an inner loop and scoring in an outer loop, so tuning never sees the scored fold.

Confidence interval

A range that reflects the uncertainty of an estimate at a stated level.

Corrected resampled t-test

Nadeau and Bengio's paired test whose variance accounts for overlapping folds.

Pre-registration

Filing a rule or prediction, timestamped, before seeing the results.

Model card

A short document stating a model's uses, metrics, limits and fairness checks.

Proxy variable

An input that carries a protected attribute indirectly.

Fairness

Absence of unjustified gaps between groups in outcomes or error rates.

Pseudo-residuals

The negative gradient of the loss that each new boosting tree is fitted to.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: SHAP, fairness, PCA and clusteringNext: Neural networks, drift, conformal prediction

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.