Metrics, calibration and a decision threshold under asymmetric costs, then a repository someone else can rerun

Accuracy, precision and recall when the positive class is rare

When only two applicants in a hundred default, a model that accepts everybody is right 98 times out of 100 and still useless, so we need rates that look at the defaults.

The deck keeps one illustrative portfolio all afternoon: 10,000 credit applications, 2 % of which default. A model that always answers "no default" scores 98 % accuracy and a recall of zero. A real model, at a given threshold, refuses 150 of the 200 defaults and lets 50 through, while wrongly refusing 800 of the 9,800 good clients. Its accuracy is (150 + 9,000) / 10,000 = 91.5 %, lower than the useless model, although it catches three defaults in four. Accuracy adds two errors that differ in count and in price.

Call a predicted default (a refusal) a positive. The four cells are TP, FP, FN and TN, with $P = TP + FN$ true defaults, $N = FP + TN$ good clients and base rate $\pi = P/n$. The rates differ only by their denominator, which says what you are talking about: the refused column, the default row, or the good-client row.

Two short rewritings explain the traps. Multiplying and dividing by $P$ and $N$ shows that accuracy is an average of recall and specificity weighted by the class sizes, so with $\pi = 0.02$ the defaults weigh 2 % of it. Dividing precision by $n$ gives Bayes' formula in credit vocabulary.

Back to contents.

ROC curve, AUC and the precision-recall view

Sliding the threshold traces a curve of caught defaults against wrongly refused clients, and the area under it is the chance that a random default is scored above a random good client.

Each threshold gives a confusion matrix, hence a point with the false positive rate on the x-axis and recall (the true positive rate) on the y-axis. Moving the threshold from 1 down to 0 draws the ROC curve. Its area has three equivalent readings: a geometric area, a probability, and a count of well-ordered pairs, which is the Mann-Whitney statistic divided by the number of pairs. The step from the first to the second is a change of variable followed by the law of total probability.

Counting the pairs on the ten applications

1. Pairs. Three defaults and seven good clients make 21 (default, good client) pairs.
2. Application 4 (0.15). It is scored above only the three good clients at 0.05, 0.08 and 0.10, so it wins 3 pairs.
3. Application 8 (0.55). It beats six good clients and loses only to application 9 at 0.70: 6 pairs.
4. Application 10 (0.90). It beats all seven: 7 pairs.
5. Result. U = 16, so the AUC is 16/21, about 0.762, and the Gini used in credit (2 AUC minus 1) is about 0.524.

Because only the order of the scores matters, any increasing transformation of the score leaves the AUC unchanged. That is its strength and its limit: it says nothing about the threshold, the costs, or whether the probabilities are right. The curve comes from radar signal detection (Peterson, Birdsall and Fox, 1954); Hanley and McNeil (1982) showed that its area is the normalised Mann-Whitney statistic, and Bradley (1997) brought it to the comparison of learning algorithms.

With a rare class, the ROC curve flatters. In the 10,000-application example, 800 false positives are 8 % of the good clients, which looks harmless, but 84 % of all refusals. The precision-recall curve shows that second fact; its random baseline is the default rate, not a diagonal. The $F_\beta$ score summarises one point of it as a weighted harmonic mean, and $\beta > 1$ gives more weight to recall. The deck warns that choosing $\beta$ is a cost decision in disguise.

Back to contents.

Calibration and the Brier score

A model is calibrated when, among the applications it scores at 10 %, about one in ten actually defaults; a good ranking does not guarantee that.

Two models can have the same AUC while one of them announces 50 % where the observed default rate is 10 %. The reliability curve shows it: group the predictions into bins and plot the mean predicted probability against the observed frequency; a calibrated model sits on the diagonal. The Brier score, born in weather forecasting (Brier, 1950), measures the values themselves as the mean squared gap between the probability and the 0/1 outcome.

Inside one bin of score $p$ with default frequency $\pi$, adding and subtracting $\pi$ and expanding the square kills the cross term, leaving the squared calibration error plus the Bernoulli variance $\pi(1-\pi)$. Averaging over bins and applying the law of total variance gives Murphy's decomposition (1973).

Worked check on the ten applications

1. Squared gaps. The ten values of (p minus y) squared sum to 1.734, so the Brier score is about 0.173.
2. The misranked default. Application 4, a default scored 0.15, costs 0.85 squared, 0.7225, that is 0.072 of the 0.173 on its own.
3. Constant model. Saying 0.3 to everyone is calibrated but has a single bin, so its Brier is the uncertainty, 0.3 times 0.7, which is 0.21.
4. Reading. The model beats the constant: its resolution earns more than its calibration errors cost.

The same reasoning gives the project's reference point: on the UCI Default dataset, with 22.1 % defaults, the constant model has a Brier of about 0.172, and a baseline should do better on validation. To recalibrate, fit Platt scaling (a logistic on the score) or isotonic regression (piecewise increasing) on the validation set, never on the test set. In banking this is not optional: the EBA guidelines on PD estimation (EBA/GL/2017/16, 2017) require probabilities of default aligned with the long-run default rate.

Back to contents.

Class weights inflate probabilities: correct them before deciding

Training with extra weight on defaults makes the model believe defaults are common, so its outputs must be scaled back before any threshold is applied.

Weighting the positives by $w$, or duplicating them, is training on a population where the prior of defaults is $w$ times larger while the class-conditional distributions are unchanged. By Bayes' formula the posterior odds are multiplied by $w$. Inverting gives back the true probability. For a logistic regression this only shifts the intercept, by the logarithm of $w$, although in practice the slopes move a little too, so the correction is followed by a recalibration.

A 12 % threshold applied to the displayed numbers would refuse all three applications, including the 2 % one the bank wants. The round trip checks the formula: 0.12 goes up to 0.870, and 0.870 divided by 49 minus 48 × 0.870 (that is 7.24) comes back to 0.120. The order of operations is therefore: correct, recalibrate on validation, and only then threshold. The deck calls a forgotten correction the most frequent bug in junior notebooks.

Back to contents.

Deriving the decision threshold from asymmetric costs

Refuse an application when refusing costs less on average than accepting it; the threshold that follows depends only on the two costs, not on the model.

For one application with calibrated probability of default $p$, accepting loses $c_{FN}$ with probability $p$ and refusing loses $c_{FP}$ with probability $1-p$. Refuse when the second expectation is smaller. Expanding, moving the $p$ term to one side and dividing by the positive sum of costs gives the rule. The default 0.5 of scikit-learn is the special case of equal costs, which credit never sees.

The deck's example, with written assumptions

1. Costs. A refused good client loses €600 of margin; an accepted default on a €10,000 loan loses €4,500, a loss given default of 45 % (the Basel II foundation value for senior unsecured corporate claims, used as a benchmark only).
2. Threshold. 600 / 5,100 is about 0.1176, so the bank refuses from roughly 12 % risk, because a default costs 7.5 times a wrong refusal.
3. Indifference. At p equal to the threshold, accepting costs 0.1176 × 4,500 and refusing 0.8824 × 600, both about €529.
4. Three applications. At 5 % risk, accepting costs €225 against €570 for refusing, so accept; at 30 %, €1,350 against €420, so refuse.

If the calibration is doubtful, the robust version minimises the total cost counted on the validation set, threshold by threshold. When the probabilities are calibrated, replacing each $y_i$ by $p_i$ shows that every term of the expected cost is minimised by the same rule, so the empirical minimum should fall near $t^{*}$, and a large gap is itself a diagnosis of poor calibration.

On the 10,000 applications, accepting everyone costs 200 × €4,500 = €900,000, which is what the bank pays without a model, and refusing everyone costs 9,800 × €600 = €5.88 million. At the cost threshold, with 800 false positives and 50 false negatives, the bill is €705,000: the model saves about €195,000 per 10,000 applications. At 0.5 the deck's curve reads about €870,000, close to accepting everyone, because with a rare class few applications ever exceed one half.

Back to contents.

A reproducible ML repository: seed, environment, tests and CI

A reported score depends on the data, the code, the random seed and the software versions, so each of the four is pinned and a neutral machine reruns everything on every push.

An AUC of 0.78 means nothing on its own. If any of the four inputs moves silently, the number moves and nobody knows why. The course project repository fixes each of them: the CSV is hashed with SHA-256 and the hash stored; the train, validation and test split is frozen by a seed and checked by its own hash; the code lives in importable modules under git, which notebooks import rather than copy; a single seed sits in config.yaml next to the costs and the threshold; the environment is a lockfile plus a fixed Python version in a devcontainer.

• Lockfile. A bare scikit-learn line means "whatever version is current on install day"; months later a changed default can shift the AUC with no code change. The lockfile pins exact versions and package hashes, is generated from a short input file (with pip-compile --generate-hashes or uv lock), never edited by hand, and committed.
• Five tests, under a minute. A schema contract on the incoming data (types, bounds, missing rate), a split with the expected hash, AUC equal to 1 on separable data, predictions unchanged when rows are permuted, and a reloaded pipeline that predicts like the trained one to $10^{-12}$. A test is a property that must always hold, not "the AUC looks good".
• One command. make reproduce checks the data hash, redoes the split, trains the sklearn Pipeline (fitted on the training fold only, so no leakage by construction) and writes the metrics file. Running it twice must give identical files; a difference means a loose seed.
• Continuous integration. On every push and pull request, a clean machine installs the lockfile, runs the tests and the reproduction, then compares AUC and Brier with a committed reference within 0.005. A red build is information: either the change was intended and the reference moves with an explanation, or a bug has just been found early.
• Branches and review. Nobody pushes to the main branch. One branch per task, one small pull request, a review checklist (green CI, a test for the change, nothing from the test set in the model, metrics and log updated when a number moves, readable without the author) and a merge by someone else.
• A decision log. Each entry gives the date, the choice, the context, the rejected alternative with the number that ruled it out, and the command that reproduces it. For example: threshold 0.12 rather than 0.5, €705k against €870k on validation. A tool can generate a model; it cannot write the history of the team's hesitations.

The same logic runs through the earlier sessions of the course, from least squares to the logistic regression and its regularisation: simple models, done cleanly and explained. AI assistants are welcome for plumbing such as a CI workflow or a Makefile, provided their use is declared; the threshold derivation and the modelling choices stay with the team.

Back to contents.

Key takeaways

1. Accuracy is a weighted average of recall and specificity, so with a rare class it mostly measures the majority.
2. Recall and the false positive rate describe the model; precision also depends on how rare the positive class is.
3. The AUC is the probability that a default is scored above a good client: it measures order, not probabilities, thresholds or costs.
4. Calibration is a separate quality, measured by the reliability curve and the Brier score, and it is what a banking supervisor reads first.
5. With calibrated probabilities, refuse when the probability exceeds the false positive cost divided by the sum of both costs; with €600 and €4,500 that is about 0.12, not 0.5.
6. Class weights multiply the odds; correct and recalibrate before applying any threshold.
7. Pin data, code, seed and environment, test properties, rerun everything with one command, and let CI compare the metrics on every push.

Glossary

Confusion matrix

The 2 × 2 table of true positives, false positives, false negatives and true negatives at a given threshold.

Precision, recall

Share of true defaults among refusals; share of true defaults that are caught.

False positive rate

Share of good clients wrongly refused; the x-axis of the ROC curve.

ROC curve, AUC

Recall against false positive rate over all thresholds; its area is the probability that a positive is scored above a negative.

Precision-recall curve

Precision against recall over all thresholds; more informative than ROC when the positive class is rare.

Calibration, reliability curve

Agreement between predicted probabilities and observed frequencies, plotted bin by bin against the diagonal.

Brier score

Mean squared gap between the predicted probability and the 0/1 outcome; reliability minus resolution plus uncertainty.

Decision threshold

The probability above which an application is refused; under calibrated probabilities, a ratio of costs.

Expected cost

Each possible loss times the probability that it occurs, summed; the quantity the threshold minimises.

Random seed

The number that fixes every random draw (split, shuffles, initialisation) so that a run can be repeated.

Lockfile

A generated file pinning exact package versions and hashes, so that the environment can be rebuilt identically.

Continuous integration (CI)

Automatic rerun of tests and reproduction on a clean machine at every push or pull request.

Reproducibility

Another person, with the same data, code, seed and environment, obtains the same numbers.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Gradient descent, ridge and lassoNext: A defensible baseline model

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.