How to build a defensible baseline model: reproducible, calibrated, thresholded by cost

What a baseline is for

A baseline is the simplest model that is honest about the data, and it sets the bar every more complicated model must clear.

Two baselines are worth having. The first is the constant model, which predicts the base rate for every case: it tells you what "no information" scores. The second is the simplest reasonable model, here a regularised logistic regression (see maximum likelihood and logistic regression). The second must beat the first on the log-loss, and the gap is reported, not asserted.

In a bank, the team that builds a score does not decide to use it. It writes a note whose outline is fixed: what decision and for whom, which data, which method, how well it ranks and how well it is calibrated, where the threshold sits and what it costs, and where it stops working. A separate validation team, often called the second line of defence, reruns the figures from the repository and looks for what is wrong. A model committee then approves, approves with conditions, or rejects. The baseline is often not what gets deployed, but it is the yardstick that any challenger is measured against.

Back to contents.

A repository that someone else can rerun

If another person cannot clone the project and get the same numbers with a single command, the numbers do not count yet.

Reproducibility is the entry ticket. The repository holds the code in src/, tests in tests/, a lockfile that pins every package, a continuous integration workflow that runs on each push, and one target, make reproduce, that goes from the raw CSV to a metrics.json file. The split into training, validation and test sets is frozen once and identified by a hash of the CSV and of the row identifiers, so that any drift is detected.

The rule that keeps a report honest is simple: every figure quoted in a document must exist in metrics.json. A number typed by hand cannot be checked, so it should be treated as wrong until proven otherwise.

The test an outsider runs

1. Clone. A member of another team clones the repository into the same development container, with no help from the authors.
2. Follow the README. They run the documented commands in order and nothing else.
3. Compare. They open the resulting metrics file and check it against the note, figure by figure.
4. Report. Everything that failed (a path, a missing package, a data file outside the repository) becomes an issue.

Back to contents.

Two models written by hand, checked against the library

Writing least squares and logistic regression yourself, then matching scikit-learn, is the proof that you know what the library computes.

The baseline package includes two notebooks. The first fits ordinary least squares twice, by the normal equations and by the singular value decomposition, as in least squares and linear regression. The second fits a logistic regression by gradient descent on the average log-loss. Each one is compared with the scikit-learn estimator on the same data, and the gap between coefficients or predictions is printed rather than described.

A notebook that imports the library model and calls it "from scratch" is not one. The course also asks that an AI assistant may explain these derivations but does not write this code: it is the code you will be asked to modify and explain line by line.

Back to contents.

Leakage and miscomputed metrics

Figures that look too good usually hide a leak, and figures that look too poor usually hide a metric computed on the wrong input.

The leakage map lists every column with the date at which its value is known, next to the date of the target. In the UCI data, the bill amount and repayment status of September 2005 are legitimate inputs for a default observed in October; anything recorded after the decision is not. In code, the usual culprits are a standardiser fitted to every row of the CSV before splitting, and a target encoding that has seen the test rows. A unit test catches the first one: the mean stored by the scaler inside the pipeline must equal the mean of the training fold alone.

The opposite symptom, an AUC that is oddly low, often comes from passing hard 0/1 predictions to the AUC instead of probabilities. A test that checks the scores take more than two distinct values catches it. For the Brier score there is a reference you can compute in your head, because the constant model that predicts the base rate $\pi$ has a known Brier score.

Worked example: the constant model on UCI Default

1. Base rate. About 22.1 % of clients default the next month, so $\pi = 0.221$ and $1 - \pi = 0.779$.
2. Product. The constant model's Brier score is $0.221 \times 0.779 \approx 0.172$.
3. Use. If your code reports another value for the constant model, the Brier function is wrong, or class-weighted probabilities were not corrected back.

Back to contents.

The two-page note: calibration, threshold and cost

A probability is only useful if it is calibrated, and a decision is only defensible if its threshold follows from the costs of each error.

The first page restates the framing: which decision is informed, what the target measures badly, which indicator matters to the business and which one is optimised, which guard-rail is watched, then the data (datasheet, frozen split, hash) and the leakage map. The second page presents the chosen model and why it beats the constant, then its calibration with a reliability curve and the Brier score, including the correction applied if class weights were used during training.

The threshold is not 0.5 by default. Once the cost of a false positive (a good client refused) and of a false negative (a default accepted) are written down, the threshold that minimises expected cost on calibrated probabilities follows, and the note shows the cost curve $C(t)$ on the validation set. The method is developed in decision threshold and reproducible ML.

A last paragraph asks what happens if the costs move by 20 % in either direction. A threshold whose expected cost collapses under a modest change of assumptions is not a robust decision, and the committee will ask.

Back to contents.

A pre-registered ablation study

An ablation removes one group of columns at a time to see what the model loses, and it is only convincing if the team wrote its predictions down first.

The order of operations is what makes an ablation serious. First the team predicts, without computing anything, the AUC and Brier it expects when each group is removed. It files those predictions as a GitHub issue, because the server timestamp of an issue proves the figures existed before the experiment, whereas commit dates can be rewritten. Only then does it compute, with the same split and seed as the baseline, and comment on each gap: a drop of 0.01 and a drop of 0.10 do not mean the same thing.

The teacher's own prior, stated openly in the brief, is that repayment history carries most of the signal and demographics very little. The point is not to agree: it is to check, and to write it down when the prior was wrong.

Back to contents.

Decision log, roles and the AI appendix

Traces written along the way show who decided what and why, and they are much harder to fake than a polished final report.

DECISIONS.md holds dated entries: the decision, its context, the alternative that was rejected with the figure that rejected it, the command that proves it, and who made it. Work is split into three roles (data, model, review) that rotate from one milestone to the next, and each member signs a function, a row of the ablation table and a log entry by name, so that questions can be addressed to the person who wrote the thing.

AI use follows the AI Assessment Scale of Perkins, Roe and Furze (2025), one level per deliverable: full use with human judgement for plumbing code that is tested and understood, explanation only for the hand-written models, drafting and proofreading for the note, and the appendix itself as an exercise in steering the tool. Declaring a use is never held against anyone.

• The appendix lists each use, its level and who reviewed it.
• It then gives at least two claims made by the assistant that the team showed to be false, each with a failing test, a computation or a figure as proof.
• Typical refuted claims: standardise before the split (a leakage test fails), keep the 0.5 threshold (the cost curve says otherwise).
• Finding nothing to refute in several weeks usually means nothing was checked.

Back to contents.

Key takeaways

1. A baseline is the yardstick: the constant model and a regularised logistic regression, with the gap between them reported.
2. One command must take a stranger from a fresh clone to the same metrics file, and every figure in the note must come from that file.
3. Implementing least squares and logistic regression by hand, then matching scikit-learn, proves you understand the library.
4. The leakage map, a scaler fitted on training data only, and AUC computed on probabilities prevent the two most common false figures.
5. The threshold follows from written costs and calibrated probabilities, with a sensitivity check on those costs.
6. Predictions filed before an experiment, a dated decision log and refuted AI claims are the evidence that the team actually thought.

Glossary

Baseline

The simple reference model any candidate must beat by a measured margin.

Data leakage

Information unavailable at decision time that reaches the model, inflating its scores.

Leakage map

A column-by-column table of when each value is known, compared with the target date.

Frozen split

A train, validation and test partition fixed once and identified by a hash.

Reliability curve

Observed frequency against predicted probability; a calibrated model sits on the diagonal.

Brier score

Mean squared error between predicted probabilities and 0/1 outcomes.

Decision threshold

The probability above which the decision changes, here derived from error costs.

Ablation study

Removing groups of inputs one at a time to measure their contribution.

Pre-registration

Writing predictions or rules down, timestamped, before seeing the results.

Continuous integration (CI)

Automatic tests and checks run on every push to the repository.

Random seed

The number that fixes pseudo-random draws, kept in a single configuration file.

Datasheet

A document describing a dataset's origin, licence, collection and gaps.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Decision threshold and reproducible MLNext: Kernels, SVM and decision trees

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.