Frame before you model, then least squares: decision, leakage, normal equations and conditioning

Start from the decision, not from the data

A model is only worth building if someone will act differently after reading its output, so the decision is written down before anything else.

The reflex when a CSV arrives is to open it and look for patterns. The deck asks for the opposite order. First write who decides, between which options, and what each kind of error costs. On the running project, a credit analyst approves, refuses or sends a file for manual review, a committee sets the threshold, and accepting a client who then defaults is more expensive than turning away a good one. That asymmetry will decide the metric and, later in the course, the threshold.

A vague objective such as predict default is a verb and a column, not a question. A usable question names four things: a decision, a measure of success, a scope (this portfolio, this month) and an owner who will act on the answer. When one of the four is missing, the project tends to optimise a number nobody reads.

The target is a dated trace

The target is the column the model learns, and it is never the phenomenon itself. In the credit data it is late payment the following month, recorded by the bank's own rule for October 2005. A client who pays late in October and settles in November counts as a default; a client who vanishes with the full credit line in December does not. So the target is neither a "bad customer" nor a loss in euros. The first question to ask a business team is how the column was built, by whom, and on which date its value is known.

Back to contents.

Two metrics, the accuracy trap and the guard-rails

One number helps the data team choose between models, another tells the business whether to deploy, and neither of them is accuracy.

The deck separates a model metric and a business metric because they have different readers. The model metric (log-loss, AUC, Brier score) is computed at every training run on the validation split and must react to small improvements. The business metric is expressed in euros lost or good customers refused, read by the credit committee after deployment. Both are fixed in the decision log before the first training run.

As soon as one class is rare, accuracy flatters a model that has learned nothing. This constant model is still useful: it becomes the first baseline of the project, and its log-loss, which equals the entropy of the target, is the floor any real model has to beat.

Guard-rails, written before training

• What the model never does. It does not refuse a client on its own, and it does not use sex, directly or through a proxy. Marital status stays as an input, under audit: kept only if it adds information beyond payment history and if the gap in treatment it creates can be justified.
• Who signs. A named person validates the threshold and the deployment.
• When to stop. The stopping criteria (AUC below the baseline, input drift, a substantiated complaint) are written before the model exists, so they cannot be bent to fit the result.
• Regulation. Credit scoring of natural persons is a high-risk use under Annex III, point 5(b) of the EU AI Act (Regulation (EU) 2024/1689), which brings documentation, human oversight and logging obligations.

Back to contents.

Data leakage: point-in-time join, frozen split, leakage map

A model that saw information from after the prediction date looks brilliant in development and fails in production, so the data is prepared in a strict order.

The deck presents the project as a loop rather than a pipeline: an evaluation can reveal a leak, the leak changes the data map, the map changes the question. Each turn is recorded in a decision log with the date, the alternative rejected, the reason and a number that a command reproduces. Within one turn, though, the order of the data steps is fixed.

The data steps, in order

1. Sources. List each table with its licence and the exact definition of every column.
2. Point-in-time join. To predict at date $t$, keep only the records dated $t' \le t$. In the credit data the features stop at the end of September 2005 and the target is October, so anything known later is forbidden.
3. Frozen split. Split train, validation and test, stratified on the target so each part keeps 22.1 % of defaults, and fix it with a random seed and the SHA-256 hash of the file. The test set is opened once, at the milestone.
4. Leakage map. One line per column: when it is known, whether it may be used, and why. It is written from the column definitions, so it can be done before looking at the data.
5. Model. Only now, and the first model is a constant.

Back to contents.

Least squares and the normal equations

We look for the line that makes the sum of squared vertical gaps as small as possible, and setting the slope of that sum to zero gives a small linear system.

Write the $n$ observations as a vector $y$ and the variables as a matrix $X \in \R^{n\times p}$ whose first column is all ones, for the intercept. The model is $y = X\beta + \varepsilon$, and ordinary least squares picks the $\beta$ that minimises the squared norm of the residual.

Expanding uses only two facts: the transpose of a product reverses the order, and a scalar equals its own transpose, so the two cross terms merge. Differentiating uses two rules, the gradient of $\beta\T b$ is $b$ and the gradient of $\beta\T A\beta$ is $2A\beta$ when $A$ is symmetric.

A zero gradient is enough because $J$ is convex: its Hessian $2X\T X$ is positive semi-definite (the proof comes with convex optimisation). $X\T X$ is invertible exactly when no column of $X$ can be rebuilt from the other columns, which rules out, for instance, the same amount in euros and in thousands of euros.

Worked example: five clients

The deck uses five illustrative clients: tenure $x = (1, 2, 3, 4, 5)$ years and average balance $y = (2, 3, 5, 4, 6)$ in thousands of euros. A hand-drawn line $\hat y = 1 + x$ leaves a sum of squares of 2.0. Can another line do better?

Step by step

1. Four sums. $n = 5$, $\sum x_i = 15$, $\sum x_i^2 = 55$, $\sum y_i = 20$, $\sum x_i y_i = 69$, so $X\T X$ has rows $(5, 15)$ and $(15, 55)$, and $X\T y = (20, 69)$.
2. Two equations. $5\beta_0 + 15\beta_1 = 20$ and $15\beta_0 + 55\beta_1 = 69$. Subtracting three times the first from the second leaves $10\beta_1 = 9$.
3. Solution. $\hat\beta_1 = 0.9$, then $\hat\beta_0 = (20 - 13.5)/5 = 1.3$. The line $\hat y = 1.3 + 0.9x$ leaves 1.9, less than 2.0.
4. Check. At $(1, 1)$ the gradient is $(0, 2)$, not zero: the hand-drawn line was not the minimum.

Back to contents.

The fitted values are a projection: hat matrix, Xᵀe = 0, R²

The best line is the point of the plane spanned by the columns that lies closest to the data, so the leftover error stands at a right angle to that plane.

Substituting $\hat\beta$ into $\hat y = X\hat\beta$ makes a matrix appear in front of $y$, the hat matrix, named for turning $y$ into $\hat y$. It is $n \times n$, while $X\T X$ is only $p \times p$.

The two identities on the right say that $H$ is an orthogonal projection: projecting twice changes nothing, and the projection is not oblique. Every $X\beta$ lies in the column space of $X$, while the observed $y$ does not. The closest point of that space is the foot of the perpendicular, $\hat y$, and the residual $e = y - \hat y$ is perpendicular to every column. This is the zero-gradient condition read as geometry.

The second formula is a proof of least squares without derivatives: the gap to any other line is the residual plus a piece inside the plane, and Pythagoras makes the cross term vanish.

On the same numbers, Pythagoras holds to the decimal: $\lVert y\rVert^2 = 90 = 88.1 + 1.9$, and for the hand-drawn line $2.0 = 1.9 + 0.1$. The trace of $H$ equals $p = 2$, a quick sanity check.

R² needs the intercept

With the five clients, predicting the mean 4 for everyone leaves $\text{SST} = 10$, the line leaves $\text{SSE} = 1.9$, so $R^2 = 0.81$. The split of the total variance relies on the residuals summing to zero, which is only guaranteed when the column of ones is in the model. Without it, the usual formula can go negative: the deck shows a two-point fit through the origin that gives $R^2 = -3$. A negative R² in a report usually means a forgotten intercept, or a test score computed after a leak.

Back to contents.

Condition number and SVD: why we never invert XᵀX

Computers round every number, and forming XᵀX squares how much those rounding errors can grow, so libraries solve least squares another way.

The singular value decomposition writes $X = U\Sigma V\T$, where $U$ has orthonormal columns spanning the same plane as $X$, $\Sigma$ holds the singular values $\sigma_1 \ge \dots \ge \sigma_p > 0$, and $V$ is a rotation of the coefficients. In that basis the solution needs no matrix inverse, only divisions, and the hat matrix becomes $UU\T$.

Each component of $y$ along $u_j$ is divided by $\sigma_j$, so a small singular value amplifies whatever lands in its direction, noise included. The condition number $\kappa$ bounds that amplification: a relative error on the data can grow by up to $\kappa$ on the coefficients. Forming $X\T X$ squares it. With $\kappa = 10^4$, the machine works at $10^8$, which already spends half the significant digits of a double. This is why np.linalg.lstsq goes through the SVD and other solvers through QR, never through $(X\T X)^{-1}$.

The practical answer is to standardise the columns and to record np.linalg.cond(X) before delivering a model. In the lab, on raw California Housing, $\kappa(X)$ is around $2\cdot 10^5$, and after standardisation it falls to about 7 and the three implementations agree to $10^{-13}$.

Multicollinearity

When two columns are almost parallel, the direction that separates them is barely present in the data, its singular value is tiny, and the variance $\sigma^2/\sigma_j^2$ along it explodes. The deck adds to the five clients a second, slightly noisy measure of tenure, correlated at 0.998 with the first. The coefficients jump to $-1.29$ and $+2.26$ with standard errors near 5, and $\kappa(X)$ goes from 8.4 to 85. Their sum, 0.97, stays close to the original slope 0.9, and the prediction for client 3 only moves from 4.00 to 4.23: the data pin down the total effect, not how to share it. Remedies are dropping a column or ridge regularisation, which adds $\lambda$ to every $\sigma_j^2$.

import numpy as np

x = np.array([1, 2, 3, 4, 5.])
y = np.array([2, 3, 5, 4, 6.])
X = np.column_stack([np.ones(5), x])

# SVD route: divisions by singular values, no inverse
U, s, Vt = np.linalg.svd(X, full_matrices=False)
beta = Vt.T @ ((U.T @ y) / s)      # [1.3, 0.9]
e = y - X @ beta
print(X.T @ e)                     # ~[0, 0]: residual orthogonal
print(s[0] / s[-1])                # condition number, about 8.4

Back to contents.

Key takeaways

1. Write the decision, its owner and the cost of each error before touching the data; the target, the metric and the threshold follow from it.
2. The target is a dated record made by a rule, not the phenomenon, and accuracy is misleading as soon as one class is rare.
3. Protect the evaluation in a fixed order: point-in-time join, split frozen by seed and file hash, leakage map, then a constant baseline.
4. Least squares solves the normal equations $X\T X\hat\beta = X\T y$, and geometrically, $\hat y = Hy$ is an orthogonal projection and $X\T e = 0$.
5. R² splits the variance cleanly only when the model contains the intercept; a negative R² usually signals a missing constant or a leak.
6. Check $\kappa(X)$ before delivering, standardise the columns, and solve through SVD or QR, never through $(X\T X)^{-1}$.

Glossary

Ordinary least squares (OLS)

Estimation of a linear model by minimising the sum of squared residuals.

Normal equations

The $p \times p$ system $X\T X\hat\beta = X\T y$ that the least-squares solution satisfies.

Hat matrix

$H = X(X\T X)^{-1}X\T$, which maps the observations $y$ to the fitted values $\hat y$.

Orthogonal projection

A linear map with $H^2 = H = H\T$, which sends a vector to the closest point of a subspace.

Residual

The gap $e = y - \hat y$ between observed and fitted values; in least squares it is orthogonal to the columns of $X$.

Coefficient of determination (R²)

Share of the variance around the mean removed by the model, $1 - \text{SSE}/\text{SST}$.

Singular value decomposition (SVD)

Factorisation $X = U\Sigma V\T$ into orthonormal columns, singular values and a rotation.

Condition number

Ratio of the largest to the smallest singular value; it bounds how much relative errors can be amplified.

Multicollinearity

Near-linear dependence between columns, which inflates the variance of individual coefficients.

Data leakage

Information unavailable at prediction time that reaches the training data and inflates scores.

Point-in-time join

A join that keeps, for a prediction at date $t$, only records known at or before $t$.

Leakage map

A table with one line per column stating when it is known, whether it may be used, and why.

Baseline

The simplest reference model, here a constant prediction, that any useful model must beat.

Guard-rail

A limit on what the model may do, written before training: excluded variables, signatory, stopping criteria.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Kaggle, GitHub, Python: the toolsNext: Maximum likelihood and logistic regression

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.