Maximum likelihood, logistic regression and honest validation, step by step

From the Bernoulli likelihood to the log-loss

A good classifier gives a high probability to what actually happened, and the log-loss is simply the price it pays when it does not.

Each client either defaults ($y_i = 1$) or does not ($y_i = 0$). Model that outcome as a biased coin whose bias $p_i$ depends on the client's features $x_i$. This is a Bernoulli distribution. A neat trick with exponents writes the probability of the observed outcome without any case distinction, since $p_i^{y_i}(1-p_i)^{1-y_i}$ equals $p_i$ when the client defaulted and $1-p_i$ when he did not.

Assuming clients are independent given their features, the probability of the whole dataset is the product of these terms. That product, seen as a function of the parameters $\beta$, is the likelihood. Maximum likelihood estimation (MLE) chooses the $\beta$ that makes it largest.

Two practical reasons push us to the logarithm. A product of 30,000 numbers below 1 underflows in floating point, and a sum is easier to differentiate than a product. Because the logarithm is increasing, the maximiser does not move. Machine learning prefers to minimise, so we flip the sign and obtain the log-loss, also called binary cross-entropy.

For a client who really defaults, only $-\log p_i$ remains. The table shows how harshly the loss treats confident mistakes (natural logarithm, so the unit is the nat).

Back to contents.

Same recipe, Gaussian target: least squares

Least squares is not an arbitrary choice: it is what maximum likelihood gives when the noise around the prediction is Gaussian.

Run the recipe on a continuous target. Suppose the observed price is Gaussian around the linear prediction, $y_i \sim \mathcal N(x_i^{\top}\beta, \sigma^2)$. Each density is a constant times $e^{-e_i^2/(2\sigma^2)}$, where $e_i$ is the residual. Take the log of the product: the constants pile up into a term that does not depend on $\beta$, and the exponents add up into a sum of squared residuals.

A positive factor and an additive constant do not move a minimum, so the Gaussian MLE is exactly the ordinary least squares estimator of the morning, with its normal equations. Had the noise been Laplace instead, the exponent would hold $|e_i|$ and the recipe would give least absolute deviations, the median rather than the mean.

The floor every model must beat

Predict the base rate $\bar p = 0.221$ for every client. Its mean log-loss is the entropy of the target, about 0.528 nats. Any credit model that cannot go below this number on validation data has learnt nothing useful.

The deck also shows the form libraries actually compute, written with the score $z_i = x_i^{\top}\beta$, namely the mean of $\log(1+e^{z_i}) - y_i z_i$. The function $\log(1+e^{z})$ is the softplus, which can be evaluated without overflow even for very negative scores.

Back to contents.

The sigmoid and the logit

The sigmoid squeezes any real score into a probability, and its inverse shows that logistic regression is a straight line on the log-odds scale.

A linear score $z = x^{\top}\beta$ ranges over the whole real line; a probability lives in $]0, 1[$. The sigmoid bridges the two. Since $e^{-z} > 0$, the denominator exceeds 1 and the result stays strictly between 0 and 1: the model never predicts certainty, which also keeps the log-loss away from $\log 0$.

The derivative is the one formula worth memorising: it makes the gradient fall out in a few lines, and its maximum of one quarter at $z = 0$ will later explain why gradients fade in deep sigmoid networks. Solving $p = \sigma(z)$ for $z$ gives the logit, the logarithm of the odds:

Check at a score of 2

1. Exponential. $e^{-2} \approx 0.135$, so the denominator is about 1.135.
2. Probability. $\sigma(2) = 1/1.135 \approx 0.881$.
3. Slope. $0.881 \times 0.119 \approx 0.105$, below the ceiling of 0.25.
4. Back to the score. $\log(0.881/0.119) = \log 7.39 \approx 2.00$, so the logit undoes the sigmoid.

Back to contents.

The gradient, the Hessian and convexity

The direction to move the coefficients is given by the gap between what the model predicted and what happened, and the loss has a single valley.

Each term of the log-likelihood depends on $\beta$ only through the score $z_i$. Differentiate with respect to the score first: thanks to $\sigma' = \sigma(1-\sigma)$, every denominator cancels and what remains is $y_i - p_i$, observed minus predicted. The chain rule then multiplies by $x_i$, and stacking the clients gives a matrix product.

Compare with least squares, whose gradient is $X^{\top}(X\beta - y)$. Same shape, a different residual. The sigmoid is what prevents a closed-form solution, so the coefficients are found iteratively, by gradient descent or by Newton's method (known to statisticians as IRLS).

Differentiating once more gives the Hessian, a $p \times p$ matrix with the same structure as $X^{\top}X$ but weighted by $w_i = p_i(1-p_i)$, numbers between 0 and one quarter. For any direction $v$, the quadratic form $v^{\top}X^{\top}WXv = \sum_i w_i (x_i^{\top}v)^2$ is a weighted sum of squares, hence never negative. The log-loss is therefore convex: every local minimum is global, and it is unique when the columns of $X$ are linearly independent.

Worked example: five clients, one feature (months in arrears)

1. Data. Arrears $x = (0, 1, 3, 0, 2)$, outcomes $y = (0, 0, 1, 0, 1)$, predicted probabilities $p = (0.10, 0.20, 0.80, 0.15, 0.70)$ (illustrative numbers).
2. Gaps. $p - y = (0.10, 0.20, -0.20, 0.15, -0.30)$, so the two defaulters are underestimated.
3. Gradient. Weight each gap by $x_i$ and add: $0 + 0.20 - 0.60 + 0 - 0.60 = -1.00$. Negative, so increasing the arrears coefficient lowers the loss.
4. Hessian. $\sum_i p_i(1-p_i)x_i^2 = 0.16 + 1.44 + 0.84 = 2.44 > 0$, so the loss curves upwards.
5. Newton step. Minus the gradient over the Hessian, $1.00/2.44 \approx 0.41$, is how much the coefficient would rise in one step.

Back to contents.

Softmax and cross-entropy for several classes

With more than two classes, each class gets a score, the softmax turns the scores into probabilities, and the gradient keeps the same predicted-minus-observed shape.

Give each of the $K$ classes its own linear score, exponentiate to make them positive, and divide by their sum. The target is a one-hot vector, so the cross-entropy keeps a single term: minus the log of the probability given to the true class. Differentiating through the log-sum-exp gives a strikingly simple result.

With scores $(2, 1, 0)$ the exponentials are about 7.389, 2.718 and 1, summing to 11.107; the probabilities are 0.665, 0.245 and 0.090. If the true class is the first, the loss is $-\log 0.665 \approx 0.408$ and the gradient $(-0.335, 0.245, 0.090)$ raises the true score and lowers the others. It sums to zero, because adding the same constant to every score changes nothing.

With two classes, dividing top and bottom by $e^{z_1}$ gives $\sigma(z_1 - z_2)$. Binary logistic regression is the softmax with $K = 2$, and only the difference of the scores matters, which is why it needs a single coefficient vector. The same layer closes almost every classification network (see the note on attention).

Back to contents.

Reading a coefficient as an odds ratio

A logistic coefficient multiplies the odds by a fixed factor, but the change in probability depends on where the client starts.

The odds are $p/(1-p)$, so a client at 10 % has odds of 1 to 9, about 0.111. Since the logit is linear, raising $x_j$ by one unit, everything else fixed, adds $\beta_j$ to the log-odds and multiplies the odds by $e^{\beta_j}$, the odds ratio. That factor is the same for every client, which is what makes it the right number to publish.

The interval is built on the coefficient, where the normal approximation holds, then exponentiated, so it is asymmetric around the odds ratio. The standard error comes from the inverse Hessian $(X^{\top}WX)^{-1}$ at the estimate.

Same odds ratio, twice the effect in probability. Writing "the risk rises by 82 %" is therefore wrong, and validators spot it at once. A sound report gives the odds ratio with its interval, plus the probability before and after for a typical client. When categorical variables are one-hot encoded, it also names the reference level.

Back to contents.

Bias-variance and validation without leakage

Test error has three sources, only one of which you control, and measuring it honestly means no information from the test set may reach the model.

For $y = f(x) + \varepsilon$ and a model $\hat f$ fitted on a random sample $D$, the expected squared error at a point splits into irreducible noise, squared bias (the systematic gap of a family that is too simple) and variance (how much the fit moves when the sample is redrawn). The proof adds and subtracts the mean prediction twice, and both cross terms vanish.

In practice the three terms are never seen separately; what you see is a training error that keeps falling and a validation error that turns back up. Cross-validation estimates that second curve without touching the test set: split the remaining rows into $k$ stratified folds, train on $k-1$, score the held-out fold, average. The mean is nearly unbiased for the procedure, but the displayed standard deviation is not a confidence interval, since the training sets overlap heavily (Bengio and Grandvalet, 2004).

Data leakage is any statistic of the test data that enters the model. The two classics: scaling features on the whole file before splitting (fix: a Pipeline fitted inside each fold), and target encoding computed on all rows, so each row sees its own label (fix: out-of-fold encoding). On dated data, shuffled folds let the model train on the future; a TimeSeriesSplit trains on the past and validates on what follows.

Finally, how far can a test score be trusted? Hoeffding's inequality bounds the gap between measured and true error for a loss in $[0, 1]$ and independent rows, with no assumption on the model. Fixing the risk $\delta$ and solving gives the margin:

With $\delta = 0.05$ and 1,000 test rows, $\ln 40 \approx 3.689$, divided by 2,000 and square-rooted, gives about 0.043: two models at 81 % and 84 % cannot be told apart. The margin shrinks like $n^{-1/2}$, so halving it costs four times the rows; the project's 6,000 test rows give about 1.8 points. The guarantee only holds if the test set is looked at once.

The kick-off of the team project applies all of this to the credit data: a SQL audit with DuckDB, a datasheet stating exactly what the target means, a leakage map dating every column against the prediction date, a stratified 60/20/20 split frozen by a seed and a file hash, and two baselines (the constant model at 0.528 and a default logistic regression in a pipeline). The project page follows it further.

Back to contents.

Key takeaways

1. Choosing a distribution for the target chooses the loss: Gaussian noise gives least squares, a Bernoulli target gives the log-loss.
2. The sigmoid maps a score to a probability, its derivative is $\sigma(1-\sigma)$, and its inverse, the logit, is linear in the features.
3. The logistic gradient is $X^{\top}(p - y)$, the same shape as for least squares; the Hessian $X^{\top}WX$ makes the loss convex.
4. A coefficient multiplies the odds by $e^{\beta_j}$. Report the odds ratio with its interval and the probabilities before and after, never a percentage change in risk.
5. Test error is noise plus squared bias plus variance; only the trade-off between the last two depends on your modelling choices.
6. Any test-set statistic that reaches the model is a leak: fit scaling and encodings inside the folds, and respect time order on dated data.
7. A test score has a margin of roughly $\sqrt{\ln(2/\delta)/2n}$, so a few hundred test rows cannot separate close models.

Glossary

Likelihood

The probability of the observed data, seen as a function of the model parameters.

Maximum likelihood estimation (MLE)

Choosing the parameters that make the observed data most probable.

Log-loss (binary cross-entropy)

Minus the mean log-likelihood of a 0/1 target; the loss minimised by logistic regression.

Sigmoid

The function $1/(1+e^{-z})$ that maps a real score to a probability strictly between 0 and 1.

Logit (log-odds)

The logarithm of $p/(1-p)$, the inverse of the sigmoid; linear in the features in logistic regression.

Odds ratio

The factor $e^{\beta_j}$ by which the odds are multiplied when a feature rises by one unit.

Hessian

The matrix of second derivatives of the loss; positive semi-definite here, hence convexity.

Softmax

Turns K scores into K probabilities by exponentiating and normalising.

Bias-variance trade-off

Simpler models err systematically, flexible ones vary with the sample; the test error adds both to the noise.

Cross-validation

Averaging the error over k folds, each held out once, to estimate the error of a procedure.

Data leakage

Any information from the test data, or from the future, that enters the training of the model.

Target encoding

Replacing a category by the mean target of that category; leaks unless computed out of fold.

Hoeffding's inequality

A bound on the gap between measured and true error that needs only a bounded loss and independent rows.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Project framing and least squaresNext: Gradient descent, ridge and lasso

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.