A Panorama of Machine Learning Models: Which Ones, What For, in Which Sectors

The map: six families and one grid

Before choosing a model, decide what kind of question you are asking: predict a known answer, find structure without one, or act on a system.

Supervised learning has examples with the right answer and learns to reproduce it: a number (regression) or a class (classification). Unsupervised learning has no answer and looks for groups, axes or odd points. A third block gathers data with a particular shape, each of which has its own models: an ordered series, a sparse matrix of users and items, a loop of actions and rewards, a distribution to imitate.

Two boxes sit apart. Causal inference asks what happens if we intervene, not what will happen. Operations research makes no forecast at all: it takes decisions from a model the analyst writes explicitly. Both belong on the map because a forecast is rarely the end of the work.

Back to contents.

Linear models: regression, GLM, logistic

A linear model adds up the effects of the variables; its strength is that every coefficient can be read, tested and defended in front of a regulator.

Linear regression finds the coefficients that bring the predictions closest to the data in squared error. Ridge and lasso put a price on large coefficients: ridge stabilises them, lasso sets some to exactly zero and therefore selects variables (see regularisation).

Generalised linear models keep the linear score but pass it through a link function, so the same idea covers counts (Poisson, log link), positive skewed amounts (Gamma, log link) and yes/no outcomes (binomial, logit link). They are fitted by maximum likelihood, and a coefficient reads as a multiplicative effect $e^{\beta_j}$. Insurance tariffs have rested on them for decades because each coefficient must be justified.

Logistic regression is the binomial case. Its loss is convex, so it always reaches the same solution, and it remains the model banks deploy most for credit scoring, often as a points scorecard built on binned variables. With rare classes, accuracy is misleading: judge it on AUC and calibration, and set the decision threshold from the costs (see decision threshold).

• Libraries. LinearRegression, Ridge, Lasso, LogisticRegression and the PoissonRegressor family in scikit-learn; statsmodels when you need standard errors and intervals; optbinning for scorecards.
• Traps. Multicollinearity makes coefficients unstable; never extrapolate outside the data cloud; with counts, check overdispersion before trusting a Poisson standard error.
• When not to. Strong non-linearities, thresholds or interactions that nobody needs to read: a boosting model will do better, including with a Poisson or Tweedie objective.

Back to contents.

Trees, ensembles, kernels and neighbours

One tree is easy to read but unstable; averaging many trees, or adding them one by one to fix each other's errors, gives the strongest models for tables.

A decision tree splits the space with one threshold on one variable at a time, choosing the split that most reduces impurity. It can be drawn and applied by hand, which is why it appears in lending rules and triage protocols. Its flaw is instability: change a few rows and the first split, then the whole tree, moves.

A random forest averages hundreds of trees, each grown on a bootstrap sample and with a random subset of variables at each node. The averaging reduces the variance, not the bias, and the random draws exist to make the trees less correlated, as this formula for the variance of an average of $B$ estimators with pairwise correlation $\rho$ shows.

Gradient boosting adds small trees one after another, each fitted to the negative gradient of the loss of the current model: gradient descent carried out in the space of functions. It has dominated tabular competitions for a decade. LightGBM is the fast one, XGBoost the reference with monotonic constraints useful in credit, CatBoost handles categorical variables natively. It overfits if left running, so use early stopping, and calibrate its probabilities before setting a threshold (details in forests and boosting).

Support vector machines draw the widest possible margin between two classes, and only the points on its edge matter. The dual problem sees the data only through dot products, so a kernel can replace them and bend the boundary without ever computing in the larger space (see kernels, SVM and trees). With a kernel, the cost grows between $O(n^2)$ and $O(n^3)$, which is fine for a few thousand rows and painful beyond tens of thousands.

Two simpler families complete the block. k-nearest neighbours stores the examples and lets the closest ones vote; it fails in high dimension unless the data are reduced first, and it has returned in force as similarity search over embeddings with approximate indexes such as faiss. Naive Bayes multiplies the evidence of each variable as if they were independent: poor probabilities, often a good ranking, and the classic five-minute text baseline with TF-IDF.

Back to contents.

Neural networks: when the data are not a table

Neural networks earn their cost on images, text, sound and sequences; on an ordinary table of a few thousand rows, a boosting model usually wins.

• Multilayer perceptron. Stacked affine layers and non-linearities trained by backpropagation and Adam on a non-convex loss, so the random seed matters. Useful as a fast surrogate for a slow simulation; rarely the best choice on medium tabular data (see neural networks).
• Convolutional networks (CNN). Small filters with shared weights slide over the image. Nobody trains one from scratch on a thousand images: take a pretrained ResNet or EfficientNet and retrain the head. Typical trap: the network learns the camera or the lighting of one production line, so test on another site.
• Recurrent networks and LSTM. Read a sequence step by step with a memory; the LSTM gates keep the gradient alive over long sequences. Still an industrial standard for forecasting many related series (DeepAR), replaced by transformers on text.
• Transformers and large language models. Each token weighs every other token through attention, computed as matrices rather than step by step, which is what makes training on huge corpora possible.
• Generative models. VAEs, GANs and diffusion learn the distribution of the data itself in order to sample new examples. The serious business use is synthetic data, with a check that no real client has been memorised.

The attention article builds this formula step by step. In companies, the established uses of language models are reading, triaging and searching documents, tasks where a human checks the output. They should not predict a default from a table, and where a regular expression or a logistic regression does the job, those are far cheaper. On credit or recruitment, the high-risk obligations of Annex III of the EU AI Act apply whatever the model.

Back to contents.

Unsupervised learning: group, compress, spot the odd one

Without labels there is no score to tell you that a grouping is right; it is only useful or useless for a decision.

k-means alternates two steps, assign each point to the nearest centre and move each centre to the mean of its points, which lowers the inertia at each round and stops in a local minimum; k-means++ and several restarts help. You choose $k$, by silhouette or by business sense. Hierarchical clustering with Ward's criterion does not ask for $k$ and shows every cut in a dendrogram, at a quadratic cost.

DBSCAN defines a group as a dense region: points with enough neighbours within a radius chain together, the rest is labelled noise. It finds crescents and needs no $k$, and is the usual tool for finding stops in GPS traces; one radius cannot fit very different densities, which HDBSCAN addresses. Gaussian mixtures give each point a probability of belonging to each elliptical group, fitted by expectation-maximisation; k-means is their limit case.

Principal component analysis is the SVD of the centred data: the first singular vectors are the directions of largest variance (see PCA and clustering). Three components summarise most moves of a yield curve, and monitoring the residual of a projection is a classic of process control. It ignores the target, so it is not a variable selection method. t-SNE and UMAP flatten a cloud onto a plane while keeping neighbours together: precious for looking at embeddings, misleading if the distances between islands are read as facts.

Anomaly detection scores how unusual a point is. Isolation forest counts how many random cuts it takes to isolate a point: few cuts, high score, linear cost. The key caution: an anomaly is not a fraud. A customer who moves house is unusual and honest. The score feeds a review queue; once confirmed fraud labels exist, a supervised model takes over.

Back to contents.

Time series, recommendation, bandits, causality, solvers

When the data have an order in time, a user-item matrix, a reward or an intervention, the generic models are no longer the right tool.

For forecasting there are two schools. On one short, regular series, ARIMA and exponential smoothing (ETS) are hard to beat and give an interval; Prophet adds holidays for the analyst in a hurry. On thousands of related series with prices, weather and promotions, the winning recipe is to turn time into a table of lags, rolling windows and calendar features, and to fit one global LightGBM model, as the winners of the M5 competition on retail sales did (see the asset price note). In both cases, validate by moving forward in time, compute every window from the past only, and always compare with the seasonal naive forecast.

Recommender systems factorise the nearly empty users × items matrix into two small matrices whose product rebuilds it; alternating least squares solves one side then the other. In production a fast model proposes a few hundred candidates and a boosting model re-ranks them. Beware cold start and the feedback loop: the system only learns from what it has shown.

Reinforcement learning maximises a discounted sum of rewards; full versions need a simulator or millions of cheap trials. The multi-armed bandit, which has no state, is the form you will certainly meet: choose which banner to show, explore a little while exploiting what works (Thompson sampling, UCB).

Causal inference and uplift estimate the effect of an action on a given person, a difference between two worlds of which only one is observed. A randomised test makes it reliable; without one, everything rests on the assumption that no confounder is missing, and the report must say so. A good predictive score is not a causal model.

Operations research is not learning: a linear or integer programme, written by hand, decides routes, schedules or assignments with a solver such as OR-Tools, HiGHS or Gurobi. The junction with ML is the trap: a point forecast enters the optimiser as if it were certain. Pass it an interval or scenarios.

Back to contents.

How to choose a model: data, task, sector

Start from the shape of the data and the task, take a modest model that gives an honest number within an hour, then try one stronger challenger.

Five questions, in this order, before opening a notebook

1. How many rows? Around a thousand calls for a regularised linear model; a boosting model would learn the noise. With a hundred thousand, everything is open.
2. Is it a table? A table points to boosting; images, text or sound point to a pretrained network.
3. Who must understand the output? If a regulator or a committee must, keep to logistic regression, a GLM or a tree. In credit and hiring this is a legal constraint, not a preference.
4. How fast must it answer? Real-time scoring favours a linear model; a nightly batch leaves the choice free.
5. What does an error cost? If it is expensive, calibrate, set the threshold from the costs, and keep a human in the loop.

What practitioners actually observe: on tables, boosting wins almost always, and by a small margin; on images, text and sound, everyone fine-tunes a pretrained network; on series, the seasonal naive baseline wins more often than people admit. Everywhere, the data, the split, the metric, the threshold and the monitoring weigh more than the model. In the course project on credit default, four models are compared (regularised logistic regression, a forest, boosting, a kernel model) with the same splits, several seeds and confidence intervals; the code is in the Python code note.

Back to contents.

Key takeaways

1. Classify the problem first: supervised, unsupervised, a special data shape (series, matrix, rewards), or not a learning problem at all (causal effect, optimisation).
2. On tabular data, start with a linear or logistic model and challenge it with gradient boosting; keep the challenger only if its gain changes a decision.
3. On images, text and sound, fine-tune a pretrained network rather than training from scratch.
4. Unsupervised results are judged by their usefulness for a decision, never as true or false; an anomaly is not a fraud.
5. For forecasting, use ETS or ARIMA on one series and lag features with boosting on many, always validated forward in time against a seasonal naive baseline.
6. Explainability is a regulatory requirement in credit and hiring, which are high-risk under Annex III of the EU AI Act.
7. Every model has a situation where it should not be used; knowing it is worth more than knowing the definition.

Glossary

Generalised linear model (GLM)

A linear score passed through a link function, fitted by maximum likelihood; covers counts, amounts and binary outcomes.

Logistic regression

The binomial GLM with a logit link; convex, readable, the reference model for credit scoring.

Random forest

An average of decorrelated trees grown on bootstrap samples; lowers variance with little tuning.

Gradient boosting

Trees added one at a time, each fitted to the negative gradient of the loss of the current model.

Support vector machine (SVM)

A classifier that maximises the margin; a kernel replaces dot products to bend the boundary.

k-nearest neighbours (kNN)

Predicts from the closest stored examples; needs scaling and suffers in high dimension.

Multilayer perceptron (MLP)

Stacked affine layers and non-linearities trained by backpropagation.

Convolutional neural network (CNN)

A network of shared-weight filters sliding over an image or signal; used pretrained in practice.

Large language model (LLM)

A transformer pretrained to predict the next token, then aligned to follow instructions.

Gaussian mixture model (GMM)

A density written as a weighted sum of Gaussians, fitted by expectation-maximisation; gives soft memberships.

Principal component analysis (PCA)

Projection on the directions of largest variance, obtained from the SVD of centred data.

Isolation forest

An anomaly score based on how few random cuts isolate a point.

Exponential smoothing (ETS)

A forecasting filter that updates level, trend and season with weights that forget the past.

Uplift

The effect of an action on a given individual, estimated reliably only with randomisation or explicit assumptions.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Data engineering, SRE and cloud

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.