Explaining a Credit Score with SHAP, Auditing its Fairness, and Finding Structure with PCA, k-means and EM

Shapley values: sharing a prediction among features

A Shapley value splits the gap between one client's prediction and the average prediction fairly among the features, by averaging what each one adds.

A model gives a client a 62 % default probability while its average prediction is 22 %. Forty points need an owner. Lloyd Shapley answered the same question for players in 1953: if a coalition wins together, how should the gain be shared? Here the players are features, and the value $v(S)$ of a coalition $S$ is the prediction when only the features in $S$ are known and the others are averaged out.

The cleanest definition uses arrival orders. Bring the features in one at a time, credit each with the change it causes when it arrives, then average over all $n!$ orders. Averaging matters because a feature's contribution depends on who is already there: interaction is shared, not given to whoever arrives last. Grouping the orders by the set $S$ of features already present gives the classical formula, a weighted average of marginal contributions:

The weight counts the orders in which exactly $S$ precedes $i$, namely $|S|!$ ways to arrange $S$ before, $(n-|S|-1)!$ ways to arrange the rest after, out of $n!$. For three features the weights are $2/6$, $1/6$, $1/6$ and $2/6$, and they sum to one.

Worked example: three features, one client

The deck uses illustrative values for the credit features PAY_0 (latest repayment delay), LIMIT_BAL (credit limit) and AGE. In percent, $v(\varnothing)=22$, $v(\{P\})=52$, $v(\{L\})=28$, $v(\{A\})=22$, $v(\{P,L\})=58$, $v(\{P,A\})=54$, $v(\{L,A\})=30$ and $v(\{P,L,A\})=62$. Each row of the table gives what each feature adds in one arrival order.

PAY_0 receives 31 points, LIMIT_BAL 7 and AGE 2, and $31 + 7 + 2 = 40 = 62 - 22$. AGE is the instructive one: alone it adds nothing, next to the other two it adds 4, and its share is the average over contexts. The same 31 comes out of the coalition formula: $(2\cdot 30 + 30 + 32 + 2\cdot 32)/6 = 186/6$.

Back to contents.

SHAP and TreeSHAP in practice, and what they do not say

SHAP turns every prediction into the average plus one share per feature; it is exact and fast for trees, but it describes the model, not the world.

Lundberg and Lee (NeurIPS 2017) made the translation explicit: players are features, $v(S)$ is the expected prediction given the known features, and every prediction decomposes as the average output plus one share per feature. Their paper also places older attribution techniques, LIME included, inside this one additive family.

The obstacle is cost. The 23 features of the UCI credit card dataset give $2^{23} = 8{,}388{,}608$ coalitions for a single client. KernelSHAP samples coalitions and works on any model, approximately and slowly. TreeSHAP (Lundberg et al., Nature Machine Intelligence, 2020) walks the tree paths and computes the exact values for trees, forests and boosting in polynomial time. On a toy tree of the deck, LIMIT_BAL receives 4.2 points although the client's path never tests it: $v(S)$ also reasons about clients who would have gone the other way, so a waterfall is not the path the model followed.

Locally, the waterfall shows one client going from the average to their prediction. Globally, the mean absolute share ranks features; the absolute value stops pushes up and down from cancelling, at the price of losing the direction, which the beeswarm plot gives back with one dot per client coloured by the feature's value. Check the scale: for boosting, exact additivity holds on the log-odds margin, not on probabilities.

• Correlated features. The six monthly bill amounts are near twins; how their credit is split depends on the definition of $v(S)$ (interventional or conditional), not on the world. Group them and read the sum.
• Explaining is not causing. A high credit limit lowers the predicted risk because banks grant high limits to clients already judged safe. Raising a fragile client's limit is a different, causal question.
• After-the-fact stories. Explanations can be fooled (Slack et al., 2020) or cherry-picked across settings. Fix the explainer, background sample and scale before looking, cross-check with permutation importance, and for high stakes prefer a model that is readable by construction (Rudin, 2019).

Back to contents.

Fairness measures: what the score does to groups

Removing the sensitive column does not make a score fair; you have to measure, group by group, how often it rejects, how often it is wrong, and whether its probabilities hold.

Dropping SEX from the inputs, called fairness through unawareness, was shown insufficient by Dwork and co-authors in 2012: other columns can carry the same information. These are proxy variables. The test turns the problem around: predict SEX from all the other columns, with the same model family as the champion and cross-validation on the training set. An AUC near 0.5 means little leaks through; further away, the permutation importance of that model names the carriers.

Barocas, Hardt and Narayanan (2023) sort the many measures into three families. With the course's coding, positive means default and $\hat Y = 1$ means rejected, so a false positive is a good client turned down.

Demographic parity is readable without knowing who defaults, so it suits monitoring from day one, but it ignores differing base rates. Equal opportunity (Hardt, Price and Srebro, 2016) asks for the same share of good clients rejected, the false positive rate; equalised odds also equalises the true positive rate. Both need the outcomes, so they are audit measures. Calibration by group is what banks often check first, because a default probability also feeds provisions and capital.

Worked example: two groups of 1,000 applications

Same model, same threshold. Group A: 150 true positives, 100 false negatives, 75 false positives, 675 true negatives. Group B: 120, 80, 60 and 740. The five rates follow.

So the score is fair in the sense of TPR and PPV, and unfair in the sense of rejections and FPR: a good client of group A is wrongly rejected one time in ten, against one in about thirteen in group B. Both readings are correct; they answer different questions. That is why the measure must be chosen, and written down, before the numbers are seen, and why every gap needs a confidence interval: with small groups the honest verdict is often inconclusive.

Back to contents.

Why you cannot equalise every rate at once

When two groups default at different rates, an imperfect score cannot give them the same precision, the same detection rate and the same false alarm rate together.

The example above is not rigged. Chouldechova (2017) wrote an identity that follows from the definitions alone, with $p$ the base rate of the group: the number of defaults caught is $\text{TPR}\cdot pn$, the false positives follow from the PPV, and dividing by the $(1-p)n$ good clients gives the FPR. The group size cancels.

With equal PPV and TPR, the FPR grows with $p/(1-p)$, which is strictly increasing. Check on the two groups: for A, $\tfrac{0.25}{0.75}\cdot\tfrac12\cdot 0.6 = 0.10$, and for B, $\tfrac{0.20}{0.80}\cdot\tfrac12\cdot 0.6 = 0.075$. Exactly the rates of the table.

Kleinberg, Mullainathan and Raghavan proved the score version in 2016: calibration within groups and balance for both classes are incompatible unless prediction is perfect or base rates are equal. Both results came out of the debate on the COMPAS recidivism tool in the United States. The practical consequence is a procedure, not a number: measure by group with intervals, choose the measure that matters for the use and justify it, then correct at the source (a proxy that can be removed, a badly defined target, a badly placed threshold).

Back to contents.

Credit scoring under the EU AI Act

European law classifies creditworthiness scoring of individuals as high risk, which brings duties to examine data for bias, document the system and explain decisions.

Regulation (EU) 2024/1689, Annex III, point 5(b), lists systems that evaluate the creditworthiness of natural persons or establish their credit score as high risk, fraud detection excepted. According to the deck, Regulation (EU) 2026/1744 moved the application of the Annex III rules to 2 December 2027; such dates have already moved once, so the consolidated text should be reread before quoting one.

• Articles 9 to 15: risk management, data governance, technical documentation (Annex IV), automatic logging, transparency to the deployer, human oversight, and accuracy, robustness and cybersecurity.
• Article 10(2)(f): examine the data for biases likely to lead to prohibited discrimination. The proxy test and the group audit are this examination.
• Article 86: a person affected by a decision based on a high-risk system can obtain a clear explanation of the system's role. The SCHUFA judgment of the Court of Justice (C-634/21, 2023) had already brought determinative credit scores under Article 22 of the GDPR.
• Directive 2004/113/EC: indirect discrimination is an apparently neutral criterion that puts one sex at a particular disadvantage without objective justification. The effect counts, not the intention, which is why the audit measures gaps even without SEX in the model.

The deck maps each of these articles onto files a student project already produces: a decision log, a model card in the format of Mitchell et al. (2019), service logs and a reject option that come later in the course (see drift and conformal prediction). For banking supervisors, the European Banking Authority report EBA/REP/2023/28 on machine learning in internal rating models found Shapley values to be the explanation method banks mention most often.

Back to contents.

Principal component analysis through the SVD

PCA turns the axes so that the first direction carries as much of the spread of the data as possible, then the next perpendicular one, and so on.

Without a target, the only material is $X$. Principal component analysis, Pearson's line of closest fit (1901) and Hotelling's direction of maximal variance (1933), looks for the viewpoint from which the cloud looks widest. The variance of the projections on a unit vector $u$ is $u^{\mathsf T}Su$, with $S$ the covariance of the centred columns. Maximising it under $u^{\mathsf T}u = 1$ with a Lagrange multiplier gives an eigenvector equation, and the variance carried is the eigenvalue.

In practice the SVD of the least squares session does the same work without forming $S$. With $X = UDV^{\mathsf T}$, the columns of $V$ are the principal directions, the coordinates are $UD$, and $d_j^2 = (n-1)\lambda_j$. Eckart and Young (1936) add why truncation is legitimate: keeping the first $q$ components is the best rank-$q$ approximation, and the error is the sum of the discarded $d_j^2$.

Five points in two dimensions

1. Centre. The points (1, 1), (2, 3), (3, 2), (4, 4) and (5, 5) have mean (3, 3); three lie on the diagonal, two sit just off it.
2. Covariance. The centred sums of squares are 10 and 10, the cross sum is 9; dividing by $n-1=4$ gives variances 2.5 and covariance 2.25.
3. Eigenvalues. Trace 5, determinant $2.5^2 - 2.25^2 = 1.1875$, so $\lambda_1 = 4.75$ and $\lambda_2 = 0.25$, with $v_1 = (1,1)/\sqrt2$, the diagonal.
4. Read the share. The first component carries $4.75/5 = 95\,\%$ of the variance; the SVD gives $d^2 = (19, 1)$, the same ratio.
5. Check the loss. Projecting on the diagonal moves the two off-axis points by a squared distance of 0.5 each: total error 1, the discarded $d_2^2$.

Back to contents.

k-means and Gaussian mixtures fitted by EM

k-means puts each client in exactly one group around a centre; a Gaussian mixture gives each client a probability of belonging to each group, and EM fits it.

k-means minimises the inertia, the sum of squared distances from each point to its centre. Lloyd's algorithm (written in 1957, published in 1982) alternates two steps, each of which can only lower the inertia: assign each point to the nearest centre, then move each centre to the mean of its group, the minimiser of squared deviations as in least squares. It stops, but at a local minimum that depends on the start.

On the six values 1, 2, 4, 7, 8, 9 started from centres 1 and 4, the inertia falls from 51 to 14.5 after the first iteration (centres 1.5 and 7), then to about 6.67 when the point 4 changes side (centres 7/3 and 8); nothing moves after that. With three groups, a poor start at 7, 8, 9 stalls at 5.17 while the optimum is 2.5, which is why k-means++ (Arthur and Vassilvitskii, 2007) draws spread-out starting centres and why several restarts are kept. The elbow of the inertia curve and Rousseeuw's silhouette help choose $k$.

A Gaussian mixture assumes $K$ bell curves with weights $\pi_k$, centres $\mu_k$ and variances $\sigma_k^2$. The log-likelihood has a sum inside the logarithm and no closed form. EM (Dempster, Laird and Rubin, 1977) alternates an E step, which computes each client's responsibility for each group by Bayes' formula, and an M step, which recomputes weights, means and variances as averages weighted by those responsibilities.

On the four values 1, 2, 5 and 8, starting from bells at 2 and 8 with unit variance and equal weights, the value 5 sits exactly halfway and gets a responsibility of one half for each bell, while the others belong almost entirely to their nearest one. After one M step the second centre is pulled from 8 towards 5, both bells widen, and the log-likelihood rises from about $-10.76$ to $-9.09$, as the 1977 theorem guarantees. Forcing responsibilities to 0 or 1 with equal fixed variances gives back k-means.

Back to contents.

Key takeaways

1. A Shapley value is a weighted average of what a feature adds across all coalitions; the shares add up exactly to the gap between the prediction and the average.
2. TreeSHAP makes the computation exact for trees, forests and boosting, but SHAP describes the model, not a causal effect in the world.
3. Removing a sensitive column is not enough: test whether the other columns predict it, then measure rates group by group with intervals.
4. With different base rates, an imperfect score cannot equalise PPV, TPR and FPR together, so the fairness measure is a choice to write down before computing.
5. Credit scoring of individuals is a high-risk use under the EU AI Act, with duties on data bias, documentation, oversight and explanation.
6. PCA is the SVD of the centred, usually standardised, columns; the eigenvalues give the variance each direction carries.
7. k-means and EM share one scheme, alternating two steps that never make the objective worse, and both need several starts.

Glossary

Shapley value

The share of a coalition's gain given to one player, averaged over all arrival orders; in machine learning, a feature's share of one prediction.

Marginal contribution

What a feature adds to the prediction when it joins a given set of features already known.

Proxy variable

A column correlated with a sensitive attribute, through which a model can recover it even when the attribute is removed.

Demographic parity

The requirement that the rejection rate be the same in every group.

Equal opportunity

The requirement that qualified people, here good clients, have the same chance of the favourable decision in every group.

Base rate

The share of actual positives, here defaults, in a group; a fact of the data, not a choice of the model.

Calibration by group

The requirement that a given score corresponds to the same observed default rate in every group.

Principal component analysis

A change of orthonormal basis that orders directions by the variance they carry, computed from the covariance or the SVD.

k-means clustering

Partitioning points into k groups that minimise the sum of squared distances to their centres.

Inertia

The k-means objective: the total squared distance of each point to its group centre.

Gaussian mixture model

A density written as a weighted sum of Gaussian bells, giving each point a probability of belonging to each component.

Responsibility

In EM, the posterior probability that a point comes from a given component, computed with Bayes' formula.

The other notes of the course

One public summary per deck, in the order of the course.

Previous: Forests, boosting, honest comparisonNext: A documented champion model

1. IntroductionAI winters, jobs and method
2. ToolsKaggle, GitHub, Python: the tools
3. Session 1Project framing and least squares
4. Session 2Maximum likelihood and logistic regression
5. Session 3Gradient descent, ridge and lasso
6. Session 4Decision threshold and reproducible ML
7. HomeworkA defensible baseline model
8. Session 5Kernels, SVM and decision trees
9. Session 6Forests, boosting, honest comparison
10. Session 7SHAP, fairness, PCA and clustering
11. HomeworkA documented champion model
12. Session 8Neural networks, drift, conformal prediction
13. HomeworkA delivered machine learning project
14. Session 9Defending a machine learning project
15. ReferenceCredit default scoring project
16. ReferenceAsset returns: XGBoost and sentiment
17. ReferencePython code for every ML model
18. ReferenceDeep learning essentials
19. ReferenceOpen data ML projects by sector
20. ReferenceData and AI engineer handbook
21. ReferenceData engineering, SRE and cloud
22. ReferencePanorama of machine learning models

All the notes, as a reading list

Attention and transformers, step by step

About these notes

Written by the lecturer from the slides of the deck, as a public summary: the key ideas, the formulas and one worked example. The exercises, their solutions, the tests and the grading stay in the deck, for the students of the course.

Revision: 16 October 2026.